Database

Import Data From Other Database and Eliminate Duplicates

7-Jan-09 19:41

Glad it helped.

The need to optimize rises from a bad design.My articles[^]

polishprogrammer5-Jan-09 9:59

Re: Import Data From Other Database and Eliminate Duplicates

5-Jan-09 9:59

I need to import data from one Access database to another with virtually identitical structurers (the target database may have more columns, but it has every column that the database to be imported has). I want to do this for specific tables using SQL, but my database is protected by a database password. How do I construct the SQL statement to include the password?

Thanks.

Wendelius5-Jan-09 10:25

Re: Import Data From Other Database and Eliminate Duplicates

5-Jan-09 10:25

polishprogrammer wrote:
How do I construct the SQL statement to include the password

The SQL statement doesn't include the password, only the connection does.

The solution varies depending which tools you're about to use, but basically you take one connection to the access db, read data from there and put it to sql server using another connection. You could do this with c#, SSIS/DTS (depending on the db version), linked server etc.

The need to optimize rises from a bad design.My articles[^]

polishprogrammer5-Jan-09 10:38

Re: Import Data From Other Database and Eliminate Duplicates

5-Jan-09 10:38

Thanks, but both the databases are MS Access databases. I need to import data from one Access database into another Access database. I will be using C# to do this operation. How would I include the connection in any SQL operation I might perform? I've tried to Merge 2 data tables. It works (usually), but takes a VERY long time. That's why I was hoping to find something quicker, such as a SQL statement perform the work. If my databases were not secure, I could do this very easily, but the database password complicates things.

Thanks.

Wendelius5-Jan-09 10:58

Re: Import Data From Other Database and Eliminate Duplicates

5-Jan-09 10:58

I guess I would open a connection to the source db and then loop through the data and put the data to the target db using another connection. So I wouldn't use data tables in the middle. Also for removing duplicates I would order the source data.

It's been awhile since I last used access so there may be another ways, but I think the logic I described would get the job done.

The need to optimize rises from a bad design.My articles[^]

polishprogrammer7-Jan-09 9:47

Re: Import Data From Other Database and Eliminate Duplicates

7-Jan-09 9:47

Mika,

I found this statement worked to import the data, but it does not take care of the issue of duplicates. I need to customize this statement a bit to ensure that I'm only selecting those records from the C:\Program Files\Northwind.mdb database, say, that are not already in the target database. Sort of, perhaps, a reverse INNER JOIN. If you have any suggestions on that, I would appreciate it. Thanks.

INSERT INTO Employees
SELECT Employees.*
FROM [;DATABASE=C:\Program Files\Northwind.mdb;PWD=none].Employees;

Wendelius7-Jan-09 10:00

Re: Import Data From Other Database and Eliminate Duplicates

7-Jan-09 10:00

Aah, they've added external sources to Access. Starts to look like SQL Server.

Anyway, I don't know how you identify a duplicate, but let's say you have a column named ID and based on that you make the decision if it exists or not. In that case you could have for example:

INSERT INTO Employees
SELECT source.*
FROM [;DATABASE=C:\Program Files\Northwind.mdb;PWD=none].Employees source
WHERE NOT EXISTS (SELECT 1
                  FROM   Employees target
                  WHERE  target.ID = source.ID)

or even better if EXCEPT is supported by Access, you could have:

INSERT INTO Employees
SELECT *
FROM [;DATABASE=C:\Program Files\Northwind.mdb;PWD=none].Employees
EXCEPT
SELECT *
FROM Employees

The columns must be in the same order in both tables in all of the statements in order for them to work.

The need to optimize rises from a bad design.My articles[^]

polishprogrammer8-Jan-09 10:56

Re: Import Data From Other Database and Eliminate Duplicates

8-Jan-09 10:56

After some search (and trial and error), I found this statement to work:

INSERT INTO Employees
SELECT e1.* FROM [;DATABASE=C:\Program Files\Northwind.mdb;PWD=none].as e1
LEFT JOIN Employees as e2 ON e1.Field1=e2.Field1 WHERE e2.Field1 IS NULL

The IS NULL comparison selects only those records from the left side of the join that did not already have a match in the right (target) table.

It seems to work well. When I tried to import records through code (DataTable.Merge or some iteration technique), the import process would take many minutes for moderately sized tables (30000 to 70000 records), but using just the SQL it took the application about 6 seconds do the same operation.

The only major change I had to make when implementing this in code was to insert the appropriate variable for the database path and implement a function to get the password, as appropriate, such as Marshal.PtrToStringAuto(logPointer).

Thanks again for your suggestions and I hope this follow up I provided is useful.

Wendelius8-Jan-09 11:06

SQL Server- Upload CSV file via Bulk Insert with double Quotation Issue?

8-Jan-09 11:06

No problem and the follow up was useful! Even though I rarely use Access, it's always good to have understanding how it's developping. Thanks!

The need to optimize rises from a bad design.My articles[^]

Tarini Singh5-Jan-09 4:21

Tarini Singh

5-Jan-09 4:21

Hi Team,

Needed functionality is upload csv file via bulk insert only.

1)I Created table structure

CREATE TABLE #Test_temp
(
Name	varchar(100),
Age	varchar(100),
Salary	varchar(100),
Dept	varchar(100)
)

2)
creating the format file (.fmt)
8.0
4

1	SQLCHAR	      0       100     ","			1       NAME		SQL_Latin1_General_CP1_CI_AS
2	SQLCHAR	      0	      100     ","			2	AGE		SQL_Latin1_General_CP1_CI_AS
3	SQLCHAR	      0	      100     ","			3	SALARY		SQL_Latin1_General_CP1_CI_AS
4	SQLCHAR	      0	      100     "\r\n"			4	DEPT		SQL_Latin1_General_CP1_CI_AS

3)
The comma delimited csv data file

Name,Age,Salary,Dept
Scott,21,2343,salary
Tiger,22,34343,acct

It was successfully uploaded but
the input changes is that

In csv file, some filed contains values within (")double quotes with (,) .
like in third row below

Name,Age,Salary,Dept
<code>Scott,21,"23,43",salary</code>
Tiger,22,34343,acct

for this scenario, I took the fmt file as below

1 SQLCHAR 0 100 "," 1 NAME SQL_Latin1_General_CP1_CI_AS

2 SQLCHAR 0 100 "," 2 AGE SQL_Latin1_General_CP1_CI_AS

3 SQLCHAR 0 100 "\"," 3 SALARY SQL_Latin1_General_CP1_CI_AS

4 SQLCHAR 0 100 "\r\n" 4 DEPT SQL_Latin1_General_CP1_CI_AS

Could anyone help me out from this scenario.

Tarini Singh
Sr Software Engg.
email-tksingh@zenta.com

Re: SQL Server- Upload CSV file via Bulk Insert with double Quotation Issue?

Wendelius5-Jan-09 9:18

Problem with large database fetching

5-Jan-09 9:18

I think you could also set quotation marks on the second field like:

2 SQLCHAR 0 100 ",\"" 2 AGE SQL_Latin1_General_CP1_CI_AS

However, it would be better, if the character data is always in quotation marks. Otherwise you may have problems if the data contains commas etc.

So could you get the data in following format (assuming that the number are actually characters as your table defines them):

"Scott","21","23,43","salary"
"Tiger","22","34343","acct"

The need to optimize rises from a bad design.My articles[^]

Member 40015965-Jan-09 2:40

Member 4001596

5-Jan-09 2:40

hi everyone

i have create a web page where i place two DropDownList in First i bound Country List and in second one i am bounding places list according to selected country my problem is that when place list have less items it works fine but when list goes large Ex more than 5000 or more it take so much time some times it it show time out error (Sql Server). i also try with Ajax tool kit with Cascading Dropdownlist extender with web service method but problem not solve. here is my code

// for country selection

Public Shared Function getContry() As DataTable
Dim Ds As DataSet
Dim contry_list As String
contry_list = "SELECT Country_List.CountryNo,Country_List.CountryName From Country_List"
Ds = SqlHelper.ExecuteDataset(connection, CommandType.Text, contry_list) ' "Get_Country_List")

Return Ds.Tables(0)
End Function

/// for Place List

Public Shared Function getplace(ByVal CountryNo As Integer) As DataTable
Dim Ds As DataSet
Dim getplacelist As String
getplacelist = "SELECT Place_List.PlaceNo,Place_List.PlaceName From Place_List " & " WHERE Place_List.CountryNo = " & CountryNo
Ds = SqlHelper.ExecuteDataset(connection, CommandType.Text, getplacelist)
Return Ds.Tables(0)
End Function

and on .aspx page i am using

Private Sub fillCountry()
Dim dt As DataTable
dt = getContry()
ComboBox1.DataSource = dt
ComboBox1.DataValueField = dt.Columns("CountryNo").ToString
ComboBox1.DataTextField = dt.Columns("CountryName").ToString
ComboBox1.DataBind()

End Sub

Private Sub fillPlace(ByVal CountryNo As Integer)
Dim dt As DataTable
dt = getplace(CountryNo)

ListBox1.DataSource = dt
ListBox1.DataValueField = dt.Columns("PlaceNo").ToString
ListBox1.DataTextField = dt.Columns("PlaceName").ToString
ListBox1.DataBind()
ListBox1.Enabled = True
End Sub

where i am wrong plz help me

Ashfield5-Jan-09 2:54

Ashfield

5-Jan-09 2:54

I suspect you need to look at the indices on the tables. An index on Place_List.CountryNo would certainly help. You could also look at caching as the places per country probably do not change much.

Bob
Ashfield Consultants Ltd

Wendelius5-Jan-09 9:22

5-Jan-09 9:22

To add to Ashfield's post, try to also cache the countries. That would help the fillCountry sub. As Ashfield wrote, indexing and caching helps the fillPlace sub.

The need to optimize rises from a bad design.My articles[^]

Aman Bhullar6-Jan-09 5:45

Aman Bhullar

6-Jan-09 5:45

Have you tried using Same query on Query Analyser. Is there same error or it displayes the result(Although it takes time).

Try checking the timeout of SQLCommand object in function
ExecuteDataset

By default there is no value assigned.
Try assisgning 40000, and check where the error of time out is there.

Alo check the indexing of table.

Member 40015966-Jan-09 20:22

Member 4001596

6-Jan-09 20:22

thx for support this was error coz thr was no indexing thx a lot

Need SQl Query

karthigachawla5-Jan-09 0:48

karthigachawla

5-Jan-09 0:48

Hai,

I need a query to list a column in order.it is varchar datatype.column name is accno.I has values like AC1,AC6,AC10,BJ8,BJ20,CK12 etc..

i have to list in order like

AC1
AC2
AC10
BK1
BK2
VH1

please any one help me

SeMartens5-Jan-09 1:09

SeMartens

5-Jan-09 1:09

Hi,

maybe you could work with substrings (if all of your values have two leading chars):

<br />
select * from mytable <br />
ORDER BY substr(mycolumn, 0, 2), cast(substr(mycolumn, 3) as int)<br />

Regards
Sebastian

It's not a bug, it's a feature!

Me in Softwareland.

Ben Fair5-Jan-09 4:13

Ben Fair

5-Jan-09 4:13

Unless I missed something, it seems like you just need an ORDER BY clause:

select * from table order by accno

Keep It Simple Stupid! (KISS)

SeMartens5-Jan-09 4:26

SeMartens

5-Jan-09 4:26

Hmm, I don't think that this is working, because a varchar-column is the order-criteria. So it will end up with this:

AC1
AC10
AC2

since it is ordering it lexicographical.

It's not a bug, it's a feature!

Me in Softwareland.

Confused about instance name in SQL cluster

Ben Fair5-Jan-09 5:51

Ben Fair

5-Jan-09 5:51

Yes, sorry, you are right; substringing will be needed.

Keep It Simple Stupid! (KISS)

George_George3-Jan-09 23:15

George_George

3-Jan-09 23:15

Hello everyone,

I am learning to install SQL Cluster. I am confused about the following concepts, does anyone have any good ideas?

1. I am always confused about the differences between instance/node concept, and also confused about instance name and virtual server name for a SQL cluster during setup process. Could anyone help to clarify please?

2. Do I need a dedicated sub-network to install a SQL Cluster and connect the dedicated sub-network to the LAN (which SQL clients are working on) by hub/switch/router? Or such dedicated sub-network is not needed and all nodes in the SQL cluster could be in the same network of the clients?

thanks in advance,
George

Re: Confused about instance name in SQL cluster

Wendelius4-Jan-09 0:49