How to fetch large amount of data from SQL server database using multiple queries through C#

Question

0.00/5 (No votes)

See more:

SQL-Server-2008R2

I want to fetch data from historical tables of SQL server database. Currently I am having a set of minimum 20 queries and each query contains "Union All" of 10 queries. It will take almost a half hour (30 minutes) to execute all 20 queries sequentially. Again, if I use thread, then all queries get fired with in few time span and due to this sometimes database gets locked as there are 20 connections are open at a time.

Is there any better way to do this, stuck with this issue so many times?

What I have tried:

Use Thread , but creates deadlock situation.

Posted 19-Mar-16 3:58am

Member 11680954

Updated 19-Mar-16 7:05am

Add a Solution

Comments

CHill60 19-Mar-16 10:16am

Perhaps if you shared your queries we could assess them for performance

[no name] 19-Mar-16 10:58am

Share relevant code which takes time..

RickZeeland 25-Mar-16 14:07pm

And ? have you fallen asleep ? did you try any of the suggestions ?

2 solutions

Solution 2

U said you used Thread. Did you try to use PLINQ?

A case like that is a candidate for data partitioning. We could partition the 20 queries in 20 parallel executions with a defined parallelism degree.

It would help if we knew about the queries.

Posted 19-Mar-16 7:05am

Veronica S. Zotali

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

RickZeeland · Accepted Answer · 2016-03-19T04:52:00

Solution 1

You might try the BCP utility, which is meant to copy large amounts of data.
This can be executed from the command prompt, example:

C#

bcp "SELECT Name FROM AdventureWorks2008R2.Sales.Currency" queryout Currency.Name.dat -T -c

More information: bcp Utility[^]

Posted 19-Mar-16 4:52am

RickZeeland