How to fetch thousand records in 100*10 batches

Question

1.00/5 (1 vote)

See more:

Hi Friends,

I need to pull the data from the SQL Db and do some calculations. I am not interested to fetch all the records in a one shot. My requirement is to run the loop in batches.
For example if suppose i have 1050 records. it should run 11 loops ie 10*100+ 50

Whats the method should i follow. Should i write a stored procedure. If yes please give me some idea.

Below i my snapshot of the code

C#

SqlDataAdapter daAuthors = new SqlDataAdapter("Select * From Authors", objConn);
DataSet dsPubs = new DataSet("Pubs");
daAuthors.Fill(dsPubs,"Authors");

DataTable tblAuthors;
tblAuthors = dsPubs.Tables["Authors"];

			foreach (DataRow drCurrent in tblAuthors.Rows)
			{
				Console.WriteLine("{0} {1}",
					drCurrent["au_fname"].ToString(),
					drCurrent["au_lname"].ToString());
			}

Posted 11-May-15 9:39am

jinesh sam

Add a Solution

Comments

PIEBALDconsult 11-May-15 16:54pm

What database system?
If SQL Server 2012 (or newer), you might want to use the offset_fetch clause:
https://msdn.microsoft.com/en-us/library/ms188385(v=sql.110).aspx

And don't use a DataAdapter.

Maciej Los 11-May-15 16:54pm

What kind of calculation? Have you tried Common Table Expressions?

jinesh sam 11-May-15 21:36pm

@PIEBALDconsult I am Using SQL Server 2008. I accept your point i will use datareader

jinesh sam 11-May-15 21:41pm

@Maciej Los Calculations is little complex.Thats the reason, i am planning to do in batches.All calculations are on front end not on server side. I think CTE is for server side

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sascha Lefèvre · Answer 1 · 2015-05-11T10:16:00

I'd suggest a third option: Query for all records at once, but don't use DataAdapter.Fill - instead use a DataReader to read row by row. That way you don't have to deal with select-paging while still not having to read all records at once. It will even have the best performance (only one query and using a DataReader is faster than DataAdapter.Fill).

SqlDataReader (MSDN)[^]
http://www.dotnetperls.com/sqldatareader[^]

edit: It will have the best performance if the calculation has, for some reason, to be done in the application. If it could be done within the query (CTE) that would most certainly be the fastest way. But we can't tell, because we don't know what your calculation involves (yet).

edit2 after comment: In that case (processing millions of records and having to compare the results to a DB2 database) batching seems indeed to be the best approach. As you're using SQL Server 2008, there's no OFFSET FETCH-clause yet, so you will have to use some some 'conventional' paging approach. I think this article explains your options well:
SQL Server 2005 Paging Results[^]
Otherwise there are even more to take a look at:
https://www.google.com/search?q=codeproject+sql+server+select+paging&ie=utf-8&oe=utf-8[^]

Nonetheless, you could use a DataReader instead of DataAdapter.Fill to improve the performance somewhat.