Click here to Skip to main content
15,891,136 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
For our Project, from C# we execute an Oracle SQL Query and store the result of the query in the output file (CSV or TXT or XML file). Since the number of records are huge (like 2,848,489 records) for some of the files, it takes more time to finish creating the Output file. So, I tried fetching the data in ranges(row limits) to a datatable in a dataset and write the corresponding data to an Output file in a loop. For each iteration, I clear the DataTable once the data from it is written into the output file, and then load the new set of data.
But, in this method (code part provided below), based on the row limits in the loop, the data load into a particular iteration of the DataTable fails with:
'System.OutOfMemoryException' error as the CPU and Memory are fully utilised. How to avoid this error? Is there any alternate method for creating the output file without memory error?

What I have tried:

This is code part from the Windows Application I created :
C#
private void button2_Click(object sender, EventArgs e)
        {
            mOraAuth = "Persist Security Info=False;Data Source=" + mOraDataSource.ToString();
            mOraAuth = mOraAuth + ";User Id=" + mOraUserID;
            mOraAuth = mOraAuth + ";Password=";
            mOraAuth = mOraAuth + mOraPassword.ToString() + ";";
            maxRows = 1000; 
            Conn = new OracleConnection(mOraAuth.ToString());
            Conn.Open();
            DataSet ds = new DataSet();
            MainQuery = "SELECT  * FROM t_ic_efeed_wsstandard WHERE JOBFILENAME = 'C416162_GE_WSSTANDARD_20150901.csv' ORDER BY confdate, part_start";
            OracleDataAdapter adapter = new OracleDataAdapter(MainQuery, Conn);

            string Count_Query = "SELECT Count(*) FROM (" + MainQuery + ")";

            OracleCommand Command = new OracleCommand(Count_Query, Conn);
            Command.CommandType = CommandType.Text;
            OracleDataReader Reader = Command.ExecuteReader();
            while (Reader.Read())
            {
                TotalRows = Reader[0].ToString();
            }
            LoopCount =  Convert.ToInt32(TotalRows) / maxRows + 1;
            
            string DataTableName = "MyTable";
            ds.Tables.Add(DataTableName);
            for (DTLoop = 0; DTLoop < LoopCount; DTLoop++)
            {
                if (DTLoop == 0) CurrRow = 1;
                adapter.Fill(ds, CurrRow, (CurrRow + maxRows - 1), DataTableName);
                CurrRow = CurrRow + maxRows;
                CreateCSVFile(ds.Tables[0], AppDomain.CurrentDomain.BaseDirectory + "\\C416162_GE_WSSTANDARD_20150901.csv");
                ds.Tables[0].Clear();
                ds.Tables[0].Dispose();
            }
            ds.Clear();
            ds.Dispose();
            Conn.Close();
        }
Posted
Updated 5-Jun-17 2:59am
v2
Comments
govindarajan k 5-Jun-17 8:52am    
Additional comments :
Testing done with :

Data Details :
Total Records : 2,848,489
MaxRows value set as :10000
Loop count : 285

Error Details :
Failed while loading data to the DataTable from 650001 to 660000 rows with System.OutOfMemoryException occurred error.
Richard MacCutchan 5-Jun-17 9:59am    
You cannot load unlimited amounts of data into a system that has limited memory. Redesign your application so it only extracts the data necessary for each transaction.

1 solution

I would recommend using an SqlDataReader, see: [SqlDataReader] instead of using DataTable and DataSet.
Also I think Oracle provides query options like Rownumber or Limit to get only a limited amount of records at a time.
 
Share this answer
 
Comments
govindarajan k 5-Jun-17 9:27am    
Thanks for your reply RickZeeland. I am not using SQL Server Database but use Oracle Database Connection for this Project.
RickZeeland 5-Jun-17 10:58am    
You are right, it would be an OracleDataReader then, see: https://msdn.microsoft.com/en-us/library/system.data.oracleclient.oracledatareader(v=vs.110).aspx
See here for information about paginating with Rownum: http://www.oracle.com/technetwork/issue-archive/2006/06-sep/o56asktom-086197.html

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900