Click here to Skip to main content
15,891,942 members
Please Sign up or sign in to vote.
1.44/5 (3 votes)
See more:
I have large db ( more than 10 M) , if it’s too huge I get out of memory exception .
I wanted to know how I can confirm that how many rows my application can support based on Ram size .

What I have tried:

I tried up to 8M rows and it looks good but beyond that it gives Out of memory exception.
Posted
Comments
Mehdi Gholam 12-Oct-18 2:26am    
Show your code, it is hard to comment without context (csv is a text file and you can buffer write to it, and you can batch read from sql) so you should not hit memory limits.
prashant190 14-Oct-18 3:42am    
Please find the code snippet , another point I would like to mention is, I had requirement to display the results in grid as well if required and colomns needs to be shown based on configuration file .
Below function takes care of writing to CSV from DV

  try
            {
                string outputFile =  ConfigurationManager.AppSettings["csvFileName"].ToString();
                outputFile = CSVFilePATH + "\\" + outputFile ;
                if (dataView.RowCount > 0)
                {
                    string value = "";
                    DataGridViewRow dr = new DataGridViewRow();
                    StreamWriter swOut = new StreamWriter(outputFile);

                    //write header rows to csv
                    for (int i = 0; i <= dataView.Columns.Count - 1; i++)
                    {
                        if (!names.Contains(dataView.Columns[i].Name))
                        {
                            if (i > 0)
                            {
                                swOut.Write(",");
                            }
                            swOut.Write(dataView.Columns[i].HeaderText);
                        }
                    }

                    swOut.WriteLine();

                    //write DataGridView rows to csv
                    for (int j = 0; j <= dataView.Rows.Count - 1; j++)
                    {
                        if (j > 0)
                        {
                            swOut.WriteLine();
                        }

                        dr = dataView.Rows[j];

                        for (int i = 0; i <= dataView.Columns.Count - 1; i++)
                        {
                            if (!names.Contains(dataView.Columns[i].Name))
                            {
                                if (i > 0)
                                {
                                    swOut.Write(",");
                                }
                                try
                                {
                                    value = dr.Cells[i].Value.ToString();
                                }
                                catch
                                {
                                    value = "";
                                }
                                //replace comma's with spaces
                                value = value.Replace(',', ' ');
                                //replace embedded newlines with spaces
                                value = value.Replace(Environment.NewLine, " ");

                                swOut.Write(value);
                            }
                            
                        }
                        lblMessage.Text = "Downloaded file " + outputFile + " Successfully...";
                    }
                    swOut.Close();
                   // dataView.Visible = false;
                }
            }
            catch (Exception exp)
            {
                
                Logger.Info("Exception during File Export..\r\n" + exp.Message.ToString());
            }
Richard MacCutchan 12-Oct-18 5:06am    
You cannot load all the data into memory at once. Read only the rows that you need for each request.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900