creating a list<struct>

Question

0.00/5 (No votes)

See more:

I have the below code To save data table data in to list I have 1 million records in datatable but after 524288 row I get this error An exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll but was not handled in user code how can I resolve it?
thanks

Posted 25-Nov-14 20:04pm

Angela 10293848

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Kornfeld Eliyahu Peter · Answer 1 · 2014-11-25T20:39:00

Solution 1

Quote:
I have the below code

What code?

Memory - by its nature - is limited!
Before you ask yourself how to fill a list with 1 million records, ask yourself: 'what will I do with that list?'...
Present to the end user? No way any user can handle a list of 1 million items?
Save onto some other media (disk? DB?). Do it bu chunks/pages, that will save some memory for you...

Posted 25-Nov-14 20:39pm

Kornfeld Eliyahu Peter

Comments

Angela 10293848 26-Nov-14 3:10am

if I select 100000 i get similliar error

Kornfeld Eliyahu Peter 26-Nov-14 3:16am

In software design 100,000 is as bad as 1,000,000!!!

Angela 10293848 26-Nov-14 3:30am

oh come on I do not think it is too large in these days

Sinisa Hajnal 26-Nov-14 3:33am

It is not too large (depending on the type you're keeping them in) - but from design perspective, it is bad. You should only keep in memory those items you're actively working with or which you plan on showing. All others should be safely in the database.

And if you get the error for 100 000 either you have really large memory footprint or you're copying it (without realizing it) several times in the memory...in the inital question you said you get the error around 500 000 so how do you run out of it in 100 000 scenario? You're doing something. Please psot some code. Thank you.

Kornfeld Eliyahu Peter 26-Nov-14 3:35am

It is a fact that your application/computer can not handle it! So what do you think is irrelevant!!!
You are however right that a million records is not to large (I have tables with hundred of millions) for data processing, but even a 100 records is large for UI...
So - as I told already - before you start running over your millions, ask yourself: what for?

Angela 10293848 26-Nov-14 3:38am

look I just want to know why I get that error?is it related to hardware or it is limitation of a list Which on?

Sinisa Hajnal · Answer 2 · 2014-11-25T20:44:00

Solution 2

AS the error says, you ran out of memory.

The solution is:
a) use smaller set of data (filtering)
b) save it partially, take first 100 000 to list and save, take another 100 000 etc
c) redesign so that you only keep single page of data in memory, the rest is called as needed from the depository (database)
d) increase the memory available to the process :)

Posted 25-Nov-14 20:44pm

Sinisa Hajnal

Comments

Angela 10293848 26-Nov-14 2:51am

I need to select 1 million

Kornfeld Eliyahu Peter 26-Nov-14 3:15am

Why? Even you had a million dollar you would never walk around with it, but put most of it in a bank (or under the bed)...

Sinisa Hajnal 26-Nov-14 3:30am

If you think you need them for some kind of processing - no, you don't. Databases are good at processing millions of rows of data. You're limited to your system resources. Simply selecting them for whatever reason is never good idea. You'll never show a list of million items so there is no need to keep them in memory.

If you specify your requirements, we might be able to help. The solution above offers you options. You cannot avoid physical limitations (in this case your memory capacity). Whatever you might need / wish / command is irrelevant, it will not change the physical realities.

Angela 10293848 26-Nov-14 3:33am

I need it to provide a report yes you are right end user can not see all records together but He can filter some data that who need so the grid view should have all data

Sinisa Hajnal 26-Nov-14 3:37am

Make them filter it before getting the data. At least in general sense (for example only current year of something)