|
Hey guys.
In the try block I am populating a datset and a table(tbl) and doing some other processing then saving the data to disk. All of the code works in the try block.
I set up an exception in the file name that I am saving to to get some error logging going.
the catch is almost a verbatum translation from some older vb code that works fine.
Rafone
Statistics are like bikini's...
What they reveal is astonishing ...
But what they hide is vital ...
|
|
|
|
|
When you declare tbl, you aren't actually assigning anything to it. So, it probably looks something like this
DataTable tbl; Then you are assigning items to it in the try block. What you actually need to do is set it up like this
DataTable tbl = null; or better still, assign it in the try block and return it there. If there's an error then the return is superfluous, and it won't be hit.
|
|
|
|
|
I just finished reading a great article on Garbage Collection in the .NET Framework:
http://msdn.microsoft.com/msdnmag/issues/1100/GCI/
This article says that a garbage collection occurs whenever a new object needs to be created and there is not enough room on the heap for it. It says that the garbage collection will clear out the garbage objects to make room for the new object, but what if you have a large number of live (non-garbage) objects and the garbage collector is unable to free enough space for the new object?
Can the size of the heap grow at this point?
If so, what is the heap's default size and by how much will it grow in this situation?
I have searched and I have been unable to find the answers to these questions documented anywhere. There needs to be some official documentation on this.
Thanks,
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
Mogaambo wrote: Can the size of the heap grow at this point?
Yes, you can check it out yourself. Open Perfmon (type perfmon.msc at the command prompt), right click on the graph and select Add Counters, select .NET CLR Memory and select one of the heap related parameters (#Total Committed Bytes, for example). Select your process and click OK. You should be able to watch the GC heap expand as you create more objects and don't release much.
Mogaambo wrote: If so, what is the heap's default size and by how much will it grow in this situation?
I don't the heap's default size (Gen 0 + Gen 1 + Gen 2 + Large Object Heap), I would imagine it would vary by platform (x86 vs x64), workstation vs server etc. The max size of the heap would be restricted to the max virtual memory of a process, which is 2 GB (actually 4, but the OS takes 2) on x86.
|
|
|
|
|
I recently read about writing Unsafe code in C#. Can anyone provide me with a simple scenario where this can prove to be useful. I can then implement it to have a better understanding of it.
I also read about stackalloc. It seems to be a very efficient way of using arrays. Still can not see it that prominent in codes. What is reason for that?
"If you had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be 'meetings'." - Dave Barry
|
|
|
|
|
d@nish wrote: Can anyone provide me with a simple scenario where this can prove to be useful.
When you have to call into unmanaged code.
d@nish wrote: I also read about stackalloc. It seems to be a very efficient way of using arrays. Still can not see it that prominent in codes. What is reason for that?
Because C# has its own managed mechanisms that are safer.
|
|
|
|
|
Thanks for reply.
Colin Angus Mackay wrote: When you have to call into unmanaged code.
In that case I can create a managed wrapper and consume it. Isn't it going to perform same thing?
"If you had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be 'meetings'." - Dave Barry
|
|
|
|
|
Yes. In some circumstances you may wish to write your own wrapper, though IMO C++/CLI is a much better tool for the job in this sitation.
|
|
|
|
|
d@nish wrote: I recently read about writing Unsafe code in C#. Can anyone provide me with a simple scenario
Look for image processing articles. They typically use unsafe code to directly manipulate the bitmap using pointers.
d@nish wrote: I also read about stackalloc. It seems to be a very efficient way of using arrays.
1. The size of the stack is usually 1MB
2. The memory will be lost when the function returns.
|
|
|
|
|
Thanks. Will try that.
"If you had to identify, in one word, the reason why the human race has not achieved, and never will achieve, its full potential, that word would be 'meetings'." - Dave Barry
|
|
|
|
|
Hi ,
I am getting out of memory exception, what i am trying to do is to read 25 text files, all files have over million records, and write them to database.
My approach....
1. Created a list of 25 datatable.
2. Read the column names from text file and add them to DataTable and with their datatype and size.
3. Pass this list to a function whose purpose is to read each file and add records to each of the datatable.
It is just an rough but an important overview.
In between i am running two progress bar.
but i get out of memory exception.
can anybody tell me any other approach......
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why?
Why not just read the files one-by-one and write them to the database?
|
|
|
|
|
Colin Angus Mackay wrote: Your strategy is to dump the entire contents of the files in to memory then write them to the database. Why?
Because I am Passing That List Datables to a function whose purpose is to write data to table using sqlbulkcopy
Colin Angus Mackay wrote: Why not just read the files one-by-one and write them to the database?
So you want me to change strategy.
1. for( int i=0; i<totalnumberoffilestoscan;> {
Step1: // Create A New DataTable
Step2: // Call a Function That accepts the ith File and newly created DataTable as parameter and do the work of adding required number of Columns with their DataType and size
Step3: // Call a function that will read the text file row by row and add them to datatable
Step 4: // Pass this DataTable to another function whose purpose is to use SQlBulkCopy to write Records from the datatable.
}
Is This Strategy Feasible, will not be again give me out of memory Exception. in each loop i am creating a new Datatable ??????
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
Mogaambo wrote: Is This Strategy Feasible, will not be again give me out of memory Exception. in each loop i am creating a new Datatable ??????
I don't know if it will work for you because I don't know enough about your data. I do know that it will use a lot less memory than you are using now.
If an individual file is large enough then you will still get an out of memory exception. It has been suggested to you else where that you read each file in chunks and process the chunks. That seems the most sensible strategy to me.
|
|
|
|
|
25 files x 1 Million rows = 25 Million
If you say each row is 1024 bytes. Its totally 25 GB of memory required. As you might already know, when RAM is not sufficent the system writes to harddisk & its slow. Unless you have free space of 40GB (to accomodate calculation issues), you cannot run your application & still it will be slow.
So your approach is to break it up into pieces. You got few approaches:
1. Dont use datatables and use memory streams, use SqlCommand - this is more code, i guess
2. If you want to use datatables. You have to run the BulkCopy, for each datatable & call DataTable.Dispose() method (and may be call to Garbage collector, as Dispose will not free memory immediatly) after each use.
3. Use SQL Server, to import the files directly. DTS / SISS supports importing of various file formats, excel, csv, access. It will need a SQL command, which will take the file location & it will import it. You can google up for the sql syntax or use DTS Export / Import wizard & in last step, save the command.
|
|
|
|
|
Didn't you already ask this question once? Actually, this is the 3rd time.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001
|
|
|
|
|
John Simmons / outlaw programmer wrote: Didn't you already ask this question once?
Yes, he did, like 10 days ago and got some replies. Must be some amount of density involved.
"The clue train passed his station without stopping." - John Simmons / outlaw programmer
"Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon
"Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham
|
|
|
|
|
Paul Conrad wrote: Didn't you already ask this question once?
Yes I did , but check yourself what answers i received, are they fruitful
no no no no
So this time I asked my question in a different manner.
If, you can provide a good path then take thanks in Advance.
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
Mogaambo wrote: check yourself what answers i received, are they fruitful
Yes, but this one and this other one are pretty right on the money.
"The clue train passed his station without stopping." - John Simmons / outlaw programmer
"Real programmers just throw a bunch of 1s and 0s at the computer to see what sticks" - Pete O'Hanlon
"Not only do you continue to babble nonsense, you can't even correctly remember the nonsense you babbled just minutes ago." - Rob Graham
|
|
|
|
|
Paul Conrad wrote: Yes, but this one and this other one are pretty right on the money.
I don't like their ideas because they say that you read the file 10000 lines and use the sqlBulkcopy to write that 10000 records and then again read 10000 records, but suppose if i read 10000 records from a file and successfully written them to table but on next time when i read next 10000 records and when in the time of writing records to table any exception occurs then my Table goes to inconsistent state, because 10000 records are in, but it should be like that either all data written or none of them should be written.
Then...........................
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
There's this really neat thing in databases called "transactions". Start a transaction, perform your operations, if all goes well you commit the transaction, if not then you rollback the transaction.
|
|
|
|
|
SqlTransaction will only be used for current 10000 records not for next 10000 records..
Like this
step 1: //DataTable dt= ReadTenThousandLines();
Step 2: // Write_to_database (dt);
In Write_to_database() function, I will use sqlBulkCopy nad their i Use SqlTransaction Class, but suppose if i again passing 10000 Record containing datatable and if it fails then ??????????????
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
Mogaambo wrote: SqlTransaction will only be used for current 10000 records not for next 10000 records..
That's a bit pointless.
Mogaambo wrote: In Write_to_database() function, I will use sqlBulkCopy nad their i Use SqlTransaction Class, but suppose if i again passing 10000 Record containing datatable and if it fails then ??????????????
My advice has not changed. You use a transaction. You start it, you do all your bulk copying then you commit the transaction. You do NOT create one transaction for each batch of data. You create ONE transaction to cover everything.
|
|
|
|
|
With your suggestion reading from file and writing to database should be done in 1 shot.
For eg.
WriteToDataBase(string filename)
{
//Begin SqlTransaction
Begin loop to read 10000 records at a time (While all the records are not read)
//write in a datatable
//End loop
//End SqlTransaction
}
// Some thing like this you want me to do
“You will never be a leader unless you first learn to follow and be led.”
–Tiorio
"Coming together is a beginning, staying together is progress, and working together is success." Henry Ford
|
|
|
|
|
You are not paying attention. Or you are not bothering to understand transactions.
Begin Transaction
Start File Loop
Start Data Chunk Loop
Read upto 10,000 records from file
Write those records to the database
End Data Chunk Loop
End File Loop
End Transaction
Each iteration around the Data Chunk Loop has one communication with the database (One "shot" per iteration)
If you have a million records you have 100 "shots" at writing to the database.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
