Split a huge CSV file using C#

Question

0.00/5 (No votes)

See more:

Hi All,

I want to split a huge 2GB csv file to multiple files.
The split has to be based on the number of records
for eg 1-100000 : file1
100001-200000 : file2
200001-300000 : file3

till end of records.

Thanks for your help...!!

thanks
Venkat.

Posted 10-May-12 12:47pm

venred04

Updated 1-Feb-22 18:46pm

Add a Solution

Comments

TRK3 10-May-12 19:10pm

What have you tried? What specifically do you not understand or need help with?

The program you want to write is one of the simplest programs one could attempt. I'd almost expect it to be the second program you learn to write right after "hello world".

We're more than happy to help you learn, but you need to do a little study yourself and make an attempt at it. Then if there is something specific you don't understand or couldn't find in the documentation, we'll be happy to help with that.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Clifford Nelson · Answer 1 · 2012-05-10T13:37:00

If you have the memory, you can read all the lines into an array using File.ReadLines (string Path). Then you can figure the number of lines an move them into new arrays and do a WriteAllLines(String, String[]) for each file. You can use LINQ to get the lines:

C#

var x = File.ReadLines(@"C:\temp\connections.xml");
File.WriteAllLines(@"C:\temp\connections2.xml", x.Take(10).ToArray());

Use Skip after the first file.

M Imran Ansari · Answer 2 · 2022-02-01T18:46:00

Solution 4

You can also achieved this through PowerShell script. See the link for script and steps to split your large file according to size/limit.
Split a large csv file into multiple csv files according to the size in powershell - Stack Overflow[^]
Then you can check, How to run PowerShell script in C#:
Invoking PowerShell Script with Arguments from C# - Stack Overflow[^]

Posted 1-Feb-22 18:46pm

M Imran Ansari

Updated 1-Feb-22 18:47pm

v2

Comments

Richard Deeming 2-Feb-22 4:53am

Why would you want to use C# to run Powershell to do something you can just as easily do directly in C#?

Hopefully the OP managed to solve their problem some time in the ten years since they posted this question.

M Imran Ansari 2-Feb-22 5:18am

According to problem statement working in C#. Secondly, Why we get old questions in Active pane?

Richard Deeming 2-Feb-22 5:24am

So as I said, why offer a Powershell solution for a C# question, when C# is perfectly capable of solving the problem?

Old questions get dragged up into the "active" list when someone posts a new solution to them - usually when a spammer tries to hide their spam link in an old thread. Those spam solutions quickly get reported and removed, so the only clue that it wasn't you who resurrected this thread is the two missing solution numbers.

M Imran Ansari 2-Feb-22 5:35am

Agreed. But when we worked with big files sometimes we get into memory issues like OutOfMemory exception while processing file due to low resource (I have also experienced such issue and achieved through PS scripts) and then we will achieve though an alternative approach.
Thanks for detail clarification on second point.

Richard Deeming 2-Feb-22 5:39am

You'd run out of memory if you tried to load the whole file into memory. But if you stream the file, either using a StreamReader, or using the File.ReadLines method (instead of File.ReadAllLines or File.ReadAllText), you can easily split a huge file into smaller chunks with hardly any memory overhead.

M Imran Ansari 2-Feb-22 6:03am

Thanks for the explanation. I’ll check and review my solution.