Throw error when read large file

Question

2.00/5 (1 vote)

See more:

I need to order according to the text of a line of text reading, when I read the file size of about 100MB in all normal, but when I read the file size is about 300MB when the memory error occurred.

function detail as below:

C#

var filePath = @"D:\junk\test.txt";
var fileSize = new FileInfo(filePath).Length;
var viewSize = fileSize
List<string> rowArray = new List<string>();
using (var mm = MemoryMappedFile.CreateFromFile(filePath, FileMode.Open, null, 0, MemoryMappedFileAccess.Read))
using (var stream = mm.CreateViewStream(0, viewSize, MemoryMappedFileAccess.Read))
using (var reader = new StreamReader(stream, Encoding.UTF8))
{
    rowArray.Add(reader.ReadLine());
}

The rowArray is provided to the LISTVIEW data source, LISTVIEW uses a virtual mode

What I have tried:

I use the third party software to read the file, whether it is the size of 100MB or 300MB size, everything is normal

Why I don't use this method of File.ReadAllLines is that it takes 28 seconds to read the 300MB file, and the Third party software takes about 10 seconds
So I used the MemoryMappedFile technology, but unfortunately, the 300MB file size out of memory error.

Posted 2-Jan-17 19:47pm

xuyunhai20160827

Updated 3-Jan-17 15:50pm

v6

Add a Solution

Comments

OriginalGriff 3-Jan-17 4:51am

Answer updated - comments are not currently being displayed, it;s a known bug and they are working on it.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2017-01-02T22:26:00

Why are you making everything so complicated?
You create a new List of strings, to which you add a single string: the entire text content of the file, after memory mapping it and creating two streams form the mapped file.
Which is the equivalent of saying:

C#

List<string> rowArray = new List<string>();
rowArray.Add(File.ReadAllText(filePath));

Try that, and see if your memory problem goes away...

I need to read one line by one line and show it in ListView

That isn't what your code is doing, not even slightly.
And that very definitely isn't what you want to do at all, and is exactly why you are getting "out of memory" errors - never try to display that much information in one go. How many lines of text do you expect a file of 100MB (or worse 300MB) to contain? If the average is 80 characters per line - and it'll normally be a lot less - that's 1,250,000 lines for the 100MB file, and 3,750,000 for the bigger file. How do you expect your user to cope with that, never mind the system? Each of those lines is a separate control in a ListView! And that's why you run out of memory!

Don't do it. Page it, filter it, search it. But just throwing that much at a user and expecting them to cope it lazy and will only ever result in your user returning the software and demanding his money back...

Jochen Arndt · Answer 2 · 2017-01-02T22:32:00

The total memory that can be used by lists is limited to 2 GB. When using .NET 4.5 or later with 64-bit builds this limit can be increased with <gcAllowVeryLargeObjects> Element[^].

You may ask now why you run into this limit with files of about 300 MB size.

You have a string list. Strings are using UTF-16 encoding internally (two bytes per charater). If your input file contains only ASCII characters, the required memory to store the strings is twice the file size.

The list must also store the references to all strings. So you have 4 bytes (or 8 with 64-bit builds) per line.

The list will grow during reading. This requires re-allocation. Upon re-allocation, new memory is allocated, and the current content is copied before the old memory is released. During such operations you will probably get the out of memory exception. This can be avoided by setting the lists capacity which requires determination of the number of lines. This avoids re-allocation and allocating memory for more lines than actually required.

A solution for your problem might be using a list containing the line offsets for your memory mapped file. This list would only use 4 or 8 bytes per line. Because you are using a virtual list view, extract the strings in view by using the offsets (distance to next offset is line length).

Throw error when read large file

2 solutions

Solution 1

Solution 2

Add your solution here

Preview 0