Hi
Anyone got any idea on processing and displaying the content of a very large text file?
Currently, my attempt fails when the file size is in tens of megabytes. (StringBuilder constructor dies with Out of Memory Exception.)
I need to:
*Display the full log
*Display the full log without lines containing certain key words
*Display only the lines containing certain key words. These must be listed per key word
*Display or hide supplementary information at will
*Track a key word line back to the full log with ease.
=========================================
Here is the longer version of the story:
I've got a 3rd party (i.e. of which I have no control over the spec) software that spits out endless amounts of log, and I need to write a tool to analyze it.
The log is a simple text file delimited by "\r\n".
A common size is around 10MB.
At this size, notepad can't cope, but there are other viewers that can display the content, e.g. wordpad.
However, displaying is one thing, making sense of it is quite another.
So I made a tool that pulls out a line of log a time according to a set of predetermined key words and lists them in list boxes per key word.
So one list box lists all the "Open file" related log, another lists a "Start analysis" related log and so on.
Also, some commands take long input strings. For example, "Start analysis" takes an XML input of how to perform the analysis. For these logs, the details can be hidden or shown as instructed by the user.
Each item in the list boxes can be traced back to the full log which is displayed in a rich text box with double click.
It's exactly what I needed. I can cross reference the key words with the full log and view just the information I need at any given time...
except...
it blows up when the file size is in the tens of megabytes.
The biggest file I have to hand is over 200MB.
How can I extend my little log analyzer to cope with much bigger log files?
Is there a better way of implementing what I mean to do?
Any suggestions greatly appreciated.
Submit an article or tip