Problems with CSV file blowing out memory - C# Discussion Boards

JoeRip19-Dec-07 17:21

19-Dec-07 17:21

You can't rely on only one create event per file created. This behavior is not deterministic between applications.

Notepad, for example, tests to see if a file exists by trying to create it first. Then it deletes it, and then it creates it. So you can get a bunch of the same FSW events for the creation of a single file.

By the way, I learned this by reading some article here on codeproject, as I recall Smile | :)

. In the comments on the article, people were also talking about a horrible memory leak their FSW was causing, and nobody had a solution for that, either.

Re: FileSystemWatcher and Threading

JoeRip19-Dec-07 17:32

JoeRip

19-Dec-07 17:32

In fact, the Net is rife with examples of your same problem. Trying to find the article I saw before, I googled "FileSystemWatcher Notepad" and got many hits, talking about the multiple creation event problem. Here's an example:

http://community.bartdesmet.net/blogs/bart/archive/2004/10/21/447.aspx[^]

Note that a google of "FileSystemWatcher multiple creation events" would have turned up an even better review of articles on this.

Some of the articles/blogs/text linked claimed to have solutions, also.

Re: FileSystemWatcher and Threading

JoeRip19-Dec-07 17:45

JoeRip

19-Dec-07 17:45

One of the articles I sited before tries to reference this, but their link is broken:

http://msdn2.microsoft.com/en-us/library/xcc1t119(VS.71).aspx[^]

This is the "Troubleshooting FileSystemWatcher Components" section of the larger item, "Reacting to File System Events", which itself is a sub-section of the "Creating File System and Timer Components" item in MSDN. It speaks directly of why you will receive multiple events when you think you should get one. Applications do strange and unpredictable things when writing to disk.

http://msdn2.microsoft.com/en-us/library/xcc1t119(VS.71).aspx[^]

All good reads.

Re: FileSystemWatcher and Threading

#realJSOP19-Dec-07 23:20

#realJSOP

19-Dec-07 23:20

So what's the solution?

"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001

Re: FileSystemWatcher and Threading

JoeRip20-Dec-07 1:40

JoeRip

20-Dec-07 1:40

I don't know. Read the articles?

Re: FileSystemWatcher and Threading

#realJSOP20-Dec-07 2:37

#realJSOP

20-Dec-07 2:37

Yes, I read the articles. No, no solutions were posted in them. I already have something in mindm, but I wanted to see what others did to resolve it.

In my particular case, all of the files are guaranteed to have unique file names, so I can maintain a list of the last 100 files that have already been processed, and ignore duplicate events on the same filename, but that doesn't address some of the other issues cited in the articles you listed.

Re: FileSystemWatcher and Threading

JoeRip20-Dec-07 1:42

JoeRip

20-Dec-07 1:42

Well, one solution would be to read the articles Smile | :)

. I did not.

Judging from what I am seeing in your code, I suggest that you check to see if you already have a thread going for that file, and terminate.

Basically, this is a design issue. The world is such that you can't count on a single create event. So, if you don't want multiple threads going, you're going to need to do your own checking.

Problems with CSV file blowing out memory

LongRange.Shooter19-Dec-07 11:05

LongRange.Shooter

19-Dec-07 11:05

I have two .csv files .. one is a filter content and the other is national data. The national data file crashes when Excel tries to open it BUT if my PM "uses Access" he can open all of the data.

I used a StreamReader and did an inline ReadLine() but that got the same result as Excel -- out of memory error. Then I read that if you get an out of memory failure using ReadLine that using the more robust Read() will resolve the issue.

I changed my code to use an inline method called GetLine() so that I didn't change my code too drasitcally and ran a test. It still runs out of memory. (code is below) Anyone have any ideas on how I can get past the I/O failure?

private string GetLine(StreamReader reader)
{
    if (reader.Peek() < 0) return null;

    List<char> buffer = new List<char>(limit);

    while (reader.Peek() >= 0)
    {
        buffer.Add((char)reader.Read());
        if (reader.Peek() < 0) return Debuff(buffer);

        if (buffer.Count > 3 &&
            (buffer[buffer.Count - 1].Equals('\r') && buffer[buffer.Count - 2].Equals('\n')) ||
            (buffer[buffer.Count - 1].Equals('\n') && buffer[buffer.Count - 2].Equals('\r')))
            return Debuff(buffer);
    }
    return Debuff(buffer);
}

Re: Problems with CSV file blowing out memory

Ed.Poore19-Dec-07 11:19

Ed.Poore

19-Dec-07 11:19

First question how big does buffer get?

My Blog[^]

Re: Problems with CSV file blowing out memory

LongRange.Shooter19-Dec-07 13:53

LongRange.Shooter

19-Dec-07 13:53

Well, believe it or not your question solved my problem. I added a try{}catch{} and found that the buffer exceeded 256MEG in size. Much larger than any record in the file could possibly be. So I traced the characters and found that it only contained a \n by itself at the end of each line.

Found many other inconsistancies in the file (like quoted numbers) but the application is now processing the file without exceeding memory!

Thanks for the help.

Michael

Re: Problems with CSV file blowing out memory

Skippums19-Dec-07 12:39

Skippums

19-Dec-07 12:39

The advice that you received that the Read method is more robust is due to the following: The readline method goes until it finds a newline character. If you have a 2,000 GB file with no newline characters, then ReadLine attempts to read it ALL into memory (which is clearly going to throw an out of mem exception). The code you provided above is using Read, this is true. But you are still getting the same exception because you basically just reimplemented the same functionality, and are still trying to load a 2000 GB file into memory. The reason why Read is suposedly "safer" is because typically, you only need a portion of the file at any given time. Read into memory only that which you need, or at minimum put a limit to the file size you can handle. Hope this helps,

Sounds like somebody's got a case of the Mondays

-Jeff

random files with different date

Shani Aulakh19-Dec-07 9:09

Shani Aulakh

19-Dec-07 9:09

The following code is generating files perfectly, But when i change the date on my laptop to get files with different modified times. It just overrides all the files with the recent date.

private void GenerateTestFiles_Click(object sender, EventArgs e)
{
StreamWriter outStream = null;
string message = null;

for (int i = 0; i < (100); i++)
{
string filename = "output";
filename += i.ToString();

if (i % 3 == 0)
{//multiple of three
filename += ".txt";
}
else if (i % 5 == 0)
{//multiple of 5
filename += ".log";
}
else if (i % 7 == 0)
{// multiple of 7
filename += ".exe";
}
else if (i % 9 == 0)
{// miltiple of 9
filename += ".dat";
}
else if (i % 11 == 0)
{// multiple of 11
filename += ".jpq";
}
else
{
filename += ".bmp";
}

FolderWatcher fw = new FolderWatcher();
filename = fw.FolderPath + "C:\\CsharpTempFolder\\" + filename;

try
{
outStream = new StreamWriter(filename);
outStream.WriteLine("Hello World");
outStream.Close();
}
catch (Exception Ex)
{
message = "Unable to create " + filename + "\r\n";
message += String.Format("Reason: {0}", Ex.Message);
MessageBox.Show(message);
}
}
}

1. Can i generate files with random numbers. Like, in the loop i takes random values using System.Random class. Or, add datetime object. what is the most simpliest way of doing it.

Re: random files with different date

pmarfleet19-Dec-07 9:20

pmarfleet

19-Dec-07 9:20

Wouldn't it be easier to just use Path.GetRandomFileName[^]?

Paul Marfleet

"No, his mind is not for rent
To any God or government"
Tom Sawyer - Rush

Re: random files with different date

Anthony Mushrow19-Dec-07 9:48

Anthony Mushrow

19-Dec-07 9:48

pmarfleet wrote:
Wouldn't it be easier to just use Path.GetRandomFileName[^]?

What the?!?... i can't beleive thats actually a built-in method. Why would that be there?

My current favourite word is: Bauble!

-SK Genius

Re: random files with different date

pmarfleet19-Dec-07 9:58

pmarfleet

19-Dec-07 9:58

SK Genius wrote:
Why would that be there?

Because .NET rocks.

Seriously, the purpose of this method is to provide the means of generating a random directory/file name that is cryptographically secure, i.e. that could not be guessed at by a malicious script.

Paul Marfleet

"No, his mind is not for rent
To any God or government"
Tom Sawyer - Rush

Re: random files with different date

Shani Aulakh19-Dec-07 17:50

Shani Aulakh

19-Dec-07 17:50

wow..........thats crazy..i will try it out tomo Smile | :)

Re: random files with different date

Dave Kreskowiak20-Dec-07 8:57

Dave Kreskowiak

20-Dec-07 8:57

pmarfleet wrote:
the purpose of this method is to provide the means of generating a random directory/file name that is cryptographically secure

It has nothing to do with cryptographically secure anything. All it does is genereate a filename that is guaranteed to be unique within the folder where it's created. This function can only generate 65535 unique temp filenames before it throws up an exception. The solution is to delete the existing temp files in the folder and try again.

A guide to posting questions on CodeProject[^]

Dave Kreskowiak
Microsoft MVP
Visual Developer - Visual Basic
2006, 2007

Re: random files with different date

pmarfleet20-Dec-07 9:14

pmarfleet

20-Dec-07 9:14

My choice of words was wrong. I should have said cryptographically strong, i.e. unpredicatable (quoting from the MSDN documentation).

Paul Marfleet

"No, his mind is not for rent
To any God or government"
Tom Sawyer - Rush

Re: random files with different date

Dave Kreskowiak20-Dec-07 15:56

Dave Kreskowiak

20-Dec-07 15:56

pmarfleet wrote:
quoting from the MSDN documentation

There's nothing crypto about the GetTempFileName or GetTempPath methods or their results. At a mere 65535 possible filenames, "strong" isn't the word I'd chose.

Perhaps you were looking at the GetRandomFileName[^] method?

A guide to posting questions on CodeProject[^]

Dave Kreskowiak
Microsoft MVP
Visual Developer - Visual Basic
2006, 2007

Re: random files with different date

pmarfleet20-Dec-07 21:16

pmarfleet

20-Dec-07 21:16

Dave Kreskowiak wrote:
Perhaps you were looking at the GetRandomFileName[^] method?

I was referring to the GetRandomFileName method. See my original post[^]. At no point did I refer to GetTempFileName or GetTempPath.

Paul Marfleet

"No, his mind is not for rent
To any God or government"
Tom Sawyer - Rush

WCF ASMX Web Service Client with HTTP Compression woes

martin_hughes19-Dec-07 9:03

martin_hughes

19-Dec-07 9:03

I have spent many hours googling this today, and I'm no closer to finding an answer 9or finding an answer I can understand!)

I have a very simple Web Service conforming to WsiProfiles.BasicProfile1_1. It runs on IIS 7, on which I have enabled dynamic content compression, and I'd like to be able to compress this content when it's to...

... my very simple WCF Service client. I created this by using the wizard and pointing it at my web service. The problem I have is that I (think) need my client to send the "yes, I understand compression" header to IIS, and then to understand the compressed response it gets. However, google as I might I can't fathom how to do this. There was some talk of needing to use a compression channel (and I found this on CodePlex: http://www.codeplex.com/wcfextensions[^]), but I'm not sure this is what I need. But in my app.config, there doesn't seem to be any option to enable compression awareness.

Surely this is a common enough thing to want to do that it's included in the box?

"On one of my cards it said I had to find temperatures lower than -8. The numbers I uncovered were -6 and -7 so I thought I had won, and so did the woman in the shop. But when she scanned the card the machine said I hadn't.

"I phoned Camelot and they fobbed me off with some story that -6 is higher - not lower - than -8 but I'm not having it."
-Tina Farrell, a 23 year old thicky from Levenshulme, Manchester.

Reg:Change Text on DataGridViewLinkColumn

P.T.R.K19-Dec-07 8:37

P.T.R.K

19-Dec-07 8:37

Hi Friends,

I have created a C# .net windows project. There is a form which has used a DataGridViewLinkColumn cell in datagridview. How can i change DataGridViewLinkColumn text when after binding value to datagridview?

Thanks in advance,

Rameshkumar Thirumalaisamy

Simple File Counting

ffowler19-Dec-07 8:20

ffowler

19-Dec-07 8:20

Is there a simple way to get the number of files in a current directory?

Re: Simple File Counting

Giorgi Dalakishvili19-Dec-07 8:24

Giorgi Dalakishvili

19-Dec-07 8:24

Use the Directory class.

#region signature
my articles
#endregion

Re: Simple File Counting

Jimmanuel19-Dec-07 8:24

Jimmanuel

19-Dec-07 8:24

Directory.GetFiles(".").Length

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.