|
hi all,
i have a multithread program that is used to parse certain xml file and then insert it into database and delete. the xml files comes continuously so i have a timer on every instance of which i check the directory so for each file in a directory i create a theread.
it all works fine as long a i get file of size 1kb but if the file gets larger then it takes time to inset intodatabse and hence to delete it so in another instance of time interval the same file gets repeated. the file can be of larger size up to 5mb now if i disable timer and then process it every things work fine but then other small file also gets stoped as larger files is being processed.
now what shall i do so that i may not have to disable timer and file doesnt get repeted.
public void CallOnCreated()
{
DirectoryInfo dirInfo = new DirectoryInfo(@"C:/datahandler1");
fInfo = dirInfo.GetFiles("*.xml");
threads= new Thread[fInfo.Length];
sFileNames=new string[fInfo.Length];
if(fInfo.Length!=0)
{
for(int i=0;i
|
|
|
|
|
you could rename the files to NewGuid() +".xml" and then act on them. If you control the code that puts them there add some uniqueness to the filename so that things don't get overwritten.
you could also load them into a queue such as MSMQ and act on them later.
HTH
Russ
|
|
|
|
|
It is very simple. First of all, use try catch finally block to handle exceptions. This is important if you want to use the solution suggested by me.
In the CallOnCreated function, stop the timer at first in the try block. Then your current code will be there. Then in finally block start the timer again. this is as simple as it sounds.
Now your code will look like this --
public void CallOnCreated()
{
try
{
<code>Timer1.Stop();</code>
DirectoryInfo dirInfo = new DirectoryInfo(@"C:/datahandler1");
fInfo = dirInfo.GetFiles("*.xml");
threads= new Thread[fInfo.Length];
sFileNames=new string[fInfo.Length];
if(fInfo.Length!=0)
{
for(int i=0;i
{
if(pics.BinarySearch(fInfo[i].FullName)<0)
{
pics.Add(fInfo[i].FullName);
}
}
foreach(string file in pics)
{
Console.WriteLine(file.ToString());
myFile myf= new myFile(file);
if(myf.isUsed())
{
continue;
}
try
{
clsCall objcall= new clsCall(file);
threads[i]= new Thread(new ThreadStart(objcall.getCallClass));
threads[i].Name="thread is " + sFileNames[i];
threads[i].Start();
}
catch
{
}
finally
{
<code>Timer1.Start();</code>
}
}
"A good programmer is someone who looks both ways before crossing a one-way street." -- Doug Linder
coolestCoder
|
|
|
|
|
I have a function which recursively reads the file names of all the MP3 files on a drive into a text array. The function works well as long as there is less than 16,000 files. On a drive containing 19,700 files the array stops being built at around 16,900 files. I have run the program on several machines with varied amount of ram and the array always stops at about 16,900. Is there some limitation on the size of an array in C# of which I am unaware.
|
|
|
|
|
Nope - I just ran the following test against an array list.
ArrayList list = new ArrayList();
for (int i = 0; i< 19000; i++)
{
list.Add(i);
}
Console.WriteLine("{0}", list.Count);
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
Could it be the the length of the string being stored is a factor?
|
|
|
|
|
I wouldn't have thought so. I changed the test to this:
<code lang="c#">string item = @"c:\documents and settings\MyDocuments\ReallyLongstring\" +
"This is a very long path\This is a really long path\" +
"It couldn't get any longer\*.mp3";
ArrayList list = new ArrayList();
for (int i = 0; i < 19000; i++)
{
list.Add(item);
}
Console.WriteLine(list.Count);</code>
and it still works.
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
Yeah! I tried something similar with a string of 250 chars and it worked with no problem. Im at a loss to what's happening in the program.
|
|
|
|
|
This is the function that is writing the array. It writes to a file the complete list but aborts at 16900 when writing to an array.
void DirSearch(string sDir)
{
try
{
foreach (string d in Directory.GetDirectories(sDir))
{
foreach (string f in Directory.GetFiles(d, "*.mp3"))
{
listBox1.Items.Add(f);
arrText.Add(f);
}
DirSearch(d);
}
}
catch (System.Exception excpt)
{
Console.WriteLine(excpt.Message);
}
}
By the way how do you get your code snippets to appear as they do in your messages?
I am an old timer that started with Cobol and a lot of this is new to me.
|
|
|
|
|
I'll have a quick try with this. I got the code snippets to appear the way that they do by using the pre and code tags around them. If you look at the formatting section below the signature, you will see it. If you highlight your code and click those tags you end up with something like this:
<pre><code>
public void MyCode()
</code></pre>
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
|
You don't need to nest tags like that. <Pre> does the code coloring as well as the block layout. The <code> tag is intended for inserting a big of code in the middle of a sentence.
--
Rules of thumb should not be taken for the whole hand.
|
|
|
|
|
electriac wrote: listBox1.Items.Add(f);
I can almost guarantee you this is the offending line, trying to add a huge number of items to a list box (or any UI element for that matter) will slow down to an almost stop. There is no absolute number where this will happen but a listbox slows down more and more as you ad items.
UI controls have a method SuspendLayout and ResumeLayout that will help with this problem, but you should really consider your UI design & useability if you are adding that many items to a listbox.
|
|
|
|
|
I have removed this from the directory search and built the listBox from the array after the array has been built and it seems to make no difference.
|
|
|
|
|
Did you actually read my post?
|
|
|
|
|
You put your finger on the problem
TNX
|
|
|
|
|
I want to thank those that made suggestions on my problem. It seems that J4amie put his finger on the problem where I populated the list box while building the array. I have removed the listBox1.Items.Add from the directory search and all the files are in the array. I have also built an array of 40,000 files of 250 chars and put the array in a list box just to test an extreme example and it worked without problem. Again many thanks from an old guy that started with punch cards and vacume tubes.
|
|
|
|
|
Well - this works for me:
private void button1_Click(object sender, System.EventArgs e)
{
DirSearch(@"c:\");
MessageBox.Show(icount.ToString());
}
private ArrayList arrText = new ArrayList();
private int icount = 0;
private void DirSearch(string dir)
{
if (++icount >= 19000)
return;
foreach (string d in Directory.GetDirectories(dir))
{
foreach (string f in Directory.GetFiles(d, "*.*"))
{
listBox1.Items.Add(f);
arrText.Add(f);
if (++icount >= 19000)
return;
}
DirSearch(d);
}
}
When you run your code, what does it write to the console window when it gets it's exception?
the last thing I want to see is some pasty-faced geek with skin so pale that it's almost translucent trying to bump parts with a partner - John Simmons / outlaw programmer
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
Pete many thanks for your instructive detective work. Although you did not find the offending line your methods were very instructive. TNX again.
|
|
|
|
|
I have tested this further and it gets curious-er. I installed a drive that has nothing but MP3 files on it. I went in a command com and issued a “Dir *.mp3 /b /s > test.txt” to create a list of all the files on the drive. I read the file with a text editor and there are 17,247 files listed. Okay I run the program I have been testing and it finds only 16800 files. Here is the code I used to which I introduced a counter with which I report to a message box in order to confirm the exact number in the array. The number in the array agrees with countFiles.
<br />
void DirSearch(string sDir)<br />
{<br />
try<br />
{<br />
foreach (string d in Directory.GetDirectories(sDir))<br />
{<br />
foreach (string f in Directory.GetFiles(d, "*.mp3"))<br />
{<br />
arrText.Add(f);<br />
countFiles++;<br />
}<br />
DirSearch(d);<br />
}<br />
}<br />
catch (System.Exception excpt)<br />
{<br />
Console.WriteLine(excpt.Message);<br />
}<br />
}<br />
Upon examintion of the array in the list box there is also 16,800 files in the list box but I notice that I have a directory for “World” music which is not present in the list. Out of curiosity I change the name of the directory from “World” to “aWorld” and run the program without any other changes. The program reports 17,005 files which reflects the approximate number of files in the “World” directory.
It seems that there is something strange happening in DirSearch that is beyond my feeble mind. Since it will not read anything alphabeticaly beyond “Rock and Roll” could there be some kind of musical prejudice at work? That’s a joke.
|
|
|
|
|
Hi,
I tried your code on .NET 1.1 and it works for me, with some comments:
- I changed *.mp3 into *.*
- I have an arrText that is not touched by another thread;
- I have a listBox1 that is not touched by another thread, and NOT added to the Controls
of some window
- In a second run I added a log line which sends the filename f to my standard log mechanism,
consistsing of logging in a file AND logging in a listbox that is visible on my main form.
I looked for all files on a single partition and managed to get all 37000 of them. So there
seems to be no fundamental limitation that you are hitting.
I guess your trouble is with listBox1, everything else is hard to suspect.
If you run your code in the UI thread, I expect in to work properly (albeit freezing
the UI for a couple of minutes); if you run it in another thread, then:
- on .NET 2.0 it will fail immediately (the cross-thread exception kicks in)
- on .NET 1.1 it will behave unreliably (for the same reason, but not explicitly tested
by the framework).
So I suspect you are using .NET 1.1 and forgot to apply the InvokeRequired & Invoke stuff
that has raised so many messages already on CodeProject...
Can you verify this ?
Luc Pattyn
|
|
|
|
|
Im using Vis 2005 and framework 2.0
|
|
|
|
|
So that tells me you are running your code on the UI thread, and if it hangs, it means
something else in your app does not survive the UI thread to be non-reponsive that long.
IMHO adding SuspendLayout/ResumeLayout (which is basically a performance optimalization)
will speed things up, but does not fundamentally solve your problem. I suspect
your app will still fail once a (much) larger number of mp3 files is present.
You could test this by inserting a Thread.Sleep(several seconds) where you now call DirSearch.
regards,
Luc Pattyn
|
|
|
|
|
Had no problem with the program hanging and there was no error report. Its just that all the files on the drive were not in the array. Removing the listBox1.Items.Add from the directoy search seems to have solved the problem. I just tried it on a drive with 34500 mp3 files and it worked fine.
|
|
|
|
|
You can try this:
List<string> my = new List<string>();
for (int i=0;i<200000;i++)
my.Add("a");
Console.WriteLine("There are {0} elements to my list",my.Count.ToString());
Console.ReadKey();
The result is: 200000
I hope that help you  
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
