Click here to Skip to main content
15,867,594 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
This program downloads images, now it keeps using more and more RAM till I get an "Out of memory exception" at 3.2 GB, how to make it release the RAM, I thought by having: "using" the WebClient is going to release the resources by the end of the using brackets and the garbage collector is going to collect it at some time.

One more question especially if you can't answer the first:
I have 32 GB RAM and 390 GB free in a 2 TB SSD, the "Out of memory" exception is thrown before even the RAM is full, so how do I make the program fill the whole RAM as it wants and even then use the SSD as RAM?

What I have tried:

Regex r = new Regex(regexString);
MatchCollection mc = r.Matches(text);
int i= 1;
foreach (Match m in mc)
{
    using (WebClient wc = new WebClient())
    {
       try
       {
          wc.DownloadFileAsync(new Uri(m.Value), @"g:\0test\" + (i++).ToString() + ".jpg");
       } 
       catch
       {
          wc.DownloadFileAsync(new Uri(m.Value), @"g:\0test\" + i.ToString() + ".jpg");
       }
    }
}



This is the error code:

C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.pdb: Cannot find or open the PDB file.
C:\WINDOWS\System.pdb: Cannot find or open the PDB file.
C:\WINDOWS\symbols\dll\System.pdb: Cannot find or open the PDB file.
C:\WINDOWS\dll\System.pdb: Cannot find or open the PDB file.
Posted
Updated 10-May-21 16:29pm
v10
Comments
Richard MacCutchan 10-May-21 4:42am    
Your catch statement is ignoring any exceptions, and may even cause a new one.

And those error message are merely telling you that the debugger cannot find debug information for some system libraries.
Alan N 10-May-21 9:15am    
To get any kind of answer you will have to provide code that makes sense. Just glancing at the current version of your question I wonder
1) What is the loop doing given that it does not use 'm' and we don't know what 's' contains.
2) Why when an exception occurs do you just try the same thing again.
3) Why dispose of the WebClient object when an async operation is still in progress.
4) Why is there no DownloadFileCompleted event handler
john1990_1 10-May-21 9:51am    
4-> DownloadFileCompleted event handler isn't firing, it's a Microsoft problem as I understood from a Google search.
john1990_1 10-May-21 9:53am    
1-> Well it's a simplified version of my code, I will edit it to make it include "m".
john1990_1 10-May-21 9:55am    
2-> I try to download the file again as when downloading again the download might succeed.

You can get OutOfMemory exceptions if you run the machine out of handles or other resources. It's not limited to just RAM.

The thing is, how many "matches" are you getting? How many simultaneous DownloadFileAsync calls are running?

Also, That empty catch block isn't doing you any favors. If there's a problem, it's swallowing the exception so you don't know about it. Get rid of the try/catch block until the problem is solved.
 
Share this answer
 
Comments
john1990_1 10-May-21 1:12am    
The catch block retries to download async again, but I removed this for simplicity and didn't realize that it's useless here, I get 30 matches and wait 2.15 seconds for the next page, the pics get downloaded, DownloadFileCompleted event isn't firing so I know how many are getting downloaded and a Google search showed it's a Microsoft problem or something, but the pics get downloaded so I know there aren't that much active downloads.
john1990_1 10-May-21 1:46am    
I updated the question with the filled catch statement, in fact I have another catch in it and so on around 5 times...
Dave Kreskowiak 10-May-21 10:34am    
Uhhhh... yeah, that's the very wrong way to do a retry operation. A catch block should NEVER be used to do the same operation over again.

A more correct way to do it would be to create a queue of operations that need to be done, and if one fails, put the item that was missed back into the queue. You might also want to keep track of how many times you tried to attempted that operation before giving up.

The catch block should always log the exception and which file was being attempted.

Oh, and what you're doing with the (i++) is a flaw in your logic. The value of i is incremented after the DownloadSync is called, so when it fails and you end up in the catch block, you're using the new value of i, not the value of i that was used in the original statement.

What you put as "the error code" isn't the error at all. Those messages are just the debugger telling you it cannot find the Program Debug Database files for those modules.
Dave Kreskowiak 10-May-21 10:44am    
The only thing I can find on a possible fix is to call CancelAsync on your WebClient object when it's done.

The other solution is to use System.Net.Http.HttpClient instead of WebClient.
john1990_1 10-May-21 12:49pm    
Do you know how to allow my program to fill all RAM and then use the 490 GB that are free in my SSD as RAM?
You need to do a better job monitoring your background workers.

You're "firing and forgetting": no progress reporting; no completion reporting.

The "workers" could all be hanging for any number of reasons, including too many threads ("millions", in your words).
 
Share this answer
 
This code seems to have worked for me.

We would have to wait for humankind to get super smarter so we don't get a programming machine that doesn't dispose its object with "using() {}" as above.

//this will be called literally a million times with waiting in between (s is url for a jpg or a png file), waiting of 2.15 seconds to the next page of 30 pics in a page
        void myDownloadFileAsyn(string s)
        {
            while(currentNumberOfDownloads>30)//31 simultaneous downloads
            {
              System.Threading.Thread.Sleep(350);
            }


            BackgroundWorker bgw = new BackgroundWorker();
            bgw.DoWork += Bgw_DoWork;
            bgw.RunWorkerCompleted += Bgw_RunWorkerCompleted;

            bgw.RunWorkerAsync(argument:s);
        }

        private void Bgw_DoWork(object sender, DoWorkEventArgs e)
        {
            string s = e.Argument as string;
            WebClient wc = new WebClient();

            currentNumberOfDownloads++;
            wc.DownloadFile(s, @"g:\folder\" + s.Substring(s.LastIndexOf(@"\")+1));
            currentNumberOfDownloads--;

            wc.Dispose();
        }

        private void Bgw_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            (sender as BackgroundWorker).Dispose();
        }

int currentNumberOfDownloads = 0;
 
Share this answer
 
v5

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900