Click here to Skip to main content
15,895,011 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
What can be Efficient Way to Crawl through Web Sites
I have two different codes, which would be better & in what respect ?
Or can there be more better code tom crawl through webpages ??

Code 1 :
private static string GetWebTest1(string url)
        {
            System.Net.WebClient Client = new WebClient();
            return Client.DownloadString(url);
        }


In Comparism with :

Code 2 :
private static string GetWebTest2(string url)
       {
           HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
           WebResponse response = request.GetResponse();
           Stream stream = response.GetResponseStream();
           StreamReader reader = new StreamReader(stream);
           return reader.ReadToEnd();
       }


Which Can be better ? & in What respect ( time consuption,handling error,etc. )
Posted
Updated 16-Jun-10 20:35pm
v2

You have more control over what's happening with method #2. That's the one I would choose. However, don't forget to add the requisite try/catch/finally block to handle any exceptions that might occur.
 
Share this answer
 
Comments
jpratik 17-Jun-10 12:26pm    
In Method #2. I might have more control.But my need is "which one is faster?"
As I need to go through this code again & again with different URL's some which fetched from the downloaded webpages also.
It would be like on this current page, there are 'n' 'web page links' which again are passed to same function and the process would continue.. for large number of pages.
So At that time,what method would be more time saving ?
I would be adding Try..Catch for Handling Exception.
#realJSOP 17-Jun-10 13:37pm    
Speed differences are negligible because your bottleneck is the speed of your internet connection. If it were me, I'd pick the way that gives me the most control of what's happening.
jpratik 22-Jun-10 1:29am    
ok Thanks for your advice.
hi friend,

According to me the Code 2 is better, because you can get all links resides in page easily by the reader,

For collect the link from reader use the regular expression and Match class in asp.net.

Yap if your are going towards performance the Code 1# is the best.

Accept this answer if you are satisfy.

Thanks,

Mahesh Patel
 
Share this answer
 
v2
Comments
jpratik 17-Jun-10 12:27pm    
In Method #2. I might have more control.But my need is "which one is faster?"
As I need to go through this code again & again with different URL's some which fetched from the downloaded webpages also.
It would be like on this current page, there are 'n' 'web page links' which again are passed to same function and the process would continue.. for large number of pages.
So At that time,what method would be more time saving ?
I would be adding Try..Catch for Handling Exception

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900