Click here to Skip to main content
15,910,797 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I am crawling a website.
The url that i am crawling is "http://www.dealsandyou.com/catalog/product/citysitedeal/productId/8060".
This url shows ahmedabad links when opened in browser. ie. link to first product BlackBerry Smartphones is "http://www.dealsandyou.com/blackberry-smartphones-ahmedabad-26092011.html". But when u read this url in asp.net it always gives data of other cities as "http://www.dealsandyou.com/bluedio-stereo-i4-headset-othercities-29092011.html". That is the products for other cities. I cant understand y is it show. Please help me & give me some solution for this.
My source code is somewhat like :
C#
 private static HttpWebRequest _request;
 private static HttpWebResponse response;
 StreamReader strreader = null;

 _request = (HttpWebRequest)WebRequest.Create("http://www.dealsandyou.com/catalog /product/citysitedeal/productId/8060");
            
 response = (HttpWebResponse)_request.GetResponse();

             
 //create a streamreader object from the response 
 strreader = new StreamReader(response.GetResponseStream());

//reading data in string. This string always give data of other cities
 string strdata = strreader.ReadToEnd();

You can also check for "http://www.dealsandyou.com/catalog/product/citysitedeal/productId/8085". That is the url for banglore products. It will also show other cities data when crawled.

Please help me. Its very urgent.
Posted
Updated 30-Sep-11 1:46am
v3
Comments
Muralikrishna8811 30-Sep-11 7:49am    
Nice editing.
developers must know words to be used

1 solution

Using a tool like Fiddler2[^] you'll have to analyze what additional information (headers) is sent in the web request when it's done via browser. When generating this web request in your program you'll have to make sure that this additional information is also sent to the server you're requesting the information from.

See here on how to add HTTP headers to your web request: http://msdn.microsoft.com/en-us/library/system.net.webrequest.headers.aspx[^].

Regards,

—MRB
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900