Click here to Skip to main content
15,892,059 members
Please Sign up or sign in to vote.
5.00/5 (1 vote)
See more:
Hi!

The problem:
I want to scrap some data from certain webpage (I have administrative access) and to store some information in db for later analysis.
Sounds easy, right?
I've decided to make simple console prototype and code look something like this:
C#
string uri =  @"http://s7.iqstreaming.com:8044/admin.cgi";
HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
                     
if(request == null)
{
     Console.WriteLine(":( This shouldn't happen!");
     Console.ReadKey();
}

request.ContentType = @"text/html";
request.Accept = @"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Credentials = new NetworkCredential("myID", "myPass");

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
     StreamReader reader = new StreamReader( response.GetResponseStream());

     while (!reader.EndOfStream)
     {
         Console.WriteLine(reader.ReadLine());
     }

     reader.Close();
     response.Close();
}


This code works on most other sites, but here I get errors 404 (most of the time), 502 or timeout.
I've consulted with Firebug (I've took Accept and compression info from there) but to no avail.
Using Win-forms and webBrowser control as an alternative is not an option (at least for now).

P.S.
Same thing happens when I try to get HTML from http://s7.iqstreaming.com:8044/index.html (doesn't need credentials).

Edit: added P.S.
Posted
Updated 12-Mar-12 0:25am
v2

1 solution

I've posted this question on other popular web site for coders and one of members there knew about this issue.
Answer seams very simple (as usual), problem was in UserAgent!

When I add this line of code everything works as expected.
C#
request.UserAgent="Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.78 Safari/535.11";
 
Share this answer
 
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900