Urgent help on httpwebrequest scrapping

Question

2.00/5 (2 votes)

See more:

, +

help me to crawl website ,scarpping website in asp.net c#,
i am unable to get data while i am trying to scrap website

HttpWebRequest newRequest ;

newRequest = (HttpWebRequest)WebRequest.Create("http://flights.makemytrip.com/makemytrip/search/O/O/E/1/0/0/S/V0/DEL_BOM_14-08-2015?intid=homepage_Widget_Search_New-Delhi_Mumbai");
newRequest.Method = WebRequestMethods.Http.Get;
newRequest.ContentType = "text/html;charset=ISO-8859-1;application/json";

newRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0";
newRequest.Referer = finalUrlRef;
response = (HttpWebResponse)newRequest.GetResponse();
string readerNewRequest = new StreamReader(response.GetResponseStream()).ReadToEnd();

Posted 2-Apr-15 9:00am

Purushottam Kumar

Updated 2-Apr-15 9:04am

v3

Add a Solution

Comments

Sergey Alexandrovich Kryukov 2-Apr-15 16:49pm

And what's the problem? One thing: you are pretending that your request content is JSON, but sending any JSON.
—SA

George Swan 2-Apr-15 17:07pm

The URL string looks incorrect, try :-
"http://flights.makemytrip.com/makemytrip/search/O/O/E/1/0/0/S/V0/DEL_BOM_14-08-2015?intid=homepage_Widget_Search_New-Delhi_Mumbai"

Purushottam Kumar 4-Apr-15 10:11am

text/html;charset=ISO-8859-1;application/json means you are sending any of this type of information to the request and you want any of them. but when i send it i was unable to get page load data in response.

and this url is ok

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Answer 1 · 2015-04-02T10:53:00

Look (or look again) at the code sample doing the same thing: https://msdn.microsoft.com/en-us/library/system.net.webrequest%28v=vs.110%29.aspx.

Find 10 differences. :-)

Or, look at my comment to the question. Why "application/json"? You are not sending any JSON, you are not sending any content at all. Maybe you mix up this content type and content type of expected HttpWebResponse, but why? It makes no sense. If you need that, use response.ContentType:
https://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.contenttype(v=vs.110).aspx,
https://msdn.microsoft.com/en-us/library/system.net.httpwebresponse%28v=vs.110%29.aspx.

—SA