Scraping secured sites

Question

5.00/5 (1 vote)

See more:

I use WebRequest() & WebResponse() methods from C# for scraping data from sites. It works well for unauthenticated sites. Now I want to scrape details from an authenticated sites which prompts for login details for getting into the site. The site is 'https://www.dandh.ca/v4/view?pageReq=dhMainE'. How do I pass the credentials while requesting the page?

I used the below code,but it doesn't work:

NetworkCredential credential = new NetworkCredential("XXXX", "XXXX"); //USERNAME,PWD
         HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(target_url);
wr.Credentials = credential;
wr.Method = "POST";

HttpWebResponse wrs = (HttpWebResponse)wr.GetResponse();
StreamReader sr = new StreamReader(wrs.GetResponseStream(), Encoding.GetEncoding("ISO-8859-1"), false);
         source = sr.ReadToEnd();

I also tried setting 'DefaultCredentials=true' and keeps the site opened in my firefox browser.

Please help me to get my problem solved.

Thanks & Regards,
Lavanya.

Posted 4-Oct-10 23:35pm

Krishna_2010

Updated 4-Oct-10 23:49pm

LittleYellowBird

v2

Add a Solution

Comments

E.F. Nijboer 5-Oct-10 14:48pm

Couldn't you simply check the "auto-login" checkbox? You would otherwise have to simulate the login procedure (the part within the loginform div)

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Pavel Yermalovich · Answer 1 · 2011-03-21T22:06:00

I was working with this issue some period of time. So, first of all you should analize all kind of requests in some http analyzer and track the behaviour of required. The most difficult level is to implement an authorization because it can be work differently for every site. As usual it's connected with cookies where the login data is stored after logging in. You need to repeat this technique in your code and then you will manage to access the page.

Sending login data can be implemented in different ways too. They are query string and post parameters not just using ready-to-use NetworkCredential object.

Before to implement scraping try to look for developer API for site cause it pretty simplifies the developing.