How to scrape a website that requires login using htmlagility pack?

Question

1.00/5 (1 vote)

See more:

I want to scrape some data from a website. the website requires to login first. I can't do it in code with the help of htmlagilitypack. I also tried to do this with webbrowsercontrol, but it doesn't work properly. Please, help me.

What I have tried:

I tried to login with htmlagility pack but couldn't do it. also I have search google extensively, couldn't get specific answer.

p.s. If any website that doesn't require login then htmlagilitypack works fine.

Posted 20-Apr-22 0:58am

hemal p.shah

Updated 20-Apr-22 12:45pm

Add a Solution

Comments

Richard MacCutchan 20-Apr-22 7:51am

If the website requires a login then that is to protect it from webscrapers.

Graeme_Grant 20-Apr-22 18:48pm

I believed the same until a couple of days ago when I read this: Web scraping is legal, US appeals court reaffirms – TechCrunch[^]

Dave Kreskowiak 20-Apr-22 21:26pm

It may be legal but that in no way means some company will allow you to do it. You still have to face the terms of service from the company, which will prevent you from doing it.

Graeme_Grant 20-Apr-22 21:28pm

I'm not saying that I agree with it but this sets a precedence and overrides any terms of service. Companies should be concerned.

Dave Kreskowiak 20-Apr-22 21:30pm

No, it does not override Terms of Service. Think about it, if the company has proprietary data that only they generate and they keep it behind a subscription, does the law give you the right to walk into the company and demand the data for free? NO IT DOESN'T!

Graeme_Grant 20-Apr-22 21:42pm

Legal precedence does actially overide any legal agreement when challenged in a court of law.

As for demanding data, there is already laws in place where you can obtain your own data held by any company.

Dave Kreskowiak 20-Apr-22 21:43pm

So you're saying that all you have to do is sue some company and you get access to their data for free?

No, that's not how it works.

Graeme_Grant 20-Apr-22 21:45pm

Read the article. It refers to public data displayed in the browser. ie: web scraping. Copywrite laws still apply.

Dave Kreskowiak 20-Apr-22 21:49pm

I know what the article says, and even though it's available, you don't get unlimited access to it. Terms and conditions apply.

If you have an application that thousands of users have, can they all use that application to scrape unlimited data from some company site? No. That company has every right to regulate access to its data, even if it's publicly available. Companies don't have unlimited bandwidth to supply your customers with their data, so they do have the right to regulate that access.

Richard MacCutchan 21-Apr-22 3:43am

Yes, but private data can still be protected.

Graeme_Grant 21-Apr-22 5:47am

As the article states in the first paragraph: "Scraping publicly accessible data is legal, according to a U.S. appeals court ruling"

Richard MacCutchan 21-Apr-22 6:20am

That may be so, but you can only scrape what the website makes available. And there is nothing in the law that says that websites are not allowed to enforce logins.

Graeme_Grant 21-Apr-22 7:21am

As stated earlier, I'm not saying that I agree with it.

However, the door has now officially been opened though & website Ts&Cs on web scraping now do not have the legal weight that they use to.

Richard MacCutchan 21-Apr-22 7:24am

Quote: In its second ruling on Monday, the Ninth Circuit reaffirmed its original decision and found that scraping data that is publicly accessible on the internet is not a violation of the Computer Fraud and Abuse Act, or CFAA, which governs what constitutes computer hacking under U.S. law.

gggustafson 21-Apr-22 14:10pm

The story cited reads

In its second ruling on Monday, the Ninth Circuit reaffirmed its original decision and found that scraping data that is publicly accessible on the internet is not a violation of the Computer Fraud and Abuse Act, or CFAA, which governs what constitutes computer hacking under U.S. law.

The key phrase is "data that is publicly accessible;" data that is protected by a login is not publicly accessible!

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Graeme_Grant · Answer 1 · 2022-04-20T12:45:00

Solution 1

I did a quick Google Search using: htmlagility pack credentials[^]

Found several solutions, including this one: [SOLVED] => HtmlAgilityPack and Authentication[^]

If this is not suitable, then there are several other solutions using the search above.

Posted 20-Apr-22 12:45pm

Graeme_Grant