Click here to Skip to main content
15,867,568 members
Please Sign up or sign in to vote.
1.00/5 (3 votes)
See more:
Pop-up windows to accept cockies appiers while web scraping


What I have tried:

Hello together, 
I´m doing web scraping by a program written with C#. Always when opening a website a popup window to accept the cookies appiers. How can I avoid the window to appear. I hope that somebody can help.
Andi
Posted
Updated 26-Aug-21 5:26am
v2
Comments
Richard MacCutchan 26-Aug-21 11:02am    
Unfortunately you cannot avoid them. It became law some time ago in most countries that websites have to provide users the option of accepting or rejecting their use of cookies.
Afzaal Ahmad Zeeshan 26-Aug-21 12:11pm    
You cannot do that. You need to detect the pop-ups and handle them.

I think the cookie pop-ups are a part of the "antibot" system on a web application. ;-)
Member 15337283 30-Aug-21 13:26pm    
The cookie pop-up window is an overlapping iframe with an "accept" button. Unfortunately I don´t know how to handle it.

1 solution

Simple. You don't.

There's nothing special about the cookie pop-up "window". It's just part of the page you're downloading, which you cannot separate out into their "window" parts.
 
Share this answer
 
Comments
Member 15337283 21-Sep-21 10:40am    
Is it possible to save all cookies in a container so that the window may don´t appier?
Dave Kreskowiak 21-Sep-21 10:56am    
You're not listening. This has nothing to do with the actual cookies. This is a window that is part of the page getting a response to a question. That response in put in the cookies so the question does not pop up again.

No, "save all cookies in a container" is not a thing. The browser handles all cookies, not your scraping code.
Member 15337283 22-Sep-21 4:21am    
And how can I deal with it? Is there a way to change the Firefox settings? I've tried the following, but it doesn't work. The cookie window still appears:

fo = new FirefoxOptions();
fo.SetPreference("network.cookie.cookieBehavior", 0);
Dave Kreskowiak 22-Sep-21 10:24am    
You don't get it. This isn't a browser thing or a storage thing. This little window is part of the page you're loading, just like a table or an image or whatever else is in the page.

You have to "click" whatever controls either set an option in that popup or an Accept button, or whatever its showing you.
Member 15337283 24-Sep-21 10:57am    
What I don't understand is why the popup window only appears in the automatic browser and not in the non-automatic browser when I manually open the link.

How can I click for example the button in the cookies pop-up window on www.finanzen.net
I usually choose the items with

driver.FindElement(By.Id(idName)) or
driver.FindElements(By.CssSelector("." + ClassName))[ClassNo] or
Node.FindElements(By.TagName(TagName))[TagNo]

But in this case it doesn't work.
Can you please give an example how to do it?

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900