Click here to Skip to main content
15,890,282 members
Please Sign up or sign in to vote.
2.00/5 (1 vote)
Hi. I have very big problem please help.

I'm scrapping website and the page don't have pagination. If you like to go to the next page you need to click on form button or select from the select option menu. With webbrowser control I'm clicking on the button or the select option like this:

HtmlElement reportDropDown = webBrowser1.Document.GetElementById("ctl00_ContentPlaceHolderProductDisplay_wuc_product_display_ddlTopPage");
reportDropDown.SetAttribute("value", Convert.ToString(c));
reportDropDown.InvokeMember("onchange");


The problem is: when I scrap the first page, and the button is clicked, after clicking the button the page have some little animation with loading on the same page, while go to the next page. When the button is clicked and the loading animation is start, my code in the while loop don't wait, so start scrapping again the same results from the first page.

I've tried webbrowser1.readystate, isbusy or similar, they not working for button!!!!
I need waiting mechanism when button is clicked!!!

little demo of my while loop:

while(i <= 10) // number of pages
{
while(...)
{
//here i'm scraping data for the current page
}
// here goes button click
++i;
// here i need some mechanism for waiting
}
Posted
Updated 22-Oct-13 18:18pm
v2
Comments
Sergey Alexandrovich Kryukov 23-Oct-13 0:47am    
How is that related to Web scraping?
—SA

1 solution

Why would you scrape Web sites with the help of a Web browser? This is some client UI, and Web scrapping has nothing to do with it. Please see:
http://en.wikipedia.org/wiki/Web_scraping[^].

For scrapping, the main thing you need is the class HttpWebRequest. For further detail, please see my past answers:
get specific data from web page[^],
How to get the data from another site[^].

—SA
 
Share this answer
 
v2
Comments
Ensiferum77 23-Oct-13 11:30am    
I need help. If you know how to help, then help. Webscrapping can be describe in different terms and I don't post the question to speak about whaat is and what is not. I use webbrowser everytime for scrapping, best solution for me. so please help if you know how you will solve javascript page, my code to wait for sometime, after button is clicked.
Sergey Alexandrovich Kryukov 23-Oct-13 12:21pm    
I just gave you all the help you need, did you noticed that?! Having that in mind "if you know how to help, then help" is simply rude and ignorant.

What you say makes no sense at all. Who care what other mistakes did you do in your life. If you fail to listen and understand, helping you will be totally useless. We all want to help only to people who are capable of using help and doing appropriate work, as no one likes wasting time.

—SA
Ensiferum77 23-Oct-13 17:24pm    
I only need help from someone on the forum if recently or in the past who have problem like mine. The solution you provide won't help me. This is not my first time to extract data with browser control. Maybe is not the fastest one, but the job do very well. so if you have experience with webbrowser control and button click, navigation etc, help otherwise there are thousand of forums maybe found help there somewhere. cheers
Sergey Alexandrovich Kryukov 23-Oct-13 18:26pm    
Okay, I gave you the complete solution and tell you that you are doing wrong thing. You decide.
—SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900