how do i scrape ajax page

Question

4.00/5 (1 vote)

See more:

i need to scrape a website that uses ajax. the website searches for listing, but uses ajax to return the result.

the result are returned progressively, not all at once.

how do i wait using httpwebrequest for the search to finish before returning the html result?

thanks

Posted 12-Sep-11 11:27am

Cool Smith

Add a Solution

Comments

Sergey Alexandrovich Kryukov 12-Sep-11 22:04pm

Yes, would not be easy, I guess...
--SA

BillWoodruff 13-Sep-11 11:37am

Why not get started by examining some of the many "web scraping" articles and posts on CP: http://www.codeproject.com/search.aspx?q=web+scraping&doctypeid=1%3b2%3b3&sort=createddesc

AspDotNetDev 23-Sep-11 16:14pm

Do any of them deal with AJAX? Usually, they just deal with a simple web request to get HTML.

2 solutions

Solution 2

I like John's idea, but if that doesn't work (perhaps because of some security feature) you can load the page in a WebBrowser control, wait a few seconds, then inject some JavaScript into the page (via the WebBrowser) that crawls the DOM.

Posted 23-Sep-11 10:13am

AspDotNetDev

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

John Y. · Accepted Answer · 2011-09-23T09:43:00

I don't believe there will be a way to get it by waiting. Because AJAX is used, its more than likely adding to the DOM, which you can't "wait" for, because this happens AFTER the OK response is sent back to your httpwebrequest. However....

I would try to find WHERE in the java script of the page you are trying to scrape it goes and gets its data. There has to be some sort of URL that the AJAX method POSTS to. Find that, and then scrape THAT page (for a certain search term), and then you'll have your information. Probably in XML/JSON or something like that, but atleast it'll be something.