Using the Response Data for html agility pack

Question

0.00/5 (No votes)

See more:

C#

string entry = bookEntry.Text;
entry.Replace(" ", "+");
// used on each read operation
byte[] buf = new byte[8192];
// prepare the web page we will be asking for
string url = null;

if (string.IsNullOrEmpty(isbnCode.Text))
{
   string urlTitle =
                    String.Format(
                    "http://www.abebooks.com/servlet/SearchResults?bi=0&bx=off&ds=30&recentlyadded=all&sortby=2&sts=t&tn={0}&x=76&y=18",
                    entry);
   url = urlTitle;
}
else if (string.IsNullOrEmpty(entry))
{
   string urlIsbn =
                    String.Format(
                    "http://www.abebooks.com/servlet/SearchResults?bi=0&bx=off&ds=30&isbn={0}&recentlyadded=all&sortby=2&sts=t&x=46&y=11",
                isbnCode.Text);
   url = urlIsbn;
}

// prepare request
CookieContainer cookies = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = null;

// set header values
request.Method = "GET";
request.CookieContainer = cookies;

// get response data
response = (HttpWebResponse)request.GetResponse();
request = null;

string responseData = new StreamReader(response.GetResponseStream()).ReadToEnd();
response.Close();

// show results
Response.Write(responseData);

HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
var rootNode = htmlDoc.DocumentNode;
var allBookResults = rootNode.SelectNodes("//div[@class='result-detail']");
string bookPrice = null;
var dataNode = allBookResults.SelectSingleNode(".//div[@Class='item-price']");
var bookPriceNode = dataNode.SelectSingleNode(".//span1");
bookPrice = bookPriceNode.InnerText.Trim();

HI all, happy new year.

The question is how do I make the response data (which is the generated link coming from the search results) be used by the html agility pack. The final result is to get the Price from the search list.

the Page Source of the website is as below:

XML

<div class="result-detail">
<span property ="url" content="http://www.abebooks.com/servlet/BookDetailsPL?bi=11698135711"></span>
<div class="item-price" typeof="Offer" property="offers">
    <span property="price" content="41.95"></span>
    <span property="priceCurrency" content="USD"></span>
    <span property="itemCondition" content="UsedCondition"></span>
    <span property ="availability" content="InStock"></span>

</div>

Thanks!

first code block added and indexation

Posted 1-Jan-14 4:47am

slayasty

Updated 2-Jan-14 4:23am

v4

Add a Solution

Comments

Taha Akhtar 1-Jan-14 11:07am

try below
dataNode.SelectSingleNode("span[@property='price']"); instead of var bookPriceNode = dataNode.SelectSingleNode(".//span1");

slayasty 1-Jan-14 15:18pm

I see. But the problem is that the book results variable is null (im guessing since html agility is not using the generated website

Taha Akhtar 2-Jan-14 6:46am

if you are using the code then it seems's that you are not loading html document. htmlDoc.Load(responseData);

slayasty 2-Jan-14 7:17am

I tried that but now when it is trying to load the responseData its saying illegal character in path

Taha Akhtar 2-Jan-14 10:20am

document. htmlDoc.LoadHtml(responseData);

make sure response data is valid

slayasty 2-Jan-14 10:28am

mm. i tried doing htmlDoc.LoadHtml(url) because that url is the generated one but it gives me URI format are not supported.

The response data i get is the page source but seems to be incomplete as i cannot find the div class item class in the response data. perhaps there is a limit on how much response data can be stored?

Taha Akhtar 2-Jan-14 11:23am

are you able to view response html(responseData), paste here

slayasty 2-Jan-14 13:28pm

well its too big to fit. here is the txt file i pasted:

http://www.mediafire.com/download/gg71q401f0759ui/responsedata.txt

slayasty 1-Jan-14 11:09am

I see. But the problem is that the book results variable is null (im guessing since html agility is not using the generated website

ZurdoDev 1-Jan-14 15:16pm

You need to reply to the comment so that the user is notified instead of adding a new comment to your question.

slayasty 1-Jan-14 15:18pm

oh. ok my bad. Thanks

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

idenizeni · Answer 1 · 2014-01-02T08:18:00

Your first issue is tied to the XPath you use to get the 'item-price' nodes. You have a capital 'C' in your XPath but the HTML from the site has a lower case 'c'. XPath is case-sensitive.

The second issue I see is you didn't set any value to your htmlDoc variable. You create a new instance of it but you never load the html into it.

I re-worked your code, I would recommend not using the var keyword unless you are using it with LINQ. You should always use the correct type for your variables.

Here's your code re-factored. I have tested this and it works.

C#

string bookPrice = string.Empty;
HtmlAgilityPack.HtmlDocument htmlDoc = new HtmlAgilityPack.HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
htmlDoc.LoadHtml(responseData1); // load html            
HtmlAgilityPack.HtmlNode rootNode = htmlDoc.DocumentNode;
HtmlAgilityPack.HtmlNodeCollection allBookResults = rootNode.SelectNodes("//div[@class='result-detail']");

foreach (HtmlAgilityPack.HtmlNode node in allBookResults)
{
    HtmlAgilityPack.HtmlNode dataNode = node.SelectSingleNode("//div[@class='item-price']");

    foreach (HtmlAgilityPack.HtmlNode bookPriceNode in dataNode.ChildNodes)
    {
        bookPrice = bookPriceNode.SelectSingleNode("//span[@property='price']").GetAttributeValue("content",null).ToString();            
    }
}