How to scrape web table and display it in VB.NET application?

Question

1.00/5 (2 votes)

See more:

I'm trying to grab a data table from a webpage and display it in a vb.NET application.

While doing some research I found out that the HTML Agility Pack contains features that will allow to easily scrape/extract data from a webpage.

The pack does not contain any documentation and I can't seem to find the right solution on the internet. Hope someone could push me in the right direction.

This is the table I'm trying to pull.

http://testapplications.net16.net/test.html[^]

This is what I found while browsing the net and trying different methods:

C#

' Load the html document
        Dim web As New HtmlWeb()
        Dim doc As HtmlDocument = web.Load("http://testapplications.net16.net/test.html")

        ' Get all tables in the document
        Dim tables As HtmlNodeCollection = doc.DocumentNode.SelectNodes("/table")

        ' Iterate all rows in the first table
        Dim rows As HtmlNodeCollection = tables(0).SelectNodes("./tr")
        For i As Integer = 0 To rows.Count - 1

            ' Iterate all columns in this row
            Dim cols As HtmlNodeCollection = rows(i).SelectNodes("./td")
            For j As Integer = 0 To cols.Count - 1

                ' Get the value of the column and print it
                Dim value As String = cols(j).InnerText
                MessageBox.Show(value)
            Next
        Next

I get a NullReference error at the following line:

Dim rows As HtmlNodeCollection = tables(0).SelectNodes("./tr")

Object reference not set to an instance of an object.

I'm guessing this means that TR can't be found within the code of the table?

What I have tried:

Browsed the web for documentation or solutions for my problem, couldn't find any articles or tutorials.

Posted 13-Apr-16 11:00am

JordyS

Updated 19-May-17 23:22pm

v3

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sinisa Hajnal · Answer 1 · 2016-04-13T21:01:00

Solution 1

There are plenty of articles, some here, some on other sites...here is one to get you started. Next time, search harder. Most examples use C#, but it is easily converted to VB.NET.

Web scraping with progress bar[^]

Posted 13-Apr-16 21:01pm

Sinisa Hajnal

Member 11981296 · Answer 2 · 2017-05-19T23:22:00

Try this one:

<pre> Dim web As New HtmlWeb()
        Dim doc As HtmlDocument = web.Load("https://www.w3schools.com/html/html_tables.asp")

        ' Get all tables in the document
        Dim tables As HtmlNodeCollection = doc.DocumentNode.SelectNodes("//table[@id='customers']")

        ' Iterate all rows in the first table
        Dim rows As HtmlNodeCollection = tables(0).SelectNodes("//tr")
        For i As Integer = 0 To rows.Count - 1

            ' Iterate all columns in this row
            Dim cols As HtmlNodeCollection = rows(i).SelectNodes("//td")
            For j As Integer = 0 To cols.Count - 1

                ' Get the value of the column and print it
                Dim value As String = cols(j).InnerText
                MessageBox.Show(value)
            Next
        Next