Click here to Skip to main content
15,923,845 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I figured out how to use scrapysharp in console application. But when I switched over to using a form (for better user interface), the program hands up on this line of code:

C#
WebPage PageResult = Browser.NavigateToPage(new Uri("http://192.168.1.99"));


What I have tried:

Here is the full Console application:

C#
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using ScrapySharp.Core;
using ScrapySharp.Html.Parsing;
using ScrapySharp.Network;
using HtmlAgilityPack;
using ScrapySharp.Extensions;
using ScrapySharp.Html.Forms;

namespace SampleScraperClient
{
    class Program
    {
        static void Main(string[] args)
        {
            // setup the browser
            ScrapingBrowser Browser = new ScrapingBrowser();
            Browser.AllowAutoRedirect = true; // Browser has many settings you can access in setup
            Browser.AllowMetaRedirect = true;
            //go to the home page
            WebPage PageResult = Browser.NavigateToPage(new Uri("http://192.168.1.99/"));
            // get first piece of data, the page title
            HtmlNode Center = PageResult.Html.CssSelect("Center").First();
            string Bobby = Center.InnerText;
            Console.WriteLine(Bobby);
            // find a form and send back data
            //PageWebForm form = PageResult.FindFormById("dataForm");
            // assign values to the form fields

            //Console.ReadKey();
            //Console.WriteLine(Bobby);
            Console.ReadKey();
            Console.WriteLine("this far");

            string test = Bobby;
            string search = "T1";
            int foundPos = findPermutation(test, search);
            Console.WriteLine(foundPos);
            if (foundPos != -1)
            {
                Console.WriteLine(test.Substring(foundPos, search.Length));
                Console.WriteLine(" ");
                
                char[] x = { Bobby[foundPos + 5], Bobby[foundPos + 6], Bobby[foundPos + 7], Bobby[foundPos + 8] };
                string T1 = new string(x);
                Console.WriteLine(T1);
                double j;
                if (double.TryParse(T1, out j))
                {
                    j = (1.8 * j) + 32;
                    Console.WriteLine(j);
                }
                else
                {
                    Console.WriteLine("String could not be parsed.");
                }
            }
            else
            {
                Console.WriteLine("Oops!");
            }
            Console.ReadKey();
        }

        static bool matchesPermutation(string test, string search)
        {
            string remaining = search;
            for (int i = 0; i < test.Length; i++)
            {
                int pos = remaining.IndexOf(test[i]);
                if (pos == -1)
                    return false;
                else
                    remaining = remaining.Remove(pos, 1);
            }
            return true;
        }

        static int findPermutation(string test, string search)
        {
            for (int i = 0; i < test.Length - search.Length + 1; i++)
                if (matchesPermutation(test.Substring(i, search.Length), search))
                    return i;
            return -1;
        }
    }
}



And here is the Windows form code:

C#
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;

using ScrapySharp.Network;
using HtmlAgilityPack;
using ScrapySharp.Extensions;

namespace UGH2
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            // setup the browser
            ScrapingBrowser Browser = new ScrapingBrowser();
            Browser.AllowAutoRedirect = true; // Browser has many settings you can access in setup
            Browser.AllowMetaRedirect = true;
            WebPage PageResult = Browser.NavigateToPage(new Uri("http://192.168.1.99"));
            //go to the home page
            // get first piece of data, the page title
            HtmlNode Center = PageResult.Html.CssSelect("Center").First();
            string fruit = Center.InnerText;
            label1.Text = fruit;
        }
    }
}
Posted
Updated 7-Aug-20 7:10am
Comments
ZurdoDev 11-Oct-16 16:23pm    
What is the error?
Bobzilla Donovan 11-Oct-16 16:57pm    
There really is no error. The code does not progress beyond the Browser.Navigate statement. If I pause debug, it always goes to that point in the code. I am not able to close the form other than hitting the "stop debugging" button on the toolbar.
Philippe Mori 13-Oct-16 11:43am    
Lot of useless code. You should always provide minimal code necessary to reproduce the problem. Dumping lot of code does not help anyone.

There is a CodeProject article on this subject at Webscraping with C#[^].
 
Share this answer
 
The only thing I can see that is a difference between the two examples is the trailing slash in the URI. That may solve the problem as a private IP address is being used.

Console:
C#
//go to the home page
WebPage PageResult = Browser.NavigateToPage(new Uri("http://192.168.1.99/"));


WinForm:
C#
WebPage PageResult = Browser.NavigateToPage(new Uri
("http://192.168.1.99"));
 
Share this answer
 
Comments
Bobzilla Donovan 11-Oct-16 18:47pm    
I just tried adding the /. It still sticks there. Very strange. But thank You.
I dropped scrappy sharp altogether. I just used webclient() inside a using statement. I also used regular expressions to find the text I needed.

Worked great.
 
Share this answer
 
Run your scrapy code within a button click event in the win form code withing a separate task similar to this

Task.Factory.StartNew(() => 
{
    Forum forum = new Forum(txtUsername.Text, pass);

    //line below calls the hanging NavigateToPage
    Subject s = forum.AnalyzeSubject(id, ct.Token);
    //displaying result in the UI thread
    
    foreach (var item in s.CategoryActivity.OrderByDescending(so=>so.Count))
    {
        this.txtRez.BeginInvoke(new Action(() =>
        {
            this.txtRez.Text += item.ToString() + "\r\n";
        }));
    }
});
 
Share this answer
 
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900