Simple scraping / data collecting with C# selenium not working

Question

1.00/5 (1 vote)

See more:

So im trying to collect some data with Selenium using C# and I am fairly new to it, atleast the collecting data part, what im trying to achieve is, when I press a button is collects some data and puts it in a textbox, simple as that, the only thing is when I run my code it doesnt collect anything

im using a label as a button

C#

private void label11_Click(object sender, EventArgs e)
{
    var getTopFive = new FirefoxDriver();
    getTopFive.Navigate().GoToUrl("http://www.rottentomatoes.com/top/bestofrt/top_100_action__adventure_movies/");

    foreach (var item in getTopFive.FindElements(By.TagName("a")))

        //this is where I left off

    {
        activeTextBox.Text = (item.GetAttribute("class"));

    }

I think I messed it with the FindElements & GetAttribute but im not sure

What I have tried:

I've tried messing with the FindElements & GetAttribute but no luck

Posted 14-Apr-16 2:27am

BladeLogan

Updated 16-Apr-16 5:31am

Add a Solution

Comments

Sergey Alexandrovich Kryukov 14-Apr-16 12:38pm

Debugger!

What exactly do you want to retrieve? :-)

Besides, what do you want to retrieve, class name? Who told you there are classes on these elements? It does not seem to make any sense. If there are classes, you are retrieving next to nothing, class names. :-)

—SA

BladeLogan 14-Apr-16 14:10pm

An answer to your answer: Im actually trying to pull data from a stock website so I can add real time stock changes inside my software at this point, its really hard to explain without showing anything visually

Sergey Alexandrovich Kryukov 14-Apr-16 14:41pm

You did not understand my question, so you answered too generally. It's clarified it below, and also in my answer, which is, by the way, quite comprehensive. This is all you need.

Say, you might want to retrieve URIs, but you never even tried it. You are doing nothing, hence my question for you.

Also, what you tried to explain would be quite easy to explain. There is a lot of well-known terms you can use; first of all, names of HTML elements. Please, don't take it as offense: it's difficult to work with you; you have too much trouble explaining simple things, even more understanding them. It's not at all about English, it's about communication logic. I'm sure if you pay more attention, you can improve it. Each simple question you ask turns into dozens of posts. I would not mind if it was helpful, but it often goes nowhere...

—SA

BladeLogan 14-Apr-16 17:28pm

Im not taking any offense, I understand that it can be hard to work with me when im being lazy typer let alone when im tired, I could indeed explain more in-depth but I felt so exhausted after being up for a very long time and I just didnt feel like being so in depth because this is the last forum I come to when I need help because I feel like im overdoing it here with all the questions im posting, so after being at around 4 different ones with no answers or nothing, its just such a bummer that when I come here and I try to explain my issue it just turns into weird blurryness. But I guess the real question I wanted a answer too which was confusing me the most is.

Lets say I have the code

IList<iwebelement> movies = getTopFive.FindElements(By.CssSelector("[itemprop='url']"));

for (int i = 0 ; i < 20 ; ++i)
{
activeTextBox.Text = movies[i].Text;
}

Im having a hard time understainding how to use the "FindElements(By.CssSelector("[itemprop='url']"));" How would I know that itemprop='url is a CssSelector for example, same goes for By.Id or By.TagName

Sergey Alexandrovich Kryukov 14-Apr-16 17:50pm

Very good. About being exhausted. You can always take some rest. Importantly, its about your preference. What would you prefer: do some action paying full attention, or do it in a sloppy way, repeating same effort 10 times? I think the choice is simple, especially if you remember that wasted time is not only yours.

Same thing again: you got my answer, but you repeat your activeTextBox.Text assignment again.

And did I mention the word "Debugger"? Do I have to repeat it again and again?

—SA

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

BladeLogan · Answer 1 · 2016-04-16T05:31:00

Solution 2

for (int i = 0; i < 5; ++i)
{
activeTextBox.Text += movies[i].Text + "\r\n";

}

Posted 16-Apr-16 5:31am

BladeLogan

Sergey Alexandrovich Kryukov · Answer 2 · 2016-04-14T06:50:00

I looked at the page. There is a number of <a class="…" … > anchors.

So what? You are not retrieving anything. A typical anchor looks like <a class="unstyled articleLink" … >, so you would "retrieve" the string "unstyled articleLink". Any point? :-)

But you don't really do even that. Note that you assign a string to activeTextBox.Text… again and again… in a loop… :-)

Each time, the previous property value is discarded, and then new one is written. :-)

Asking "what to do" would not make any sense. Nothing except applying some elementary thinking can help.

By the way, there is no a serious need to use any 3rd-party software for such a simple task. You don't even need a real HTML browser, as it can be done with primitive text search, and the HTML string can be easily obtained with HttpWebRequest class, even simpler, with rudimentary HttpClient… All one needs is quick look at the documentation, and following it.

—SA

Simple scraping / data collecting with C# selenium not working

2 solutions

Solution 2

Solution 1

Add your solution here

Preview 0

Existing Members

...or Join us