Link extractor without unrelated links

Question

0.00/5 (No votes)

See more:

i have following code for link extractor which extracts all internal links for given url

C#

SearchEngines Search = SearchEngines.Google;
LinksExtractor extractor = new LinksExtractor("http://yahoo.com/",Search,10);
          
for (int i = 0; i < extractor.Links.Count; i++)
{
    Console.Write(extractor.Links[i].Href.ToString());
    //Console.ReadKey();
    Console.ReadLine();
}

This Code giving me all inks inside yahoo.com
like yahoo.com/sports
yahoo.com/business
but it also gives unwanted links like if some advertisement on yahoo for shadi.com
then it give shadi.com's link also
that i dont want
please help

Posted 6-Aug-11 1:53am

tejuv

Updated 6-Aug-11 2:00am

#realJSOP

v2

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

#realJSOP · Answer 1 · 2011-08-06T02:05:00

Solution 1

Is it that hard to ignore the links you don't want? For instance, anything that doesn't start with "http://yahoo.com/"?

Posted 6-Aug-11 2:05am

#realJSOP

BillWoodruff · Answer 2 · 2011-08-06T08:24:00

I wonder if you can make use of Google's Advanced filtering capabilities in creating your WebRequest ?

For example, this Google search[^] shows you only sites within Yahoo.com, and only sites in English.

But, perhaps you've already eliminated that as a strategy, so:

If extractor.Links is a collection of type IEnumerable<Link>, then you should be able to use a relatively simple Linq filter operation like:

string matchStr = "yahoo.com";

var filteredMatches = extractor.Links.Where(link => link.Href.ToString().Contains(matchStr)).ToList<Link>();

Disclaimer: this code fragment is off the 'top-of-my-head' and may not work for you as is, is not tested, and may be flawed: it is intended only to suggest a strategy to you.