Collect links from websearch

See more:

C#6.0

C#7.0

Before we start I want to say that I understand google and others have specific programs for http parsing and extraction but due to time constrains I have not had time to learn how to use them and some cost money which I don't have much of.

So I made a regex based link collector approach based of a keyword that is collected from the http title of another website, I am simply trying to find all websites that talk about similar stuff to the article the title is taken from. Hence I produced this code but it outputs 0 links can anybody help ...thanks much appreciated.

Overview of how it works. A URL is input in urlAddress, and the program collects all the https of that website, and I use a bit of code that extracts the title, I then join that title onto urlAddressofW and repeat but this time it does not represent a specific website but instead the screen after you have pressed search and before you open a specific website, when there are links to all the websites, the search results page. The code then extracts all the html of the first page of search results and feeds it through a regex "FindLinks" which I note, is kind of haphazardly combined so that may have been the problem, and this outputs all the links between the two points WebsiteExtractS and WebsiteExtractE which represents a link between the two, the code is then finally supposed to output each link it found, but outputs 0 links as the check NumofLinks stay as zero links.

string Titlestart = $"<title>";
string Titleend = $"</title>";
string WebsiteExtractS = $"iUh30 bc";
string WebsiteExtractE = $"&rsaquo";
string ADDING = "";
int NumofLinks = 0;
bool contains = false;


//Finding Website title
string urlAddress = "https://www.bbc.co.uk/news/uk-politics-50305284";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    Stream receiveStream = response.GetResponseStream();
    StreamReader readStream = null;
    if (response.CharacterSet == null)
    {
        readStream = new StreamReader(receiveStream);
    }
    else
    {
        readStream = new StreamReader(receiveStream,
        Encoding.GetEncoding(response.CharacterSet));
        readStream.ToString();
        // Console.WriteLine(readStream);
    }
    string data = readStream.ReadToEnd();
    string ImpureText = data;

    int titlestart = ImpureText.IndexOf(Titlestart) + Titlestart.Length;
    int titleend = ImpureText.IndexOf(Titleend) - 10; // this will take x amount of characters off but unknow how many characters are present that need taking off.
    string PureTexttitle = ImpureText.Substring(titlestart, titleend - titlestart);
    Console.WriteLine("Title-" + PureTexttitle);












    //Using title to find Links
    ADDING = PureTexttitle;


    string urlAddressofW = "http://google.com/search?q=" + ADDING;

    HttpWebRequest requestW = (HttpWebRequest)WebRequest.Create(urlAddressofW);
    HttpWebResponse responseW = (HttpWebResponse)requestW.GetResponse();
    if (responseW.StatusCode == HttpStatusCode.OK)
    {
        Stream receiveStreamW = responseW.GetResponseStream();
        StreamReader readStreamW = null;
        if (responseW.CharacterSet == null)
        {
            readStreamW = new StreamReader(receiveStreamW);
        }
        else
        {
            readStreamW = new StreamReader(receiveStream,
            Encoding.GetEncoding(responseW.CharacterSet));
            readStreamW.ToString();

            //Console.WriteLine(readStreamW);

        }
        string dataW = readStreamW.ReadToEnd();
        string ImpureTextW = dataW.ToString();

        if (ImpureTextW.Contains("iUh30 bc"))
        {
            contains = false;
            Console.WriteLine("YELP");
        }


        //this code is never reached
        MatchCollection Findlinks = Regex.Matches(ImpureTextW, "iUh30 bc(.*?) &rsaquo");

        //var Findlinks = Regex.Matches(ImpureTextW, "<p>(.*?)</p>");


        foreach (Match Link in Findlinks)
        {



            Console.WriteLine(Link.Value);

            NumofLinks += 1;




        }
        Console.WriteLine(NumofLinks);



    }
}
Console.ReadLine();

What I have tried:

Tried editing the regex to see if that was the issue and I think that seems to be it, as it says it can not find anything that is containing within the regex code.

Doing checks to see if links are even being collected.