Develop a BHO for IE that retrieves search results URL's from Google and posts them in a text file

Question

5.00/5 (1 vote)

See more:

The purpose is to eventually develop an extension for IE which is capable of detecting all search results URL's when searching for any topic on Google, and save these URL's in a text file.

The main problem I am facing is that all search results on Google are not visible, if you try inspecting the source code of a Google search result page, you'll find that their pages were built dynamically and therefore, there are no links available on the pages, just a bunch of dynamic java scripting.

How can I retrieve the true URL links using the extension?

Posted 1-Feb-13 0:07am

Daroosh

Add a Solution

2 solutions

Solution 1

You can use Google API to get search results. And search word could be retrieved from URI.

Posted 1-Feb-13 1:59am

Kurkul

Comments

Daroosh 1-Feb-13 8:20am

Which Google API? Can you offer me some Source code to further elaborate your solution.

Kurkul 1-Feb-13 8:29am

Please look at https://developers.google.com/custom-search/docs/xml_results?hl=en#wsSampleQueries to get result in xml

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Daroosh · Accepted Answer · 2013-02-04T06:50:00

I managed to reach a solution to this problem.

My problem was that i thought i couldn't access the URL's on the Google search page. You won't be able to view them if you try viewing the PAGE SOURCE in IE however, the URL's are not stored in the PAGE SOURCE anyway since that's only the MAIN HTML SOURCE Page. The URL search list is actually built in dynamically and its available in the pages' DOM.

What you'll need to do is capture the URL's by catching them in one of the DISPID_DOM**** Events. The DOM events are fired only when the DOM is completely loaded, unlike the DISPID_DOCUMENTCOMPLETE event which is fired when the Source HTML is loaded.

The URL's are nested in the DOM in this way:

XML

<div id="ires">
       <a class="1" href="URL">
      </a>
</div>

What you'll need to do is Search for the "div" Tag with an "id=ires" value using the get_id(&idStr) method. Then traverse every node within this tag till you find the "href" you are looking for.