Navigating all the pages in a website.

Question

0.00/5 (No votes)

See more:

Hi everyone,

Does anyone have an idea of how to programmatically navigate through all the pages of a website(probably pages where login is not required), when the index(home URL) of the website is known. Any help or ideas will be highly appreciated.

Thanks
Anurag

Posted 20-Dec-10 19:12pm

@nuraGGupta@

Add a Solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

AspDotNetDev · Answer 1 · 2010-12-20T19:16:00

Solution 1

Yeah, there are several articles on Code Project that implement "web crawlers" / "web spiders" (the name for the thing you're trying to do). Read them. The short version is: download the HTML, check for links, navigate to those links and download that HTML until you have no more links to check.

Posted 20-Dec-10 19:16pm

AspDotNetDev

Comments

@nuraGGupta@ 21-Dec-10 1:32am

Ok, thanks. It was really helpful. :-)

Al-Farooque Shubho · Answer 2 · 2010-12-20T19:21:00

You really need some kind of information source to know the name and location of all pages within a web site, to be able to navigate those programmatically.

For example, Google searches for SiteMap.xml within the web root directory of a web site, and if it is found, it reads the page locations written within it and then crawls the pages to read SEO oriented data from each page.

So, what you need is some kind of SiteMap.xml within your target web site to know the name and location of all pages within the site so that you can programmatically navigate through the pages.

prayerlin rex · Answer 3 · 2010-12-20T19:17:00

Solution 2

Hi ,

In the page load event you have to redirect to related page .

Or

U have try with Application Start Event

Posted 20-Dec-10 19:17pm

prayerlin rex

Comments

@nuraGGupta@ 21-Dec-10 1:21am

Not a good answer, I was asking for getting all the related pages, and you are telling me the way to navigate through them.