Click here to Skip to main content
15,868,016 members
Articles / Web Development / IIS
Article

Microsoft Content Management Server Crawl Page for Search

Rate me:
Please Sign up or sign in to vote.
4.50/5 (6 votes)
17 May 2006CPOL1 min read 60.3K   882   35   3
Provides a start page for search engines to crawl a Content Management Server (MCMS) web site.

Sample Image - sample.gif

Introduction

This Crawl Results user control provides a start page for a search engine to crawl a Content Management Server (MCMS) web site.

For search purposes, it is recommended to hide a site's navigation menus when the browser user agent is detected to be a robot. This prevents words in the navigation menus from appearing in every page of the search result when a user searches for those words. With the navigation menus turned off, the search crawler will have no way to crawl through all the pages. This user control generates links to channels and postings, and allows a search engine to recursively crawl through all the pages.

As the user control does not generate links to all the channels and postings in one go, it is scalable to large sites. A meta robot tag of NOINDEX,FOLLOW should be included in the crawl page using the user control so that the crawler will not index the channel and posting lists but follow them.

Both C# and VB.NET versions are included.

Installation

  1. Copy CrawlResults.ascx and its code-behind file to the user control directory in your MCMS site.
  2. Create an ASPX page and insert CrawlResults.ascx. Change the StartChannelPath property to the root of your MCMS site. For example:
    ASP.NET
    <%@ Register TagPrefix="uc1" TagName="CrawlResults" 
                Src="~/UserControls/CrawlResults.ascx" %>
    . . .
    . . .
    <uc1:CrawlResults id="CrawlResults" 
      StartChannelPath="/Channels/WoodgroveNet" runat="server">
    </uc1:CrawlResults>
  3. Add the following meta tag to the HTML header in the ASPX page:
    HTML
    <meta name="ROBOTS" CONTENT="NOINDEX,FOLLOW">
  4. Set the start page of the content source in your search engine to the URL of your ASPX page. For example, http://<server name>/<site name>/crawlpage.aspx.

History

  • V1.0 - 2004.09.05 - Base.
  • V1.1 - 2005.02.12 - Converted page to user control. Added option to specify the starting channel path.
  • V1.2 - 2006.05.14 - ASP.NET 2.0 version added.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Software Developer (Senior)
Canada Canada
A Microsoft Certified Professional Developer and Technology Specialist.

Experience and expertise in SharePoint 2016 / 2013 / 2010 / 2007.

Role ranges from a developer in a multi-person team to a solution consultant with expert-level skills, leading a project to completion status.

Proven experience working effectively in a team environment and a self-managed environment.

Comments and Discussions

 
GeneralContent Search Pin
RoopeshPerlaKumar3-Feb-06 0:06
RoopeshPerlaKumar3-Feb-06 0:06 
GeneralUsing Google Search Engine Pin
kanid4-Feb-05 0:55
kanid4-Feb-05 0:55 
GeneralRe: Using Google Search Engine Pin
Stephen Huen7-Feb-05 21:04
Stephen Huen7-Feb-05 21:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.