Click here to Skip to main content
16,020,567 members
Articles / Programming Languages / C#
Article

Seekafile Server 1.0 - Flexible open-source search server

Rate me:
Please Sign up or sign in to vote.
4.66/5 (31 votes)
8 Mar 20065 min read 172.4K   2.5K   120   56
A Windows Service that indexes DOC, PDF, XLS, PPT, RTF, HTML, TXT, XML, and other file formats. Desktop and ASP.NET search samples included.

Sample ASP.NET client

Image 1

Sample Windows Forms client

Image 2

Introduction

It has never been easier to create applications with search capabilities - open-source DotLucene [dotlucene.net] allows building powerful and super-fast full-text search applications. Moreover, it's easy to use. Let's demonstrate it by exploring Seekafile Server [seekafile.org] - a flexible indexing server with capabilities similar to that of Windows Indexing Service [microsoft.com].

This article is a follow-up of Desktop Search Application: Part 1. In that article, I have discussed indexing and searching Office document using DotLucene [dotlucene.net]. This time we will build a more serious application that can be either used directly or as a studying material for practical usage of DotLucene.

In this article, you will learn:

  • How to perform indexing in the background.
  • How to update documents in DotLucene index.
  • How to create queries for DotLucene programmatically.
  • How to use IFilter to parse Office documents, Adobe PDF and other file types correctly (it includes the updated parsing code from Desktop Search Application: Part 1).

Features

Seekafile Server [seekafile.org] is a Windows service that indexes documents in the specified directories and watches them for changes.

  • Background indexing
    • The indexer runs as a Windows service.
    • You specify the directories to be watched for changes in the configuration file.
    • Indexer works on the background (it doesn't slow down other operations).
    • It recognizes any change within a second.
  • Powered by DotLucene [dotlucene.net]
    • Super-fast searching.
    • The index is stored in DotLucene/Lucene 1.3+ compatible format.
    • The index can be accessed directly from other applications (you can search even when the indexing is in progress).
    • Access the index from any custom application (ASP.NET, Windows Forms application, Java application).
  • Built-in support for common file formats:
    • Microsoft PowerPoint (PPT)
    • Microsoft Word (DOC)
    • Microsoft Excel (XLS)
    • HTML (HTM/HTML)
    • Text files (TXT)
    • Rich Text Format (RTF)
  • Supports custom plug-ins written in C# or VB.NET.
  • Supports IFilter for searching other extensions:
    • Adobe Acrobat (PDF)
    • Microsoft Visio (VSD)
    • XML
    • and other...
  • Runs on Windows 2000/XP/2003

How it works

Architecture

The is the overview of the architecture:

Image 3

The architecture is index-centric. It uses the index to communicate with the client search applications. The index is flexible enough to allow this:

  • It is possible to search the index while the Seekafile Server is modifying it.
  • There can be multiple clients accessing the index simultaneously.
  • The changes are visible immediately to all the clients.
  • The only information clients need to know is the index location and the available DotLucene document fields [dotlucene.net].
  • The index is compatible with the Java version - you can access it from a Java client as well.

Watching changes

This is an overview of the indexing process:

  1. When the service is started it checks whether the index was already created at the specified location; if not it creates a new one:
    C#
    if (!IndexReader.IndexExists(cfg.IndexPath))
    {
        Log.Echo("Creating a new index");
        IndexWriter writer = new IndexWriter(cfg.IndexPath, 
                              new StandardAnalyzer(), true);
        writer.Close();
    }
  2. It goes through all the indexed directories and adds all the files to the IndexerQueue (to ensure that everything is indexed properly):
    C#
    foreach (string folder in cfg.Items)
    {
        IndexerQueue.Add(folder);
        startWatcher(folder);
    }
  3. It starts the FileSystemWatcher to watch all file changes in the indexed directories:
    C#
    private void startWatcher(string directory)
    {
        watcher = new FileSystemWatcher();
        watcher.Path = directory;
    
        watcher.NotifyFilter = NotifyFilters.LastWrite | 
                                 NotifyFilters.FileName | 
                                 NotifyFilters.DirectoryName;
        watcher.IncludeSubdirectories = true;
        
        watcher.Filter = "";
        
        watcher.Changed += new FileSystemEventHandler(OnChanged);
        watcher.Created += new FileSystemEventHandler(OnChanged);
        watcher.Deleted += new FileSystemEventHandler(OnChanged);
        watcher.Renamed += new RenamedEventHandler(OnRenamed);
    
        // start watching
        watcher.EnableRaisingEvents = true;
    }
  4. If there is a change event, it adds the file to the IndexerQueue:
    C#
    private void OnChanged(object source, FileSystemEventArgs e)
    {
        // skip directory changes if it's not a name change
        if (Directory.Exists(e.FullPath) && 
                e.ChangeType == WatcherChangeTypes.Changed)
            return;
    
        IndexerQueue.Add(e.FullPath);
            
    }
    
    private void OnRenamed(object source, RenamedEventArgs e)
    {
        IndexerQueue.Add(e.OldFullPath);
        IndexerQueue.Add(e.FullPath);
    }

IndexerQueue

The IndexerQueue works this way:

  1. It works in a separate thread. There is only a single thread processing a single queue at any moment:
    C#
    public static void Start()
    {
        if (instanceDirectory == null)
            throw new ApplicationException("You must " + 
               "initialize the queue first by calling Init().");
    
        lock (runningLock)
        {
            if (!isRunning)
            {
                indexerThread = new Thread(new ThreadStart(Run));
                indexerThread.Name = "Indexer";
                indexerThread.Start();
            }
        }
    }
  2. It processes the items from the queue. It waits if there is nothing in the queue:
    C#
    while (!shouldStop)
    {
        if (nextPath != null)
        {
            // process nextPath
            // ...
    
            // remove it from the list
            lock (items.SyncRoot) 
            {
                items.Remove(nextPath);
            }
        }
        // nothing to do, let the processor do something else
        else
        {
            Thread.Sleep(100);
        }
        // try to take a next item
        nextPath = next();
    }
  3. If the path is a directory, it goes through it and adds all its content to the queue:
    C#
    private static void parseDirectory(DirectoryInfo di)
    {
        foreach (FileInfo f in di.GetFiles())
        {
            Add(f.FullName, false);
        }
    
        foreach (DirectoryInfo d in di.GetDirectories())
        {
            parseDirectory(d);
        }
    }
  4. If the path does not exist, it deletes it from the index (deleteDocuments) including all subfiles if there are any (deleteDirectory):
    C#
    private static void deleteDocuments(string fullName)
    {
        IndexReader r = IndexReader.Open(instanceDirectory);
        int deletedCount = r.Delete(new Term("fullname", fullName));
        r.Close();
    }
    
    private static void deleteDirectory(string fullName)
    {
        IndexReader r = IndexReader.Open(instanceDirectory);
        int deletedCount = r.Delete(new Term("parent", fullName));
        r.Close();
    }
  5. If the path is already in the index, it checks whether there is any change in file length, creation time, or last write time. To check whether the document is in the index, we create a query programmatically using BooleanQuery and TermQuery classes:
    C#
    private static bool isInIndex(FileInfo fi)
    {
        IndexSearcher searcher = new IndexSearcher(instanceDirectory);
    
        BooleanQuery bq = new BooleanQuery();
        bq.Add(new TermQuery(new Term("fullname", 
                             fi.FullName)), true, false);
        bq.Add(new TermQuery(new Term("length", 
                             fi.Length.ToString())), true, false);
        bq.Add(new TermQuery(new Term("created", 
               DateField.DateToString(fi.CreationTime))), true, false);
        bq.Add(new TermQuery(new Term("modified", 
              DateField.DateToString(fi.LastWriteTime))), true, false);
    
        Hits hits = searcher.Search(bq);
        int count = hits.Length();
        searcher.Close();
    
        return count == 1;
    }
  6. If there are changes it updates the document in the index. Updating requires deleting the old document and adding a new one:
    C#
    // updates are expensive - proceed only if the 
    // file is not up-to-date
    if (isInIndex(fi))
        return;
    
    // delete all existing document with this name
    deleteDocuments(fi.FullName);
    
    // add the document again
    addDocument(fi);
  7. When adding a document, we record the following metadata:
    • name: file name, e.g. document.doc,
    • fullname: path, e.g. c:\storage\marketing\document.doc,
    • parent: all parent directories, inserted as multiple fields, e.g. c:\; c:\storage; c:\storage\marketing,
    • created: creation time,
    • modified: last write time,
    • length: file length in bytes,
    • extension: file extensions, e.g. .doc.
    C#
    Document doc = new Document();
    doc.Add(new Field("name", fi.Name, true, true, true));
    doc.Add(new Field("fullname", fi.FullName, true, 
                                            true, false));
        
    DirectoryInfo di = fi.Directory;
    while (di != null)
    {
        doc.Add(new Field("parent", di.FullName, true, 
                                            true, false));
        di = di.Parent;
    }
        
    doc.Add(Field.Keyword("created", 
                DateField.DateToString(fi.CreationTime)));
    doc.Add(Field.Keyword("modified", 
                DateField.DateToString(fi.LastWriteTime)));
    doc.Add(Field.Keyword("length", fi.Length.ToString()));
    doc.Add(Field.Keyword("extension", fi.Extension));

Parsing the files

DotLucene is able to index only plain text. Therefore, we need to extract the plain text from the rich file formats like Microsoft Word DOC, RTF, or Adobe PDF. The parsing can be done using a .NET plug-in found in the plugins subdirectory of the Seekafile Server or by IFilter interface (which is available in all Windows 2000/XP/2003 installations).

Read more about IFilter:

Plug-ins

Generally, there are two ways of extending the parsing system:

Read more about custom plug-ins:

There is also a sample plug-in included in Seekafile Server download [seekafile.org].

Sample ASP.NET client search application

Image 4

This ASP.NET application accesses the index directly to search it. It searches the file content only (file and directory names are ignored). It shows a relevant snippet from the document.

Read more about building an ASP.NET client search application [seekafile.org].

Download [seekafile.org] this sample as a part of the Seekafile Server from seekafile.org.

Sample Windows Forms client search application

Image 5

This Windows Forms application accesses the index directly to search it. It searches the file content only (file and directory names are ignored).

Read more about building a Windows Forms client search application [seekafile.org].

Download [seekafile.org] this sample as a part of the Seekafile Server from seekafile.org.

Features planned for next versions

  • Exclude filters.
  • Multiple indexes per service.
  • Windows Forms client search application.
  • Simple GUI management.
  • Convenient installer.
  • Indexing status and notification support.
  • Multi-user desktop search.

Acknowledgements

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Czech Republic Czech Republic
My open-source event calendar/scheduling web UI components:

DayPilot for JavaScript, Angular, React and Vue

Comments and Discussions

 
QuestionCreating a plugin ? Pin
Sairaj Sunil29-Aug-06 18:31
Sairaj Sunil29-Aug-06 18:31 
AnswerRe: Creating a plugin ? Pin
Dan Letecky29-Aug-06 20:52
Dan Letecky29-Aug-06 20:52 
QuestionHow do I index pdf documents using Seekafile Server Pin
Sairaj Sunil28-Aug-06 22:04
Sairaj Sunil28-Aug-06 22:04 
AnswerRe: How do I index pdf documents using Seekafile Server Pin
Dan Letecky28-Aug-06 22:44
Dan Letecky28-Aug-06 22:44 
GeneralRe: How do I index pdf documents using Seekafile Server Pin
Sairaj Sunil29-Aug-06 2:58
Sairaj Sunil29-Aug-06 2:58 
GeneralRe: How do I index pdf documents using Seekafile Server Pin
Dan Letecky29-Aug-06 20:53
Dan Letecky29-Aug-06 20:53 
GeneralProblems Pin
Bassam Abdul-Baki10-May-06 6:26
professionalBassam Abdul-Baki10-May-06 6:26 
Generalgreat job, but some questions Pin
margiex19-Apr-06 23:42
margiex19-Apr-06 23:42 
1. when in Indexing process, If I restart the index server, can it indexing directory from the break point or beginning?
2. when the index service is running, I added watch folder into config file, can it indexing it immediately?

thanks
GeneralRe: great job, but some questions Pin
Dan Letecky20-Apr-06 0:49
Dan Letecky20-Apr-06 0:49 
GeneralRemovable Storage/Static Index Pin
kennster6-Apr-06 9:38
kennster6-Apr-06 9:38 
GeneralRe: Removable Storage/Static Index Pin
Dan Letecky14-Apr-06 9:48
Dan Letecky14-Apr-06 9:48 
GeneralDotlucene 1.9 Pin
joecod22-Mar-06 1:57
joecod22-Mar-06 1:57 
GeneralRe: Dotlucene 1.9 Pin
Dan Letecky22-Mar-06 3:29
Dan Letecky22-Mar-06 3:29 
GeneralRe: Dotlucene 1.9 Pin
NGHIA.Mr8-Apr-09 18:59
NGHIA.Mr8-Apr-09 18:59 
GeneralFAILs scanning directory Pin
brunocol12-Mar-06 12:03
brunocol12-Mar-06 12:03 
GeneralRe: FAILs scanning directory Pin
Dan Letecky16-Mar-06 0:59
Dan Letecky16-Mar-06 0:59 
GeneralCrawler Pin
John Osborn12-Mar-06 2:35
John Osborn12-Mar-06 2:35 
GeneralRe: Crawler Pin
brunocol12-Mar-06 11:02
brunocol12-Mar-06 11:02 
GeneralGreat article...you get 5 from me... Pin
Brian Pautsch8-Mar-06 3:53
Brian Pautsch8-Mar-06 3:53 
GeneralRe: Great article...you get 5 from me... Pin
Dan Letecky8-Mar-06 4:34
Dan Letecky8-Mar-06 4:34 
GeneralURL's in asp.net search app Pin
tony_bbc6-Mar-06 23:52
tony_bbc6-Mar-06 23:52 
GeneralRe: URL's in asp.net search app Pin
tony_bbc7-Mar-06 0:45
tony_bbc7-Mar-06 0:45 
GeneralRe: URL's in asp.net search app Pin
Dan Letecky7-Mar-06 1:34
Dan Letecky7-Mar-06 1:34 
GeneralGreat! much needed... Pin
DaberElay28-Feb-06 10:26
DaberElay28-Feb-06 10:26 
GeneralRe: Great! much needed... Pin
DaberElay28-Feb-06 10:55
DaberElay28-Feb-06 10:55 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.