Ignore the files that lucene is creating, just let it do what it does. Your issue is that you keep adding new documents. Think of the index like a database table, every time you add a document with properties you add a new row. So on day one your data for page1 is
ID Title
1 Hello
let's say on day two you change the title to Hello World and re-index, you now have
ID Title
1 Hello
1 Hello World
That might be what you want, it might not. It's probably not. The easiest cheapest way to deal with this is to clear the documents before you index
using (var writer = new IndexWriter(directory, analyzer, IndexWriter.MaxFieldLength.UNLIMITED))
{
writer.DeleteAll();
Another way that takes more work is when you re-index a page, first find the matching document if it exists, so you'll look for "+ID:1". If you don't find it, build the document and add it. If you do find it then remove the existing fields and re-add fields with the new data. Doing it this way also gives you another issue in that you'll need to know when to remove documents when pages get deleted.
When working with Lucene I stringly advise you to download Luke
Google Code Archive - Long-term storage for Google Code Project Hosting.[
^]
It's a tool that lets you see what data is in your index, see the documents, their fields, ratings, lets you run ad-hoc searches with different analysers etc. You'll need java on your machine to run it, but its worth its weight in gold.
Submit an article or tip