Re: Efficient Search/Comparison Algorithm - Algorithms Discussion Boards

My requirement is : I will be having billions of data records coming in per day.From those I have to filter some 1000-2000 records based on record names which are stored in a filter.
My doubt is what is the suiatble data structure to be used and which search algo will meet this high volume requirement.

Sanchit

Sanchit,

You need to be more specific. In what format are the input records or is this something which can be defined for this implementation? In what form are the record names in the filter and can this format be defined for this implementation?

Dave.

The most efficient data structure for lookups is the hash table.

Create a hash table with a size that's about three times the number of names you're looking for, that's a prime number. Insert the 1000-2000 names you're looking for.

Then for each data record that comes in, look up the name in the hash table. Use a language like C++, which executes faster than C#.

You may also want to look at a radix tree:
http://en.wikipedia.org/wiki/Radix_tree[^]

It will save the time required to calculate a hash for each of the billions of records at the cost of a longer search per record.

Hash table:
- calc hash of record (slow)
- check table for hash value (quick)
- handle searching collisions (quick)
- if found matching hash in table do a full string compare on string record and hash table record as hash algo can produce same hash for different strings. (slow)

Radix tree:
- search tree for record string (slow)

The radix should be faster. The hash function will likely have a similar cost to the radix tree search. It may even be worse as the hash looks at each character in the string whereas the radix search can fail out early.

...cmk

The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying.
- John Carmack

The initial post mentioned that the search would be for record names that were stored in a filter. If only a field of the record (the name) needed to be checked, wouldn't that reduce the effort to create a hash - in fact wouldn't a Radix search be faster? That is why I asked for more information about the input and filter format.

Dave.

Member 4194593 wrote:
If only a field of the record (the name) needed to be checked,

I understood that each record would contain a (single) filter field, that's why i suggested as radix tree. A hash would likely be more approriate if you wanted to match several fields.

Member 4194593 wrote:
wouldn't that reduce the effort to create a hash

Yes. Sounds like you're in favour of using a hash table. Smile | :)

Member 4194593 wrote:
in fact wouldn't a Radix search be faster?

Yes, based on my op. Sounds like you're in favour of using a radix tree. Smile | :)

Original post: 'based on record names'
I assume this to mean strings, in which case my op stands.

...cmk

The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying.
- John Carmack

Good point. Some other pros and cons: A radix tree would require more memory accesses. But on the other hand, a radix tree would take less memory than a hash table, so would be more likely to fit entirely in a cache.

The hash-value calculation wouldn't necessarily be slow; it would be all integer operations on a key that could be in a single cache line. Also the number of collisions can be reduced in two ways:

1. Randomly-hashed values follow a Poisson distribution, so the expected number of collisions can be calculated for a given hash-table size. The table size can be increased to achieve an acceptable collision rate.

2. Assuming the list of names is fixed, a perfect hash function http://en.wikipedia.org/wiki/Perfect_hash_function[^] can be constructed, which will have no collisions.

If it's necessary to check billions of records per day, that sounds like enough computer work--regardless of algorithm--that profiling will be necessary to achieve optimal performance. Radix trees can be good if the records to be searched for differ from other records in the characters that are examined first. They can be bad if records often differ only in the characters that are examined much later.

With hash functions, there is often going to be some trade-offs involving the time required to perform a hash, the hash table size, and the frequency of hash collisions and false matches. Increasing the complexity of a hash function such that the time required increases by more than half the cost of a direct record comparison won't make much sense if doing so unless it saves an average of one direct record comparison for every two records. Having a hash table so large that every access entails a cache miss may slow things down more than would the occasional hash collision or false match.

One approach that may be helpful would be to compute a somewhat long hash of a record, hash that to produce an index into a hash table, and have the table itself store the 'long' hash in addition to the actual record. One could thus at very little expense test the long hash of the record to be checked against the long hash in the table. This would eliminate a large fraction of the direct record comparisons that would otherwise be needed.

BTW, how fast are modern machines at 32x32 integer multiplication? If the strings are zero-padded as well as zero-terminated and multiplies are cheap, it might be good to do something like:

unsigned long *ptr;
unsigned long hash,temp;

/* Point ptr at the start of the data, preferably word aligned */
do
{
  temp = *ptr++;
  hash += hash >> 23;
  hash ^= temp;
  hash *= some_big_prime_number;
} while(temp);

That would gobble up 32 bits of string data per loop iteration; I would expect the multiply, even if it takes multiple cycles, could be done concurrently with the next word fetch. Not the world's most sophisticated hash function by any stretch, but the execution time should be pretty minimal. If you want to use the hash function to yield a table index, use the upper bits. Taking the modulus of a prime number (distinct from the one used for the multiply) may be better, but modulus operations are more expensive than shifts.

Interesting points. Your comment about better performance if the records differ in the characters examined first reminded me of the ID3 algorithm http://en.wikipedia.org/wiki/ID3_algorithm[^]. Basically it builds a decision tree that first examines the attributes of a record that differ the most. This avoids wasting time examining attributes that don't differ much among the records.

Your point about cache misses is good; for this volume of data the best data structure is one that fits in the cache.

[Message Deleted]

Vij@y wrote:
anyone can convert this code to quick sort.

So convert it, then.

If you can't convert it, which part are you having problems with?

I need help from you who have experience with wumpus problem.
I would like to ask that at one step, how can we choose the adjacent field to go ?
I know one method that we choose the adjacent OK field (without mummies or pits) that has fewest visited times. My teacher said that it's not the most optimal method...
Would you mind helping me. Thanks in advance.

an excellent solution can be found here:
http://www.codeguru.com/forum/showthread.php?threadid=465309[^]

Smile | :)

I just think that if you can not help other people, you shouldn't consider them as a joke... Sigh | :sigh:

I think it was a way to indicate to you that you should look to other currently available sources before you ask others to spend their time trying to help you. Did you google "Wumpus"? Many available references. If you have checked those and still do not have an answer, then ask for individual help, but maybe on a gaming forum.

Sir, you win at the internet. Please remember to collect your prize before you disconnect.

Mark Churchill
Director, Dunn & Churchill Pty Ltd
Free Download: Diamond Binding: The simple, powerful, reliable, and effective data layer toolkit for Visual Studio.

Alpha release: Entanglar: Transparant multiplayer framework for .Net games.

The original BASIC code has been translated into C here[^].

I am not familiar with the Wumpus problem, but perhaps you can dissect the code. Google will also give Java implementations and I'm sure you can find others. Maybe looking at the code can help.

especially for special word, like "tooth", "teeth".
is there any snippet exists for this?
thanks

Use an "if" "elseif" structure since words like "tooth" and "teeth" are exceptions in English. Basically you'll need a lookup dictionary. To handle other cases you test if the last (and second to last) letter of the word is a vowel or not.

John, what is your opinion of this as a resource.

http://www.sequencepublishing.com/academic.html[^] or TheSage dictionary (and other applications) from this company. I have good things to say about TheSage - excellent freeware stuff.

modified 1-Aug-19 21:02pm.

Looks quite useful, Richard. I know when I write papers the biggest challenge is trying not to be repetitive. Too many phrases begin: "the model suggests that...", or "so and so, in their paper find that...."

Any resource (free!) that can improve the quality of academic publications is good value in my opinion. In fact, any resource that can improve the quality of writing in university courses is well worth it - even more so if the product is good quality and freely available. Others differ, but I don't mind open source/free software for educational purposes.

To supplement what 73Zep said, http://aspell.net/[^] - Free and Open Source spell checker. Have a look at their Manual [^] for some info ...

modified 1-Aug-19 21:02pm.

Hi,

I urgently need some research info or articles about Text Tables Comparison algorithms. Something like

An O(ND) Difference Algorithm and Its Variations ∗
[^]

but for tables.

Maybe someone was working in this field and have some old links.

Thanx a lot.

I have messed around in this area a bit - in this forum, as a matter of fact. I don't know if this would be of any use (depends upon what you mean by "Text Tables") - see http://www.codeproject.com/script/Forums/View.aspx?fid=326859&msg=2485913[^]

Nothing that is of research material, just a working program for massive file compares.

Dave.

Hi Dave,

Thank you very much for your reply. I have many tab delimited text documents like:

Person \t Name \t Title
1 \t NoName \t NoTitle
...

I need to find differences in this files. I used GNU diff. But if one tab will be missing it's showing full add full delete.

One document:
1 \t NoName NoTitle
Another document:
1 \t NoName \t NoTitle

So i'm trying to make my own simple algorithm for such tabular text.