|
I am working on an idea, but I am looking for any prior knowledge to the problem.
I have a dataset that contains records with a large number of searchable fields. I want to do a search for a set of criteria where I get matches with more than n% criteria matched.
Given Cn as a criteria that matches to between 0 and 1, it would be something like sum[C0..Cn]/n > 0.75 for a match of over 75%.
Is there any way beyond scanning the entire data set that this can be achieved?
My current thinking is to use several probable fields that will be in most [if not all] searches and apply indexes to them and use some jiggery-pokery to join data sets.
I'd take this to QA, but /someone/ will just tell me it is well known problem and to try looking on the internet. Well FU internet, I don't know what the name of the problem is, so I can't search!
veni bibi saltavi
|
|
|
|
|
Sounds very similar to spatial search trees[^].
You will not get around building the tree every time the criteria change, but then you can get any result you are looking for by simply navigating through the tree.
The language is JavaScript. that of Mordor, which I will not utter here
This is Javascript. If you put big wheels and a racing stripe on a golf cart, it's still a f***ing golf cart.
"I don't know, extraterrestrial?"
"You mean like from space?"
"No, from Canada."
If software development were a circus, we would all be the clowns.
|
|
|
|
|
That was something I looked at, but it will it work with a large number of dimensions but only searching for things intersecting a sub-set? Such as near an arbitrary line in 2d.
veni bibi saltavi
|
|
|
|
|
Such trees are often used in computer graphics where the objects are loaded once, sorted into the tree and then searched before every frame is drawn to exclude objects which are not visible at the moment. That's (among some other tricks) how your favorite shooter can still feature a high level of detail without having to draw thousands of unseen objects.
I see no reason why this would not work with many dimensions. The real question is, wether or not your search(es) justify the effort of building the tree first. Navigating the tree to find out what you are looking for also is not a big issue. the real work already has been done when building the tree.
The language is JavaScript. that of Mordor, which I will not utter here
This is Javascript. If you put big wheels and a racing stripe on a golf cart, it's still a f***ing golf cart.
"I don't know, extraterrestrial?"
"You mean like from space?"
"No, from Canada."
If software development were a circus, we would all be the clowns.
|
|
|
|
|
It is rather annoying, because I wont know if it's worth while until I try it. 
veni bibi saltavi
|
|
|
|
|
In the past I've gotten good responses to posting questions like this in the CP Algorithms forum: [^].
What you are describing sounds to me like what I read folks do with graph-databases, like Neo4j: [^].
Or, perhaps, what you can do with Microsoft's Trinity graph-database: [^].
«I want to stay as close to the edge as I can without going over. Out on the edge you see all kinds of things you can't see from the center» Kurt Vonnegut.
|
|
|
|
|
Thank you, oh Font of Abstraction! My reading list has just got longerer.
veni bibi saltavi
|
|
|
|
|
Not sure I fully understand as I am drunk. that said.
Is the 0..1 matching an essentially analog match (i.e. any value between 0 and 1) or are there only specific (i.e. digital) values
If the latter, and there are a reasonable number of discrete values, one can turn the problem around by storing each possible value as a boolean, in a bit pattern
we did this in a real estate system (as a concrete example)
so bit 0 meant 1 bedroom
bit 1 meant 2 bedroom
bit 45 meant swimming pool
bit 312 meant close to bus stop
etc.
then we read and logically AND ed each record with a search bit pattern.
because it's bits it's small and you can load a surprising number into memory for comparison.
Probably not your problem - but I think interesting anyway - much like this local Red
PooperPig - Coming Soon
|
|
|
|
|
Sadly, it varies from value to value. Some are true/false but a lot are in a range. Using your example, would be price and location as obvious analog values, but also something like number of bedrooms.
So when there is a range, let's say 3-5 bedrooms, those value would be 1.0, anything outside half the range would be proportional to how close it was so 2 and 6 bedrooms would be 0.5 match. That is my problem. So even a failure to match on one criteria may not mean that there is no match as it depends on the number of other things that match. Don't get me started on the weighting issues as that just makes your brain go off in a sulk and stick its tongue out.
veni bibi saltavi
|
|
|
|
|
Sounds like you need to use a similar method to that I proposed but using qubits...
PooperPig - Coming Soon
|
|
|
|
|
Starting from endless battles, via a storage building, to a higher place? – flotsam at journey’s end
(6, 2) 
modified 4-Sep-15 5:18am.
|
|
|
|
|
I would say washed up if it was (6, 2)
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.
|
|
|
|
|
erm - it should have been 6,2.. my brain needs debugging.
|
|
|
|
|
In that case;
Quote: Starting from endless battles, via a storage building, to a higher place? – flotsam at journey’s end
endless battles - wars
a storage building - shed
a higher place - up
flotsam - washed up
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.
|
|
|
|
|
You win 
|
|
|
|
|
Nuts, just remembered I'm going to Legoland on Monday.
Why do I only win these things when I'm going to a theme park the next (working) day?
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.
|
|
|
|
|
The answer is either you don't win this very often or you go to a lot of theme parks. 
veni bibi saltavi
|
|
|
|
|
Well I was Alton Towers last Thursday, going to Legoland Sunday and Monday, then to Thorpe Park the following Monday.
This is unusual activity though.
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.
|
|
|
|
|
chriselst wrote: This is unusual activity though.
Mid-life crisis unusual? Psychotic break? Lottery win? I feel the nation should know!
|
|
|
|
|
Bought Merlin Annual passes at the start of the summer, my daughter is about 5 cm too short to get full use of them in the bigger parks, but we've already saved more than we spent (if you ignore the fact we wouldn't have paid to go so many times if we didn't have them).
Some men are born mediocre, some men achieve mediocrity, and some men have mediocrity thrust upon them.
|
|
|
|
|
chriselst wrote: I'm going to Legoland
For some reason this made me think "Pull yourself together"
and I laughed out loud.
PooperPig - Coming Soon
|
|
|
|
|
I WANT![^]
Anything that is unrelated to elephants is irrelephant
Anonymous
-
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-
I'd just like a chance to prove that money can't make me happy.
Me, all the time
|
|
|
|
|
Trouble is, if you wear it while posting here, you'll get attacked by hungry CPer's...
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
|
|
|
|
|
It could be worse than merely wearing the t-shirt. You could also be using THIS[^] deodorant! 
[EDIT] DANG! I wanted to buy one of those deos just for fun. It's $15. But they want $70 in postage to Sweden! Stupid arses! It can't possibly be that expensive to ship such a small item - even to Europe!  [/EDIT]
Anything that is unrelated to elephants is irrelephant
Anonymous
-
The problem with quotes on the internet is that you can never tell if they're genuine
Winston Churchill, 1944
-
I'd just like a chance to prove that money can't make me happy.
Me, all the time
|
|
|
|
|
It's got to be better than "Brut" or "Old Spice" 
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
