How to compare two ranking algorithms?

Question

0.00/5 (No votes)

See more:

Hello all,

I want to compare two ranking algorithms. In these algorithms, client specifies some conditions in his/her search. According to the client`s requirements, these algorithm should assign a score for each items in data base and retrieve items with highest scores.

I have searched the net. According to my searches, the most relevant article which explains about some metrics for comparing ranking algorithms, was this:
http://www.icml2010.org/papers/504.pdf
I think prec@k, MAP, MRR, and NDCG, are good metrics to use, but I have a problem:

My algorithm sort results, so the first item in my result list is the best one with highest score, the second result have the second top score, and so on. I limit my search algorithm to for example find 5 best results.The results are the most top 5 items. So,precision will be 1. When I limit my search to find best result,It finds the best one. Again, precision will be 1.But the problem is that,it is unacceptable for people who see this result.

What can I do? How I can I compare these algorithms and show one is better than the other?
Please help me. Thanks a million.

Posted 1-Nov-14 19:47pm

k5_ce

Updated 1-Nov-14 19:49pm

v3

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

George Jonsson · Answer 1 · 2014-11-01T23:25:00

Solution 1

Who is the client? You teacher? (Teachers usually want it in the way they specify.)

But regardless of if this is home work or not, you need to understand the client requirements.
(I did not read the academic paper you linked. This is just common criteria)

The usual everyday measurements used to compare two algorithms are
1. Time, which is the fastest.
2. Accuracy, how often does the algorithm get the correct result.
3. Resource usage. How much memory, CPU etc is used.
(for example, does your algorithm totally block the CPU)
4. How specific is the algorithm. Can it be adopted to other tasks.

Posted 1-Nov-14 23:25pm

George Jonsson

Comments

k5_ce 2-Nov-14 8:10am

Thanks for your attention.
I`m the writer of one of these algorithms for a project. I want to show my algorithm is better than the other, so I have implemented my algorithm, and the other algorithm. I should show my algorithm is better than the other. Accuracy needs number of true positives, like precision. And this is the problem: all of the results are true, so precision@k will be 1, but when I want to defend my algorithm, it is not acceptable for people who hear this. Because when an algorithm's precision is one, it shows it has no problem, but it is not acceptable. Each algorithm can be improved. That is my problem and I want to show efficiency of my algorithm. I don`t want to show my algorithm is better from resource usages and it is specific for a subject. I have compared time, but it is not the case, I want to show my algorithm is better from some metrics.
Please help me.
Thanks a lot.

George Jonsson 2-Nov-14 13:58pm

So what is your definition of efficiency if it is not time or resources spent?
In most cases it boils down to money. What is the cheapest one? Too often, however, it is the cheapest short term, not long term.

k5_ce 5-Nov-14 0:45am

Thanks a lot. You are right. But, My algorithm`s time is not much better than those.Its accuracy and precision is better. I want to show this. I don't know how to show this????

Member 15657643 · Answer 2 · 2022-05-31T15:29:00

Construct your confusion matrix measurements with non-binary input. Scale your relevance or score so it is between 0 and 1, like a probability, call it 'p'. (Ignore ranking for now.)
If the item scored p is selected by the querier, TP += p, FN += 1-p
If the item scored p is not selected, TN += 1-p, FP += p.
Then use the Ln(Diagnostic Odds ratio), which is a balanced metric.
This accounts for both your scoring and whether it is useful to the query.