Automatic answer selection algorithm - please fix the discrepancy (Staff: done)
Thread poster: Katalin Horváth McClure

Katalin Horváth McClure  Identity Verified
United States
Local time: 19:52
Member (2002)
English to Hungarian
+ ...
Aug 17, 2009

There are two cases when the "robot" would grade a KudoZ question.
Case 1: Asker leaves the question open, ignoring all reminders.
Here is the description of what happens then:
http://eng.proz.com/faq/2731#2731
An 'autograding robot' makes its selection based on the net number of peer 'agrees' received by each answer, according to the following rules:
- To be considered, an answer must have at least two "net agrees" (agrees-disagrees).
- The answer with the highest number of net agrees is selected. If two answers have the same number of net agrees, the last one entered is selected. The idea is that if two answers gather the same number of net agrees, the one that did so in a shorter period of time may be better.


Case 2: Asker wants to close the question within 24 hours, but blocked by the system's new feature, and chooses to "leave grading the the community".
Here is the description of what happens in this case:
http://eng.proz.com/faq/5449#5449
About 72 hours after this submission, the system will grade the question in accordance with the following criteria:

a) If none of the answers has a net peer grading (agrees minus disagrees) of 2 or more, then the option selected by the asker will be used.

b) If at least one answer has a net peer grading higher than 1 and one of the answers has a net peer grading higher than the others, then this answer will be selected.

c) If at least one answer has a net peer grading higher than 1 and two or more answers share the same top peer grading, the answer posted first will be selected.

Bold emphasis is mine in both quotes.

The bold text shows the difference in the algorithm used when two answers have the same net peer grading. There is no reason to have a different algorithm, there is no logic that supports this difference.
Case 1 is OK, I have a problem with the bold part in Case 2.
I think there is no perfect algorithm, (there can be various scenarios how two answers can end up with the same number of net agrees), but the algorithm used in Case 1 is logical, represents the most probable scenario (that is, all those agreed to the second answer have seen the first one and still voted for the second, therefore preferring the second answer, while those that voted for the first one may not have seen the second answer when voted) and it has already been used by the system for years.

I would like to ask to make the selection algorithm for Case 2 to be the same as Case 1, when choosing between two or more answers that share the same top peer grading. In other words, select the last answer entered.

Thank you
Katalin

[Edited at 2009-08-17 22:28 GMT]

[Subject edited by staff or moderator 2009-08-25 18:54 GMT]


 

Enrique Cavalitto  Identity Verified
Argentina
Local time: 20:52
Member (2006)
English to Spanish
Good thinking Aug 17, 2009

Katalin Horvath McClure wrote:

all those agreed to the second answer have seen the first one and still voted for the second, therefore preferring the second answer, while those that voted for the first one may not have seen the second answer when voted



Hi Katalin,

There is a logic behind this difference and it had to do with time spans (over two weeks in the first case, 72 hours in the second), but I find your statement above to be very logical.

Thanks,
Enrique


 

Katalin Horváth McClure  Identity Verified
United States
Local time: 19:52
Member (2002)
English to Hungarian
+ ...
TOPIC STARTER
Time span difference doesn't matter, IMHO Aug 18, 2009

Enrique wrote:

There is a logic behind this difference and it had to do with time spans (over two weeks in the first case, 72 hours in the second)


I don't think it matters how long the time span is. In both cases, the answer posted later had less time until grading to collect the same amount of agrees.

(Besides, even if a question is open for 2 weeks, there is hardly any serious activity on them after 72 hours - in most cases activity ceases much earlier.)

I find your statement above to be very logical.


I am glad you find it logical. So, can we expect the algorithm change for Case 2?
Katalin


 

Enrique Cavalitto  Identity Verified
Argentina
Local time: 20:52
Member (2006)
English to Spanish
Implemented, thanks for the suggestion Aug 25, 2009

[quote]Katalin Horvath McClure wrote:

Enrique wrote:

I find your statement above to be very logical.


I am glad you find it logical. So, can we expect the algorithm change for Case 2?
Katalin


Thanks Katalin, the algorithm used for grading when a help question was pre-graded before 24 hours and also for GBK questions was modified as suggested. Now in all cases of robot-grading, when all other criteria are tied, the question posted last will be selected.

Regards,
Enrique


 

Katalin Horváth McClure  Identity Verified
United States
Local time: 19:52
Member (2002)
English to Hungarian
+ ...
TOPIC STARTER
Thank you Aug 30, 2009

Sorry for not responding earlier - I was on vacation.
Thanks for the modification in the algorithm and also in the corresponding FAQs.
Katalin


 


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Automatic answer selection algorithm - please fix the discrepancy (Staff: done)

Advanced search






SDL MultiTerm 2019
Guarantee a unified, consistent and high-quality translation with terminology software by the industry leaders.

SDL MultiTerm 2019 allows translators to create one central location to store and manage multilingual terminology, and with SDL MultiTerm Extract 2019 you can automatically create term lists from your existing documentation to save time.

More info »
WordFinder Unlimited
For clarity and excellence

WordFinder is the leading dictionary service that gives you the words you want anywhere, anytime. Access 260+ dictionaries from the world's leading dictionary publishers in virtually any device. Find the right word anywhere, anytime - online or offline.

More info »



Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search