New rating methodology for contests: segment-based approach
Thread poster: Henry Dotterer

Henry Dotterer
Local time: 08:32
SITE FOUNDER
Oct 20, 2012

A number of the language pairs in the current translation contest have just been moved into the "qualification" phase. That means you can review and rate translations submitted by contestants in various pairs. (For a list of pairs in which that is possible, go to the contest page and scan the list of languages in the box to the right, looking for the word "Qualification" beside the language name.)

The "qualification" phase is the mode a pair enters when more than seven (7) entries have been received. The idea in this mode of the contest is to allow peer rating, as a means of selecting which of the high number of entries will advance to a finals round.

Until now, qualification round rating has been carried out on an entry-by-entry basis. This time around an additional (optional) approach is available: rating on a segment-by-segment basis. Member José Henrique Lamensdorf expressed one of the intended benefits of this approach when he observed: "It becomes an interesting learning experience to compare how different translators handled the very same segment."

If you work in any of the pairs now in the qualification round, please give the new rating system a try. Your feedback on the approach, and suggestions for improvement, would be appreciated here.


Direct link Reply with quote
 

Alexandra Schneeuhr  Identity Verified
Cyprus
Local time: 15:32
Member (2012)
English to Russian
+ ...
The segment-based approach Oct 20, 2012

Hi all,

I strongly favour the segment-based approach; in fact I'm totally unable to evaluate/rate any entry as a whole, as none are 100% perfect: there are some in my language pair where 10 to 12 segments are translated absolutely brilliantly, and then there comes a huge blunder (either a wrong or poorly-worded translation, or a dissapointing typo).

BTW, how are we supposed to treat typos and misspellings? Is it a minus one point in 'quality' or 'accuracy'? ))


Direct link Reply with quote
 

Dr. Matthias Schauen  Identity Verified
Germany
Local time: 14:32
Member
English to German
Weaknesses of segment-based apporach Oct 31, 2012

Hi,

I see at least two problems with rating segments instead of complete entries:

1. The average ratings can be skewed by people only rating good segments or only rating bad segments.

2. Someone with only one great idea for only one of the segments could theoretically reach the finals, even if their other translations were substandard. Example: In the recent Yogi-Berra-Contest my total ratings were based on the ability of the contestant to deal with all of the about five really difficult quotes satisfactorily. A contestant having a great idea for only one of the quotes could get high ratings for only this and end up in the finals, even though they might have misunderstood a few of the other segments.

I see the benefit José mentioned ("It becomes an interesting learning experience to compare how different translators handled the very same segment"), but this (and also the group commentary function IMO) moves the contest in a different direction: From a competition enabling good translators to gain exposure to a general translation discussion board.

I propose to get rid of the segment-wise rating for future contests.


Direct link Reply with quote
 

Henry Dotterer
Local time: 08:32
SITE FOUNDER
TOPIC STARTER
Thanks, Dr. Matthias Schauen Nov 1, 2012

Dr. Matthias Schauen wrote:
1. The average ratings can be skewed by people only rating good segments or only rating bad segments.

Insightful. I don't have a reason to believe that results have been skewed, but we do see that a non-trivial percentage of raters seem to be rating only the segments that they find particularly good or particularly bad. (Fortunately, enough people rate enough of the segments that we were able to require a certain minimum number of ratings as a basis for finalist selection... so maybe it doesn't matter. Still, it would stand to reason that a person who skips a segment has an opinion on that segment, and capturing that opinion could lead to more reliable selection. Any ideas?)
2. Someone with only one great idea for only one of the segments could theoretically reach the finals, even if their other translations were substandard.

That's not correct. The average for each entry is calculated first on a per-segment basis, and then the averages for all segments are averaged to get the overall average for the entry. So with the source text we used this time, for example, which had 14 segments, one segment can carry only 1/14th of the overall weight.

If anything, the argument could be made that some segments (the challenging ones, ie. those with a high "degree of difficulty") ought to be weighted more highly. We plan to experiment with that.


Direct link Reply with quote
 


To report site rules violations or get help, contact a site moderator:

Moderator(s) of this forum
Jared Tabor[Call to this topic]

You can also contact site staff by submitting a support request »

New rating methodology for contests: segment-based approach

Advanced search






SDL MultiTerm 2017
Guarantee a unified, consistent and high-quality translation with terminology software by the industry leaders.

SDL MultiTerm 2017 allows translators to create one central location to store and manage multilingual terminology, and with SDL MultiTerm Extract 2017 you can automatically create term lists from your existing documentation to save time.

More info »
LSP.expert
You’re a freelance translator? LSP.expert helps you manage your daily translation jobs. It’s easy, fast and secure.

How about you start tracking translation jobs and sending invoices in minutes? You can also manage your clients and generate reports about your business activities. So you always keep a clear view on your planning, AND you get a free 30 day trial period!

More info »



Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search