# Displaying peer's kudoz rating in given field for agree/disagree comments

Pages in topic:   [1 2 3] >
Displaying peer's kudoz rating in given field for agree/disagree comments
Russian Federation
Local time: 09:57
English to Russian
 Sep 9, 2007

That would make a sense. Agree/Disagree from a person with 500 kudoz in the area has much more weight, than from person with no points (in the area) at all. IMHO it will give an asker valuable data for evaluating the answers. What do you think?

Sweden
Local time: 08:57
English to Swedish
+ ...
 QUALITY rather than QUANTITY! Sep 9, 2007

There are, in my opinion, 3 relevant quality criteria:

1. Average Kudoz earned per answer given
2. Average Kudoz awarded per answer agreed to
3. Average Kudoz awarded per answer disagreed to

Would a Reliability Index (RI) based on the above-mentioned point not be a better indicator than number of Kudoz earned? There is e.g. obviously a difference between somebody who earned 500 Kudoz with 125 answers and somebody who needed to submit 1000 answers to earn 500 Kudoz!

Proposed RI formula:

Ke = Kudoz earned (total, points)
Ka = Kudoz awarded to answers agreed to (total, points)
Kd = Kudoz awarded to answers disagreed to (total, points)
Ag = Answers given (total, number)
Aa = Answers agreed to (total, number)

Note 1: Ag, Aa and Ad must obviously each be larger than 0 to avoid divide by 0 problems, but that is not an unreasonable constraint. I would personally be in favour of that no RI was displayed if they were not above each 10.

Note 2: The formula gives equal weight to the 3 criteria; one could argue that they should be given different weights.

Narcis Lozano Drago
Spain
Local time: 08:57
Member (2007)
English to Spanish
+ ...
 Not really... Sep 9, 2007

Ke = KudoZ earned (total, points)
Ka = KudoZ awarded to answers agreed to (total, points)
Kd = KudoZ awarded to answers disagreed to (total, points)
Ag = Answers given (total, number)
Aa = Answers agreed to (total, number)

Case A: 1000 answers, 3999 kudoz points
Case B: 1 answer, 4 points

The RI for Case B > Case A. That would be fair?

Also, if one could get Kudoz for giving agrees I think that would be the end of the Kudoz system. After all, one could go around giving agrees to all +3 agrees answers, even if they don't have the slightest idea about the field or the correct answer and still get an impressive Kudoz point record. I agree with the quality rather than quantity (I myself have a good acceptance rate), but I don't think these obscure formulae are the solution.

Regards.

Kim Metzger
Mexico
Local time: 01:57
German to English
 Another good formula Sep 9, 2007

Narcis Lozano Drago wrote:

The RI for Case B > Case A. That would be fair?

Also, if one could get Kudoz for giving agrees I think that would be the end of the Kudoz system. After all, one could go around giving agrees to all +3 agrees answers, even if they don't have the slightest idea about the field or the correct answer and still get an impressive Kudoz point record. I agree with the quality rather than quantity (I myself have a good acceptance rate), but I don't think these obscure formulae are the solution.

I agree that it would be a good idea to somehow indicate how credible a peer commenter is and that it would only make sense to do so after a member has answered at least 100 questions maybe. The best indicator of credibility for both answerers and peer commenters that I have found is Kirill Semanov's formula:

Questions answered x 4 = a
Pro points earned divided by a = %

Example:
Points earned - 7763
4593 x 4 = 18372
7763 divided by 18372 = 42%

http://www.proz.com/topic/24865

According to this formula, Narcis, your reliability ratio is 61%, which is outstanding. The best ratio I've found so far is 65%.

P.S. the 4 indicates the maximum number of points possible for a correct answer. Maybe 3.5 would be a better number.

[Edited at 2007-09-09 15:22]

teju
Local time: 00:57
English to Spanish
+ ...
 This might have been mentioned before... Sep 9, 2007

... in other threads about the same subject, but two things come to mind:

1- You could be fairly new in ProZ, and incredibly smart. Your kudoz number and/or any rating number would not reflect the quality of your answers, misleading the asker.

2- The asker doesn't always pick the best answer. And I'm not talking about shades of gray here. In my language pair, I've seen some things that give me chills. This is also a problem for the glossary.

Anyway, there's always the option of clicking on the person's name and checking out their profile, that's what I've done many times.

Christel Zipfel
Italy
Member (2004)
Italian to German
+ ...
 I heartfully agree Sep 9, 2007

Kim Metzger wrote:

I agree that it would be a good idea to somehow indicate how credible a peer commenter is and that it would only make sense to do so after a member has answered at least 100 questions maybe.

BTW, I had already proposed several times to base the KudoZ ranking system on the realiability ratio rather than on the number of points, which alone doesn't mean anything, like Sven said.

This would also be much fairer for newcomers - with much less answers they can reach the same ratio as someone who participated since years - and should, in my opinion, hopefully also discourage those who actually jump onto every question hoping to earn some points.

I also agree that there must be a minimum number of questions answered - you cannot get a credible average out of 1, 2... 10 answers:-)

[Bearbeitet am 2007-09-09 18:12]

Heike Behl, Ph.D.
United States
Local time: 23:57
Member (2003)
English to German
+ ...
 numbers can be deceiving Sep 9, 2007

Although to some extent informative, a ratio based on these calculations would still not necessarily give an accurate picture.

People who choose to receive notifications of new questions are obviously at an advantage since the chance that they will provide the correct answer to the easier questions is pretty high.

People answering at a later point (or people not willing to answer easy "dictionary questions") will have no easy answers left, but maybe like the challenge of the more difficult questions with no or not the correct answer supplied by then. By default, the ratio would be not as good as for those in the above category, but you'd probably find the more reliable answerers in this category.

These are, of course, the extremes, and most people fall somewhere in between, but there are some extreme cases.

E.g. I came across the answers supplied by a translator with a high number of questions answered plus a high number of points received. This would no doubt result in a good ratio, however, the questions answered were so simple that I could answer most of them without even using a dictionary, and my language skills in that particular language are rather limited. I would also not have rated these questions as Pro.

Each answer should be evaluated on its own merits. Agrees/disagrees (particularly those with additional comments) can be very helpful, but should not be the decisive factor in choosing the best answer. A ratio next to the person's name would not - and maybe should not - make any difference for me.

Anybody participating regularly in Kudoz knows the more reliable people anyways, with or without ratios.

xxxFrancis Lee
Local time: 08:57
German to English
+ ...
 The onus should be on the Asker Sep 9, 2007

I appreciate what Vadim means, but if the Asker ultimately chooses an answer simply (or primarily) on the basis of peer agreements then that's their own fault. I sometimes feel there should be a disclaimer somewhere saying that just because an answer has a lot of Agrees, this is no guarantee of quality.

IMHO it will give an asker valuable data for evaluating the answers

Personally, when I ask a question I don't care how many points an Asker has. Two reasons:

- There are colleagues on the GermEng site who have relatively high Kudoz points/rankings but who IMO are poor translators. A lot of their points are from a) guesses b) being the first to answer an easy question. There is in particular one individual who has a surprisingly high Kudoz rating - and to be honest, when she has in the past given me an Agree (no longer the case since I gave her a Disagree) my first thought was "oh no, respected colleagues will treat any Agree from her like a red flag"

- There are perhaps a couple of dozen colleagues (both German- and English-speaking) whose opinions I automatically respect due to past experience. And a few of these are not very active re. Kudoz, i.e. have relatively few points. Plus teju raises the issue of new Kudozians who have no points but - in certain cases - high-quality suggestions.

If anything, then perhaps a ratio (as opposed to pure points) would work - but that can also be misleading. My own ratio would be higher were it not for the many questions where I take the time to research and provide references - and then "lose" the points because someone else posted the same answer but a minute quicker. But this is a side-issue.

teju wrote:
Anyway, there's always the option of clicking on the person's name and checking out their profile, that's what I've done many times.

Exactly - hence the title of my post.

xxxmediamatrix
Local time: 02:57
Spanish to English
+ ...
 Nice idea! Won't work! Sep 9, 2007

I agree it would be a good idea to give some kind of numerical indicator of an answerer's (apparent) abilities in the field of the question, rather than just showing their specialist or working interest in the field as now.

The main problem - as I've pointed out elsewhere http://www.proz.com/topic/80531 and http://www.proz.com/topic/81118?post_id=641182 - is that so many askers do not correctly categorise their questions, often putting engineering or medical questions as 'law/patents', for example.

As a result, numerical indicators linked to the fields applied by the asker will often not correspond to the answerer's ability in the true field of the question (in just the same way as the specialist/working flags in the present system are often incorrect).

Sloppy askers shoot themselves in the foot every time they don't set the fields correctly. First because peers who rely on e-mail notifications to alert them to the existence of questions in their fields will never be alerted and hence will not jump in to help. Second, because those sloppy askers won't see valid 'worker/specialist' flags and be guided to selected the most appropriate answer.

A side issue here is that they also mess up everyone's KudoZ ranking - and hence our positions in the freelancer directory.

So, before trying to do anything to improve the chances of an asker picking guidance from the 'most reliable' answerers we really do need a mechanism to ensure that askers set up their questions properly in the first place.

MediaMatrix

Russian Federation
Local time: 09:57
English to Russian
TOPIC STARTER
 agreed, but... Sep 9, 2007

Introducing reliability ratio is good idea and it was discussed many times, but it will require some willpower from the staff. I doubt they will implement it any time soon.

There's lot of pros as well as contras, e.g. it leaves behind people, who give a lot of not for grading answers. Transition must be slow and as painless for community as possible.

My idea idea is much more simple. If they show answerer's kudoz rating in the field, why not show peers' rating as well? And it won't require serious changes in site policy.
As simple as that.

[Edited at 2007-09-09 18:07]

xxxjacana54
Uruguay
English to Spanish
+ ...
 agree with teju's comments Sep 9, 2007

After a few days of visiting the Kudoz section, it's fairly easy to identify the answerers you can rely on, or whose style is more acceptable.
Sometimes people will provide answers saying "this may be another option" or "maybe you will find this useful". If we all become too conscious of statistics, this kind of answer, which can be very useful, will tend to disappear.

Kim Metzger
Mexico
Local time: 01:57
German to English

jacana54 wrote:

After a few days of visiting the Kudoz section, it's fairly easy to identify the answerers you can rely on, or whose style is more acceptable.
Sometimes people will provide answers saying "this may be another option" or "maybe you will find this useful". If we all become too conscious of statistics, this kind of answer, which can be very useful, will tend to disappear.

Yes, I agree with you jacana. This is one reason I proposed making it possible to opt out of the points race when you just want to provide some information that might help others find a good solution.

http://www.proz.com/topic/81997

Kirill Semenov
Ukraine
Local time: 09:57
Member (2004)
English to Russian
+ ...
 Worth considering, but let's start from something more simple Sep 9, 2007

Hi Sven,

Your idea of including "agree/disagree" factor to the RI (or RR, reliability ratio) formula is basically good, but there are two problems:

1) theoretical: the exact weights assigned to agreed/disagreed answers are most likely empirical, so it may take a long time to estimate the exact numbers. I'm not going even try to do it from my first glance, but any weights assigned to this or that variable in an arbitrary or even empirical way are always doubtful or at least need a lot of observations;

2) practical: including "agrees/disagrees" into the formula means a great load on the ProZ server. I don't even mean recalculating everything "on-the-fly" which is not done for a while even for kudoZ points (which, as we all know, are updated automatically only once per day). But even once-a-day recalculation of the rating formula for each and every proZ member will take a huge amount of time.

For a while, I would start from the simplest idea of ratio: KudoZ points won/questions answered. Then we'll see. And the technical capabilities of such busy servers will probably rise with time.

To all those who concerned about trivia examples of only 1 question answered and whole 4 points gained - no, statistics is about large numbers, by definition, so no less than 50-100 questions must be answered by any proZian to enter any kind of RI/RR competition.

PS. Dear Kim, thank you for your support and enthusiasm about the matter

Kim Metzger
Mexico
Local time: 01:57
German to English
 Members vote to place questions on hold Sep 9, 2007

mediamatrix wrote:

The main problem - as I've pointed out elsewhere http://www.proz.com/topic/80531 and http://www.proz.com/topic/81118?post_id=641182 - is that so many askers do not correctly categorise their questions, often putting engineering or medical questions as 'law/patents', for example.

So, before trying to do anything to improve the chances of an asker picking guidance from the 'most reliable' answerers we really do need a mechanism to ensure that askers set up their questions properly in the first place.

MediaMatrix

Poorly classified questions are indeed a problem, but just one among many. Questions asked without context is another major one as well as questions asked that have already been answered before. Not too long ago Jabberwock proposed placing sloppy questions on hold until the asker has prepared his question properly.

Jabberwock wrote:
I would rather have an option to put the question "on hold", i.e. it is not possible to answer the flagged question (the reasons should be very specific, preferably selected from a list) until the asker reformulates the question according to the rules. I think that the educational value would be greater (more of an encouragement than a slap).

http://www.proz.com/post/577464#577464

[Edited at 2007-09-09 19:22]

xxxmediamatrix
Local time: 02:57
Spanish to English
+ ...
 Excellent idea! Sep 9, 2007

Kim Metzger wrote:

mediamatrix wrote:

The main problem ... - is that so many askers do not correctly categorise their questions, often putting engineering or medical questions as 'law/patents', for example.

So, ... we really do need a mechanism to ensure that askers set up their questions properly in the first place.

MediaMatrix

Poorly classified questions are indeed a problem, but just one among many. Questions asked without context is another major one as well as questions asked that have already been answered before. Not too long ago Jabberwock proposed placing sloppy questions on hold until the asker has prepared his question properly.

I would rather have an option to put the question "on hold", i.e. it is not possible to answer the flagged question (the reasons should be very specific, preferably selected from a list) until the asker reformulates the question according to the rules. I think that the educational value would be greater (more of an encouragement than a slap).

http://www.proz.com/post/577464#577464

Yes. let's start 'educating' askers in the best interests of the whole community - including themselves!

MediaMatrix

Pages in topic:   [1 2 3] >

To report site rules violations or get help, contact a site moderator:

 Moderator(s) of this forum Jared Tabor [Call to this topic] Lucia Leszinsky [Call to this topic]

You can also contact site staff by submitting a support request »

### Displaying peer's kudoz rating in given field for agree/disagree comments

The leading translation software used by over 250,000 translators.

SDL Trados Studio 2017 helps translators increase translation productivity whilst ensuring quality. Combining translation memory, terminology management and machine translation in one simple and easy-to-use environment.

PDF Translation - the Easy Way

TransPDF converts your PDFs to XLIFF ready for professional translation. It also puts your translations back into the PDF to make new PDFs. Quicker and more accurate than hand-editing PDF. Includes free use of Infix PDF Editor with your translated PDFs.

P.O. Box 903
Syracuse, NY 13201
USA
+1-315-463-7323
ProZ.com Argentina
Calle 14 nro. 622 1/2 entre 44 y 45
La Plata (B1900AND), Buenos Aires
Argentina
+54-221-425-1266
ProZ.com Ukraine
6 Karazina St.
Kharkiv, 61002
Ukraine
+380 57 7281624