Mobile menu

Pages in topic:   [1 2 3] >
Displaying peer's kudoz rating in given field for agree/disagree comments
Thread poster: Vadim Poguliaev
Vadim Poguliaev  Identity Verified
Russian Federation
Local time: 23:23
English to Russian
Sep 9, 2007

That would make a sense. Agree/Disagree from a person with 500 kudoz in the area has much more weight, than from person with no points (in the area) at all. IMHO it will give an asker valuable data for evaluating the answers. What do you think?

Direct link Reply with quote
 

Sven Petersson  Identity Verified
Sweden
Local time: 21:23
English to Swedish
+ ...
QUALITY rather than QUANTITY! Sep 9, 2007

There are, in my opinion, 3 relevant quality criteria:

1. Average Kudoz earned per answer given
2. Average Kudoz awarded per answer agreed to
3. Average Kudoz awarded per answer disagreed to

Would a Reliability Index (RI) based on the above-mentioned point not be a better indicator than number of Kudoz earned? There is e.g. obviously a difference between somebody who earned 500 Kudoz with 125 answers and somebody who needed to submit 1000 answers to earn 500 Kudoz!

Proposed RI formula:

RI = 25*[Ke/Ag + Ka/Aa + (4*Ad-Kd)/Ad]/3

Ke = Kudoz earned (total, points)
Ka = Kudoz awarded to answers agreed to (total, points)
Kd = Kudoz awarded to answers disagreed to (total, points)
Ag = Answers given (total, number)
Aa = Answers agreed to (total, number)
Ad = Answers disagreed to (total, number)

Note 1: Ag, Aa and Ad must obviously each be larger than 0 to avoid divide by 0 problems, but that is not an unreasonable constraint. I would personally be in favour of that no RI was displayed if they were not above each 10.

Note 2: The formula gives equal weight to the 3 criteria; one could argue that they should be given different weights.


Direct link Reply with quote
 

Narcis Lozano Drago  Identity Verified
Spain
Local time: 21:23
Member (2007)
English to Spanish
+ ...
Not really... Sep 9, 2007

RI = 25*[Ke/Ag + Ka/Aa + (4*Ad-Kd)/Ad]/3

Ke = KudoZ earned (total, points)
Ka = KudoZ awarded to answers agreed to (total, points)
Kd = KudoZ awarded to answers disagreed to (total, points)
Ag = Answers given (total, number)
Aa = Answers agreed to (total, number)
Ad = Answers disagreed to (total, number)

Case A: 1000 answers, 3999 kudoz points
Case B: 1 answer, 4 points

The RI for Case B > Case A. That would be fair?

Also, if one could get Kudoz for giving agrees I think that would be the end of the Kudoz system. After all, one could go around giving agrees to all +3 agrees answers, even if they don't have the slightest idea about the field or the correct answer and still get an impressive Kudoz point record. I agree with the quality rather than quantity (I myself have a good acceptance rate), but I don't think these obscure formulae are the solution.

Regards.


Direct link Reply with quote
 

Kim Metzger  Identity Verified
Mexico
Local time: 14:23
German to English
Another good formula Sep 9, 2007

Narcis Lozano Drago wrote:

The RI for Case B > Case A. That would be fair?

Also, if one could get Kudoz for giving agrees I think that would be the end of the Kudoz system. After all, one could go around giving agrees to all +3 agrees answers, even if they don't have the slightest idea about the field or the correct answer and still get an impressive Kudoz point record. I agree with the quality rather than quantity (I myself have a good acceptance rate), but I don't think these obscure formulae are the solution.



I agree that it would be a good idea to somehow indicate how credible a peer commenter is and that it would only make sense to do so after a member has answered at least 100 questions maybe. The best indicator of credibility for both answerers and peer commenters that I have found is Kirill Semanov's formula:

Questions answered x 4 = a
Pro points earned divided by a = %

Example:
Questions answered - 4593
Points earned - 7763
4593 x 4 = 18372
7763 divided by 18372 = 42%

http://www.proz.com/topic/24865

According to this formula, Narcis, your reliability ratio is 61%, which is outstanding. The best ratio I've found so far is 65%.

P.S. the 4 indicates the maximum number of points possible for a correct answer. Maybe 3.5 would be a better number.


[Edited at 2007-09-09 15:22]


Direct link Reply with quote
 
teju  Identity Verified
Local time: 14:23
English to Spanish
+ ...
This might have been mentioned before... Sep 9, 2007

... in other threads about the same subject, but two things come to mind:

1- You could be fairly new in ProZ, and incredibly smart. Your kudoz number and/or any rating number would not reflect the quality of your answers, misleading the asker.

2- The asker doesn't always pick the best answer. And I'm not talking about shades of gray here. In my language pair, I've seen some things that give me chills. This is also a problem for the glossary.

Anyway, there's always the option of clicking on the person's name and checking out their profile, that's what I've done many times.


Direct link Reply with quote
 

Christel Zipfel  Identity Verified
Partial member (2004)
Italian to German
+ ...
I heartfully agree Sep 9, 2007

Kim Metzger wrote:

I agree that it would be a good idea to somehow indicate how credible a peer commenter is and that it would only make sense to do so after a member has answered at least 100 questions maybe.



BTW, I had already proposed several times to base the KudoZ ranking system on the realiability ratio rather than on the number of points, which alone doesn't mean anything, like Sven said.

This would also be much fairer for newcomers - with much less answers they can reach the same ratio as someone who participated since years - and should, in my opinion, hopefully also discourage those who actually jump onto every question hoping to earn some points.

I also agree that there must be a minimum number of questions answered - you cannot get a credible average out of 1, 2... 10 answers:-)

[Bearbeitet am 2007-09-09 18:12]


Direct link Reply with quote
 

Heike Behl, Ph.D.  Identity Verified
Local time: 13:23
Member (2003)
English to German
+ ...
numbers can be deceiving Sep 9, 2007

Although to some extent informative, a ratio based on these calculations would still not necessarily give an accurate picture.

People who choose to receive notifications of new questions are obviously at an advantage since the chance that they will provide the correct answer to the easier questions is pretty high.

People answering at a later point (or people not willing to answer easy "dictionary questions") will have no easy answers left, but maybe like the challenge of the more difficult questions with no or not the correct answer supplied by then. By default, the ratio would be not as good as for those in the above category, but you'd probably find the more reliable answerers in this category.

These are, of course, the extremes, and most people fall somewhere in between, but there are some extreme cases.

E.g. I came across the answers supplied by a translator with a high number of questions answered plus a high number of points received. This would no doubt result in a good ratio, however, the questions answered were so simple that I could answer most of them without even using a dictionary, and my language skills in that particular language are rather limited. I would also not have rated these questions as Pro.

Each answer should be evaluated on its own merits. Agrees/disagrees (particularly those with additional comments) can be very helpful, but should not be the decisive factor in choosing the best answer. A ratio next to the person's name would not - and maybe should not - make any difference for me.

Anybody participating regularly in Kudoz knows the more reliable people anyways, with or without ratios.


Direct link Reply with quote
 
xxxFrancis Lee
Local time: 21:23
German to English
+ ...
The onus should be on the Asker Sep 9, 2007

I appreciate what Vadim means, but if the Asker ultimately chooses an answer simply (or primarily) on the basis of peer agreements then that's their own fault. I sometimes feel there should be a disclaimer somewhere saying that just because an answer has a lot of Agrees, this is no guarantee of quality.

Vadim Poguliaev wrote:
IMHO it will give an asker valuable data for evaluating the answers


Personally, when I ask a question I don't care how many points an Asker has. Two reasons:

- There are colleagues on the GermEng site who have relatively high Kudoz points/rankings but who IMO are poor translators. A lot of their points are from a) guesses b) being the first to answer an easy question. There is in particular one individual who has a surprisingly high Kudoz rating - and to be honest, when she has in the past given me an Agree (no longer the case since I gave her a Disagree) my first thought was "oh no, respected colleagues will treat any Agree from her like a red flag"

- There are perhaps a couple of dozen colleagues (both German- and English-speaking) whose opinions I automatically respect due to past experience. And a few of these are not very active re. Kudoz, i.e. have relatively few points. Plus teju raises the issue of new Kudozians who have no points but - in certain cases - high-quality suggestions.

If anything, then perhaps a ratio (as opposed to pure points) would work - but that can also be misleading. My own ratio would be higher were it not for the many questions where I take the time to research and provide references - and then "lose" the points because someone else posted the same answer but a minute quicker. But this is a side-issue.

teju wrote:
Anyway, there's always the option of clicking on the person's name and checking out their profile, that's what I've done many times.


Exactly - hence the title of my post.


Direct link Reply with quote
 
xxxmediamatrix
Local time: 17:23
Spanish to English
+ ...
Nice idea! Won't work! Sep 9, 2007

I agree it would be a good idea to give some kind of numerical indicator of an answerer's (apparent) abilities in the field of the question, rather than just showing their specialist or working interest in the field as now.

The main problem - as I've pointed out elsewhere http://www.proz.com/topic/80531 and http://www.proz.com/topic/81118?post_id=641182 - is that so many askers do not correctly categorise their questions, often putting engineering or medical questions as 'law/patents', for example.

As a result, numerical indicators linked to the fields applied by the asker will often not correspond to the answerer's ability in the true field of the question (in just the same way as the specialist/working flags in the present system are often incorrect).

Sloppy askers shoot themselves in the foot every time they don't set the fields correctly. First because peers who rely on e-mail notifications to alert them to the existence of questions in their fields will never be alerted and hence will not jump in to help. Second, because those sloppy askers won't see valid 'worker/specialist' flags and be guided to selected the most appropriate answer.

A side issue here is that they also mess up everyone's KudoZ ranking - and hence our positions in the freelancer directory.

So, before trying to do anything to improve the chances of an asker picking guidance from the 'most reliable' answerers we really do need a mechanism to ensure that askers set up their questions properly in the first place.

MediaMatrix


Direct link Reply with quote
 
Vadim Poguliaev  Identity Verified
Russian Federation
Local time: 23:23
English to Russian
TOPIC STARTER
agreed, but... Sep 9, 2007

Introducing reliability ratio is good idea and it was discussed many times, but it will require some willpower from the staff. I doubt they will implement it any time soon.

There's lot of pros as well as contras, e.g. it leaves behind people, who give a lot of not for grading answers. Transition must be slow and as painless for community as possible.

My idea idea is much more simple. If they show answerer's kudoz rating in the field, why not show peers' rating as well? And it won't require serious changes in site policy.
As simple as that.

[Edited at 2007-09-09 18:07]


Direct link Reply with quote
 
xxxjacana54  Identity Verified
Uruguay
English to Spanish
+ ...
agree with teju's comments Sep 9, 2007

After a few days of visiting the Kudoz section, it's fairly easy to identify the answerers you can rely on, or whose style is more acceptable.
Sometimes people will provide answers saying "this may be another option" or "maybe you will find this useful". If we all become too conscious of statistics, this kind of answer, which can be very useful, will tend to disappear.


Direct link Reply with quote
 

Kim Metzger  Identity Verified
Mexico
Local time: 14:23
German to English
Not-for-points answers Sep 9, 2007

jacana54 wrote:

After a few days of visiting the Kudoz section, it's fairly easy to identify the answerers you can rely on, or whose style is more acceptable.
Sometimes people will provide answers saying "this may be another option" or "maybe you will find this useful". If we all become too conscious of statistics, this kind of answer, which can be very useful, will tend to disappear.


Yes, I agree with you jacana. This is one reason I proposed making it possible to opt out of the points race when you just want to provide some information that might help others find a good solution.

http://www.proz.com/topic/81997


Direct link Reply with quote
 

Kirill Semenov  Identity Verified
Ukraine
Local time: 22:23
Member (2004)
English to Russian
+ ...
Worth considering, but let's start from something more simple Sep 9, 2007

Hi Sven,

Your idea of including "agree/disagree" factor to the RI (or RR, reliability ratio) formula is basically good, but there are two problems:

1) theoretical: the exact weights assigned to agreed/disagreed answers are most likely empirical, so it may take a long time to estimate the exact numbers. I'm not going even try to do it from my first glance, but any weights assigned to this or that variable in an arbitrary or even empirical way are always doubtful or at least need a lot of observations;

2) practical: including "agrees/disagrees" into the formula means a great load on the ProZ server. I don't even mean recalculating everything "on-the-fly" which is not done for a while even for kudoZ points (which, as we all know, are updated automatically only once per day). But even once-a-day recalculation of the rating formula for each and every proZ member will take a huge amount of time.

For a while, I would start from the simplest idea of ratio: KudoZ points won/questions answered. Then we'll see. And the technical capabilities of such busy servers will probably rise with time.

To all those who concerned about trivia examples of only 1 question answered and whole 4 points gained - no, statistics is about large numbers, by definition, so no less than 50-100 questions must be answered by any proZian to enter any kind of RI/RR competition.

PS. Dear Kim, thank you for your support and enthusiasm about the matter


Direct link Reply with quote
 

Kim Metzger  Identity Verified
Mexico
Local time: 14:23
German to English
Members vote to place questions on hold Sep 9, 2007

mediamatrix wrote:

The main problem - as I've pointed out elsewhere http://www.proz.com/topic/80531 and http://www.proz.com/topic/81118?post_id=641182 - is that so many askers do not correctly categorise their questions, often putting engineering or medical questions as 'law/patents', for example.

So, before trying to do anything to improve the chances of an asker picking guidance from the 'most reliable' answerers we really do need a mechanism to ensure that askers set up their questions properly in the first place.

MediaMatrix


Poorly classified questions are indeed a problem, but just one among many. Questions asked without context is another major one as well as questions asked that have already been answered before. Not too long ago Jabberwock proposed placing sloppy questions on hold until the asker has prepared his question properly.

Jabberwock wrote:
I would rather have an option to put the question "on hold", i.e. it is not possible to answer the flagged question (the reasons should be very specific, preferably selected from a list) until the asker reformulates the question according to the rules. I think that the educational value would be greater (more of an encouragement than a slap).

http://www.proz.com/post/577464#577464


[Edited at 2007-09-09 19:22]


Direct link Reply with quote
 
xxxmediamatrix
Local time: 17:23
Spanish to English
+ ...
Excellent idea! Sep 9, 2007

Kim Metzger wrote:

mediamatrix wrote:

The main problem ... - is that so many askers do not correctly categorise their questions, often putting engineering or medical questions as 'law/patents', for example.

So, ... we really do need a mechanism to ensure that askers set up their questions properly in the first place.

MediaMatrix


Poorly classified questions are indeed a problem, but just one among many. Questions asked without context is another major one as well as questions asked that have already been answered before. Not too long ago Jabberwock proposed placing sloppy questions on hold until the asker has prepared his question properly.

I would rather have an option to put the question "on hold", i.e. it is not possible to answer the flagged question (the reasons should be very specific, preferably selected from a list) until the asker reformulates the question according to the rules. I think that the educational value would be greater (more of an encouragement than a slap).

http://www.proz.com/post/577464#577464


Yes. let's start 'educating' askers in the best interests of the whole community - including themselves!

MediaMatrix


Direct link Reply with quote
 
Pages in topic:   [1 2 3] >


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Displaying peer's kudoz rating in given field for agree/disagree comments

Advanced search






Wordfast Pro
Translation Memory Software for Any Platform

Exclusive discount for ProZ.com users! Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value

More info »
WordFinder
The words you want Anywhere, Anytime

WordFinder is the market's fastest and easiest way of finding the right word, term, translation or synonym in one or more dictionaries. In our assortment you can choose among more than 120 dictionaries in 15 languages from leading publishers.

More info »



All of ProZ.com
  • All of ProZ.com
  • Term search
  • Jobs