Pages in topic:   [1 2 3 4 5 6 7] >
Add a KudoZ "confidence/hit rating" field to each user profile (T)
Thread poster: Stuart Allsop
Stuart Allsop
Stuart Allsop  Identity Verified
Chile
Local time: 21:24
Spanish to English
+ ...
Sep 19, 2004

Kudoz points are a great way of judging the level of expertise of Proz members.... but only up to a point (no pun intended).

What I mean is, who can say if the opinion of a member with 3,000 Kudoz points is really worth more than a member with 300? Right now, there is just no way to figure that out easily.

My suggestion is to implement a simple system that would do just that.

It would be very valuable and quite easy to implement a system that gives us a mu
... See more
Kudoz points are a great way of judging the level of expertise of Proz members.... but only up to a point (no pun intended).

What I mean is, who can say if the opinion of a member with 3,000 Kudoz points is really worth more than a member with 300? Right now, there is just no way to figure that out easily.

My suggestion is to implement a simple system that would do just that.

It would be very valuable and quite easy to implement a system that gives us a much better feeling for the REAL value of someone's Kudoz and opinions. If the guy with 3000 points took five years to accumulate them, and had to answer 20,000 questions (getting 19,000 wrong), but the guy with 300 points joined last month, earned all his points from answering just 100 questions, 80 of which he got right, then clearly the guy with 300 points is much better than the guy with 3,000, and his opinion is vastly more valuable!

The Kudoz data base already tracks how many questions each of us has answered, on how many of those we were selected as the best answer, and how many points we earned as a result. It should therefore be very easy to add a "confidence rating" field to each user profile, as well as to the leadership rankings, showing the "hit rate" PERCENTAGE of answers selected as "best", and a "confidence rating" calculated as the total number of Kudoz divided by the total number of questions answerd, times 0.25 (this 0.25 factor is necessary, since you can get a maximum of 4 Kudoz points for a correct answer).

In my hypothetical example above, the guy with 3,000 Kudoz would have a percentage hit rating of correct answers of 5% and a confidence rating of 0.0375, while the newcomer would have a "hit rate" of 80% and a confidence rating of 0.75.

This would make it VERY much easier to figure out whose "agrees" and "disagrees" carry more wieght, and whose opinion is more trustworthy. It would also put the leadership ranking in a much more realistic perspective: Newcomers would be rated on their REAL skill, not just on how long they have been accumulating points.

I believe that the minor programming effort needed to implement this rating system would really level the expertise playing field, helping all of us to gauge the opinions and skills of others based on much more meaningful paramaters than only the total number of accumulated Kudoz.



[Subject edited by staff or moderator 2005-02-01 22:42]
Collapse


 
IanW (X)
IanW (X)
Local time: 01:24
German to English
+ ...
Excellent idea Sep 19, 2004

Hi Stuart and everyone else,

I think this is an excellent idea - and it would help to dissuade people from messing around with language pairing and subjects with which they are not sufficiently familiar.

However, there are other factors which need to be taken into account. For example, if I answer a German-English question, the chances are that six or seven other good answers will be posting, meaning that the chances are that my answer will not be chosen, even if it is
... See more
Hi Stuart and everyone else,

I think this is an excellent idea - and it would help to dissuade people from messing around with language pairing and subjects with which they are not sufficiently familiar.

However, there are other factors which need to be taken into account. For example, if I answer a German-English question, the chances are that six or seven other good answers will be posting, meaning that the chances are that my answer will not be chosen, even if it is good. On the other hand, if I answer a Gaelic-English one, I may well be one of just two people who answer. Or the only one.

Still, I think this is definitely an idea worth pursuing.

All the best


Ian

[Edited at 2004-09-19 12:08]
Collapse


 
Derek Gill Franßen
Derek Gill Franßen  Identity Verified
Germany
Local time: 01:24
German to English
+ ...
In memoriam
I agree... Sep 19, 2004

I agree with Ian; while this might not always indicate how competent a member actually is, it would be interesting to see (and give a bit of guidance). Personally, I enjoy viewing the statistics now and again - and this would be a great addition (together with break-downs in months, weeks and days like I've also suggested).

Good idea!


 
Anjo Sterringa
Anjo Sterringa  Identity Verified
Netherlands
Local time: 01:24
English to Dutch
+ ...
Not necessary to refine the system Sep 19, 2004

I hope you do not mix up translating skills with KudoZ skills. It only shows that at some point you answered some of your fellow translators' questions, which is nice, and perhaps that you know a lot about certain specialised fields. It does not mean you can actually translate a sentence. I think KudoZ are fun, and that the KudoZ rating should not be taken too seriously (as has been pointed out many times before and again in the pr... See more
I hope you do not mix up translating skills with KudoZ skills. It only shows that at some point you answered some of your fellow translators' questions, which is nice, and perhaps that you know a lot about certain specialised fields. It does not mean you can actually translate a sentence. I think KudoZ are fun, and that the KudoZ rating should not be taken too seriously (as has been pointed out many times before and again in the previous posting). Please, let's keep it that way! I think adding a 'trustworthiness percentage' would give a false impression, and it might even stop people adding another option for the translator that asked the question.Collapse


 
Yuri Smirnov
Yuri Smirnov  Identity Verified
Local time: 03:24
English to Belarusian
+ ...
YES! Sep 19, 2004

anjoboira wrote:

I hope you do not mix up translating skills with KudoZ skills.


Absolutely!
I was literally SHOCKED a couple of times when I got to read the translations of people high up in the Kudoz rating. They were NO translators at all, like had no idea different languages have different word order etc.


 
Michele Fauble
Michele Fauble  Identity Verified
United States
Local time: 17:24
Member (2006)
Norwegian to English
+ ...
Kudoz points do not (always) reflect translation skill Sep 19, 2004

This is something I have learned during the short time I have been answering Kudoz questions. Many of the questions (at least in my language pairs) are asked by translators translating out of their native language. Many of the answers are also suggested by answerers without native ability in the language. These answers are often too literal and not what a native speaker would write, or even understand. Yet these answers are often the ones awarded the points by an asker unable to judge the sugges... See more
This is something I have learned during the short time I have been answering Kudoz questions. Many of the questions (at least in my language pairs) are asked by translators translating out of their native language. Many of the answers are also suggested by answerers without native ability in the language. These answers are often too literal and not what a native speaker would write, or even understand. Yet these answers are often the ones awarded the points by an asker unable to judge the suggested answers as non-native.

To help askers to choose answers I think it would be helpful if the answerer's native language was indicated.


My thoughts,
Michele Fauble
Collapse


 
Klaus Herrmann
Klaus Herrmann  Identity Verified
Germany
Local time: 01:24
Member (2002)
English to German
+ ...
Hit ratio permits to reset Kudoz on a regular basis Sep 19, 2004

Acutally, if a hit ratio was kept, it would be possible to reset KudoZ points at regular intervals in order to give new members a chance. (Unless a KudoZ player with 100,000 points takes a 3-year sabbatical, there's no chance in the world for a new member to catch up.)

OTOH I don't think there's a compelling relation between points and quality. After some time you'll know how to weigh contributions by different members. When in doubt, looking up the answers of a peer grader (accessi
... See more
Acutally, if a hit ratio was kept, it would be possible to reset KudoZ points at regular intervals in order to give new members a chance. (Unless a KudoZ player with 100,000 points takes a 3-year sabbatical, there's no chance in the world for a new member to catch up.)

OTOH I don't think there's a compelling relation between points and quality. After some time you'll know how to weigh contributions by different members. When in doubt, looking up the answers of a peer grader (accessible through his/her profile) can be quite revealing.
Collapse


 
Kim Metzger
Kim Metzger  Identity Verified
Mexico
Local time: 18:24
German to English
Rating real value of KudoZ points Sep 19, 2004

Ian Winick wrote:

However, there are other factors which need to be taken into account. For example, if I answer a German-English question, the chances are the six or seven other good answers will be posting, meaning that the chances are that my answer will not be chosen, even if it is good. On the other hand, if I answer a Gaelic-English one, I may well be one of just two people who answer. Or the only one.



Good point Ian. While I agree that we ought to look for ways to help askers know who\'s got a track record for reliability as opposed to just a track record for answering questions, we need to be careful how this reliability is determined.

Factors going into establishing a reliability ratio should include some combination of confidence level and peer grading.

What was the confidence level indicated by the answerer on his confidence bar? More points would be deducted for an incorrect answer (one that received no agrees, only neutrals and disagrees?) given with a high confidence level.

There have been many discussions of this topic over the years. Just enter \"reliability\" in the search box and tick \'forums.\'


 
lien
lien
Netherlands
Local time: 01:24
English to French
+ ...
Don't change anything Sep 19, 2004

Stuart Allsop wrote:

This would make it VERY much easier to figure out whose "agrees" and "disagrees" carry more wieght, and whose opinion is more trustworthy. It would also put the leadership ranking in a much more realistic perspective: Newcomers would be rated on their REAL skill, not just on how long they have been accumulating points.



To figure out who to trust with the answers, that is something you learn as you are on kudoz helping others with theirs translations.

If you are that good people will notice you sooner or later.

Proz is primarily a place to help each others.

As for the new, they can go and check the site of anybody.

"the leadership ranking"...


 
Christine Andersen
Christine Andersen  Identity Verified
Denmark
Local time: 01:24
Member (2003)
Danish to English
+ ...
KudoZ is a game, and it's also a bit of a lottery Sep 19, 2004

Some of the best translators may not have time to 'play' on KudoZ... while being a KudoZ leader says as much about your collection of dictionaries and on-line internet connection as your ability to use them.

I 'disconnect' KudoZ when I'm really busy, and others probably do too. I have chosen simply never to get e-mails about monolingual English - they interrupt my work too often. I deeply admire the people who do take the time for that forum - I benefit from it a lot, and could undo
... See more
Some of the best translators may not have time to 'play' on KudoZ... while being a KudoZ leader says as much about your collection of dictionaries and on-line internet connection as your ability to use them.

I 'disconnect' KudoZ when I'm really busy, and others probably do too. I have chosen simply never to get e-mails about monolingual English - they interrupt my work too often. I deeply admire the people who do take the time for that forum - I benefit from it a lot, and could undoubtedly earn points if I had time.

And as Ian says, there are often several very good answers to a question, but only one person gets points.

Sometimes four points are awarded for a very elementary answer that really only deserves one or two, and occasionally most of us think the 'wrong' answer has been selected.

Some languages simply do not generate very many questions...

I think it would be a mistake to draw too many conclusions from leadership in the KudoZ game. The points are an indicator, but not usable as statistics.

All members are invited to show on their profiles how many years of experience they have as translators, and it is easy to see that someone who has been a Platinum member for three or four years has had more chances than someone who has only 'been there' a couple of months. That is why the 'last month' and 'last three months' tables are shown.

Keep it as a game. It's a great game and great exercise, but it should not be stretched beyond its limits or used in ways it is not suited for.


[Edited at 2004-09-20 09:07]
Collapse


 
RobinB
RobinB  Identity Verified
United States
Local time: 19:24
German to English
The KudoZ lottery Sep 19, 2004

Christine,

I couldn't agree with you more. The lottery element of KudoZ significantly impairs its potential usefulness.

Looking at German-English finance questions, over half the answers given are misleading, inappropriate or downright wrong (and often with a High or Highest rating!). So it's hardly surprising that a large proportion of the answers selected by askers are also wrong.

"I think it would be a mistake to draw too many conclusions from leadership
... See more
Christine,

I couldn't agree with you more. The lottery element of KudoZ significantly impairs its potential usefulness.

Looking at German-English finance questions, over half the answers given are misleading, inappropriate or downright wrong (and often with a High or Highest rating!). So it's hardly surprising that a large proportion of the answers selected by askers are also wrong.

"I think it would be a mistake to draw too many conclusions from leadership in the KudoZ game."

Indeed. I don't think that anybody could or should infer a link between KudoZ points and translation ability.

"Keep it as a game. It's a great game and great exercise, but it should not be stretched beyond its limits or used in ways it is not suited for."

Brilliant. Can we have that added to the KudoZ FAQs?

Robin
Collapse


 
Kirill Semenov
Kirill Semenov  Identity Verified
Ukraine
Local time: 02:24
Member (2004)
English to Russian
+ ...
I evaluate this ratio Sep 19, 2004

Dear Stuart,

First of all, you are not the first who proposes such a feature.
A lot of people (including myself) proposed to use the ratio of given and won answers as a relative or additional indicator of kudoZers' expertise and credibility. I still remember that when I first proposed it (more than a year ago, when I was a newbie on proZ), I faced quite a harsh responses from older members who suspected that my proposal might "overthrown the rulers" (i.e. drastically change th
... See more
Dear Stuart,

First of all, you are not the first who proposes such a feature.
A lot of people (including myself) proposed to use the ratio of given and won answers as a relative or additional indicator of kudoZers' expertise and credibility. I still remember that when I first proposed it (more than a year ago, when I was a newbie on proZ), I faced quite a harsh responses from older members who suspected that my proposal might "overthrown the rulers" (i.e. drastically change the ratings). I'm glad to see that yours proposal did not produce such a reaction!

As for me, personally, I've been using this approach for my own evaluations for quite a long time by now (since the moment when they started to show the number of questions asked under an Asker's name). It's not a final analysis, but still this ratio looks very important to me. It works very good for active kudoZers. It shows the credibility of our colleagues (not mentioning our own). I'm not sure if this ratio should be the basis of kudoZ leaders ratings, but I have no doubts that showing it explicitly would diminish the number of guesswork and (I'll put it straight) utterly stupid answers in kudoZ area.

To conclude, I would like to note that the feature is described in proZ FAQs (it's called "reliability" there if I'm not mistaken), but still is not implemented.

My best regards to everyone.
Collapse


 
Kirill Semenov
Kirill Semenov  Identity Verified
Ukraine
Local time: 02:24
Member (2004)
English to Russian
+ ...
Exactly, Kim Sep 19, 2004

Kim Metzger wrote:
What was the confidence level indicated by the answerer on his confidence bar? More points would be deducted for an incorrect answer (one that received no agrees, only neutrals and disagrees?) given with a high confidence level.


The formula described (but not implemented) at proZ FAQ includes the 'confidence level' which answerers put in their answer. Obviously, a `lost' answer given with higher or highest confidence should suffer the most penalty in such a system. We all know examples of super-confident answers which... lead nowhere.

And some accident fluctuations (I noted that many kudoZers consider these very important, though I think these are nothing more than random fluctuations) will compensate themselves in the long run. The law of big numbers, etc.

[Edited at 2004-09-19 13:36]


 
Aleksandr Okunev (X)
Aleksandr Okunev (X)
Local time: 03:24
English to Russian
So what? Sep 19, 2004

Good point Ian. .... While I agree ... There have been many discussions of this topic over the years.


Well, it would be very nice to know if these years of discussions yielded anything. I've been on Proz just for a little while, but I already know a couple of people who pollute Kudoz with their stupid answers, and I know those, whose answers I must take for granted even if I see seemingly better answers.

However, I am a guy who started translating over 10 years ago, I have seen 'pals' and situations of every hue. What about clients and newbies who get impressed with the 4-digit scores in 17 languages, if you know what I mean?

It's time to think of what people want on the receiving end of Kudoz, or there'll be another kid on the block soon, who'll be doing it better.

Have fun (and Kudoz)


 
Sarah Downing
Sarah Downing  Identity Verified
Local time: 20:24
German to English
+ ...
Kudoz Points don't always represent translation skills Sep 19, 2004

I agree with what a lot of people have been saying here.

1. Just because your answer doesn't get chosen doesn't mean it was wrong (sometimes wrong answers get chosen due to lack of knowledge and sometimes answers are simply chosen based on personal preference).
2. I currently have 150 points and have been a proz member for a couple of years. So why, you might ask, do I have so few points - it's quite simple: I also regard kudoz as I game, I don't really give a **** about the
... See more
I agree with what a lot of people have been saying here.

1. Just because your answer doesn't get chosen doesn't mean it was wrong (sometimes wrong answers get chosen due to lack of knowledge and sometimes answers are simply chosen based on personal preference).
2. I currently have 150 points and have been a proz member for a couple of years. So why, you might ask, do I have so few points - it's quite simple: I also regard kudoz as I game, I don't really give a **** about the points, I just answer questions now and then because I find them interesting or because I think I may be able to help. And as someone else said - I often don't have time to answer the questions because I have so much other work to do ...

Ultimately, the best way to judge a translator's skills is not by some points system but by giving them a test translation.

Have a nice Sunday!

Sarah
Collapse


 
Pages in topic:   [1 2 3 4 5 6 7] >


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Add a KudoZ "confidence/hit rating" field to each user profile (T)






Protemos translation business management system
Create your account in minutes, and start working! 3-month trial for agencies, and free for freelancers!

The system lets you keep client/vendor database, with contacts and rates, manage projects and assign jobs to vendors, issue invoices, track payments, store and manage project files, generate business reports on turnover profit per client/manager etc.

More info »
Wordfast Pro
Translation Memory Software for Any Platform

Exclusive discount for ProZ.com users! Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value

Buy now! »