| Pages in topic: [1 2] > | 'hit rate' or 'points to closed answers ratio' (PTA) Thread poster: lbone
|
|---|
lbone 
China
Local time: 03:54
Member (2006)
English to Chinese
+ ...
Those of us often in KudoZ can often see some bad answerers who answer very fast on almost all questions in some periods with little professional thoughts and usually without references or with references that can be easily found but often not convincing. They may also be translators with some experience. They may be able to come up with better answers for some of the bad answers they provided if they really want to. But to gain total points faster, they chose a faster approach that makes peers ... See more Those of us often in KudoZ can often see some bad answerers who answer very fast on almost all questions in some periods with little professional thoughts and usually without references or with references that can be easily found but often not convincing. They may also be translators with some experience. They may be able to come up with better answers for some of the bad answers they provided if they really want to. But to gain total points faster, they chose a faster approach that makes peers uncomfortable. Sometimes they can get some points when some simple questions show up or when they really know it clearly or not clearly (because they are translators, I do not want to deny it). I am not talking about those professional and decent answerers who really know the answers solidly and answer fast for some questions. You know the difference.
Most of us know they are ruining KudoZ, because thoughtless answers are seldom useful, but actually often misleading if the askers are novices. Without these answerers or without their answers, KudoZ can be better.
I am not going to say removing any people or anything. But I suggest some measures be taken so people will not be encouraged to answer questions in that way.
People answer questions often because high total KudoZ points can help them gain better positions in the freelancer directory. Here are my suggestions to improve the system:
************
Suggestion 1: Add an index showing the 'hit rate' or 'points to closed answers ratio' (PTA?).
************
The denominator for both can be qualified closed pro questions with points (let's simply name it 'qualified closed questions' - QCQs). So the following questions will not be added to the denominator (or they are not 'qualified closed questions'):
a) 'open' questions
b) 'just_closed' questions
c) 'easy' questions (points from ‘easy’ questions do not contribute to PRO-points)
d) ‘NP’ questions (no points will be given)
e) questions for which answers by this answerer were submitted after the questions have been closed (KudoZ does not prevent an answerer from adding an answer to an already closed question, but obviously, the answerer has no chance to gain points from such answers).
So 'hit rate' means the ratio of the QCQs this answerer won (by being selected as 'most helpful') to all QCQs.
While 'points to closed answers ratio' (PTA?) means the ratio of all points gained from the QCQs this answerer won to all QCQs.
I personally like PTA.
I suggest that a 'hit rate' or PTA be listed together with a member's KudoZ points in all occasions where KudoZ points really mean something - such as in his/her profile, in all the directories or search results.
************
Suggestion 2: The system should prevent those with unbearably low 'hit rates' or PTAs from being ranked high based on KudoZ points.
************
My specific suggestions:
1) Do not do anything to those with overall KudoZ points below 200. They are novice KudoZ answerers who need to take time to learn the game rules in KudoZ.
2) For those who earned more than 200 KudoZ points, if their 'hit rates'/PTAs are below some unbearably low threshold, their KudoZ points should be replaced with 0 when KudoZ points participate in ranking.
My suggested thresholds are:
hit rate threshold: 0.3
PTA threshold: 1.2
We can calculate some typical members’ 'hit rates' and PTAs. For a clearer understanding, we define a 'raw ratio', which means the ratio of PRO-points to answers. PRO-points and answers are clearly shown in each member’s profile.
I checked the parameters of 5 typical members that I guess are either typical or can be influenced by this index. The following is the first (she has clearly agreed).
Sample 1 (answers 65, PRO-points 71)
raw ratio: 71/65=1.09
won-18 (not including 'easy'), open-21, NP-0, just_closed-1, easy-1, answers submitted after questions closed-2, QCQ=65-21-0-1-1-2=40
hit rate: 18/40=0.45
PTA: 71/40=1.775
PTA/raw ratio=1.775/1.09=1.63
Note: The typical value for “PTA/raw ratio” for a mature answerer is usually 1.15-1.3. Her high rate on it is due to the many open questions. Nothing really important is wrong to this answerer.
I do not want to disclose the details of other four except when site stuff need it, and I guess it is not necessary to do that as the method is clear and you can easily find any member you are interested in and calculate his/her scores in the same way. Now I just make a summary:
1) 2 failed. One is someone I clearly believe as not responsible in answering questions. With his high total KudoZ points, he is listed in a relatively good position in the directory, but his PTA is 0.922. Another is a KudoZ 'novice' whose total points are within 100 and PTA is 0.887. The reasons for the second to have a low PTA, based on my analysis, is a combination of just barely over average skills (based on my own standard) and unfamiliarity with KudoZ game rules - though he spent time and answered seriously, he too often answered in a way that his answers have little chance to be acknowledged as the most helpful ones - he just very seldom answered the many questions that should have clear specific answers, or answered questions that should have already well-accepted translations with the translations he unnecessarily originally created, or he too often answered questions that tend to have many answers and his answers were really hard to be selected as the best comparing with other also good answers convincingly presented, or he too often answered questions from those askers that experienced answerers knowingly skip.
2) 3 (including the above one with details) clearly passed with PTAs between 1.71 and 2.36. They are just normal answerers new to KudoZ.
3) The first-page PTA for a typical well-known good answerer is usually between 2.5 and 3.6.
Final comments:
1) This scheme will not make trouble to general answerers who conduct well. It adds an index that is meaningful to some extent. We can understand if someone somewhat low in it may not like to see the difference be published, but we also know those seriously and constantly providing quality help to others in KudoZ deserve the respect, and I hope this index can differentiate them from others.
2) If someone's raw ratio (PRO-points/answers) is clearly above 1, the scheme is harmless to him/her, except that s/he does not want to see the fact that his/her PTA is lower than others be revealed. But as we have been facing many facts (such as agencies have to face the fact that their Blue Board records are open to public), facing a new one can be acceptable.
3) New answerers need time to be familiar with the KudoZ system, if s/he really cares about his/her KudoZ points-based ranking. This index is meaningful even for them - if their PTAs are low, which means the possibilities askers give points to them are low, it usually means the possibilities that they are really helpful for others are low. It just indicates a fact that may deserve to be revealed, though still hasn't been expressly indicated before.
4) It works by (1) showing this index as an adjusting reference to KudoZ points to those for whom we expect KudoZ points are meaningful, and (2) it can be used to prevent someone not really helpful in KudoZ from being ranked at a position that Prozians widely believe s/he should not be there.
5) Those who were not serious enough before can regain respect if they are serious in KudoZ later. Their previous help to others are still in record and bringing respect to them.
6) Some people hide answers AFTER questions are closed. I suggest that they can hide answers, but the answers hidden AFTER questions are closed still be included in QCQs to discourage a possible rush of such actions.
[Edited at 2008-10-29 18:06] ▲ Collapse
| | | | Yaotl Altan
Mexico
Local time: 13:54
Member (2006)
English to Spanish
+ ...
I agree in general terms but it sounds too complex.
In our Kudoz stats we have the number of questions answered, answers accepted and acceptance rate. An easiest way to sort translators could be the acceptance rate besides the number of Kudoz. colleagues, users, agencies, et al. could sort translators according to one of these fields, the one they want, Kudoz or acceptance rate. They could toggle between one or another.
| | | | Jussi Rosti
Finland
Local time: 22:54
Member (2005)
English to Finnish
+ ...
| Complex, and not very scalable | Oct 29, 2008 |
Although I agree with the idea, I find this much too complex. Moreover, the solution is not scalable, because the KudoZ languages are of different size. So, if in some very active language a low quality answer is often overridden with an answer of better quality, this may not be the case in such a language, where activity is so low that a competing answer is a rarity.
In one case, the hasty answerer will get bad PTA ratio, while in the other the PTA ratio may well be better than ave... See more Although I agree with the idea, I find this much too complex. Moreover, the solution is not scalable, because the KudoZ languages are of different size. So, if in some very active language a low quality answer is often overridden with an answer of better quality, this may not be the case in such a language, where activity is so low that a competing answer is a rarity.
In one case, the hasty answerer will get bad PTA ratio, while in the other the PTA ratio may well be better than average.
But please, keep on thinking about more fair ways to use the KudoZ information. There is certainly much to improve. ▲ Collapse
| | | | Yaotl Altan
Mexico
Local time: 13:54
Member (2006)
English to Spanish
+ ...
Jussi Rosti wrote:
.... Moreover, the solution is not scalable, because the KudoZ languages are of different size. So, if in some very active language a low quality answer is often overridden with an answer of better quality, this may not be the case in such a language, where activity is so low that a competing answer is a rarity....
Yes, this is a very important point. Some pairs would need to receive a "bonus" factor in order to compete with highly demanded pairs. For instance, how could a Euskera>Japanese could face a English>Spanish pair?
Just look at the FIFA ranking. If Nicaragua beats 1-0 Solomon Islands, Nicaragua gets some points but if Nicaragua defeats 3-0 Italy at Napoli, then Nicaragua could get a lot of points. This is because the difference of levels in soccer from Europe to Oceania. A similar situation happens with Jussi's point.
| | |
|
|
|
lbone
China
Local time: 03:54
Member (2006)
English to Chinese
+ ...
TOPIC STARTER | Computer does the complex part | Oct 29, 2008 |
Yaotl Altan wrote:
complex
We human do not need to calculate anything - ProZ.com can do it automatically. We just need to read results. We do not need to care how complex it is produced. The only thing we need to know is the final value of this quality index should be as accurate as possible.
Yaotl Altan wrote:
I agree in general terms but it sounds too complex.
In our Kudoz stats we have the number of questions answered, answers accepted and acceptance rate. An easiest way to sort translators could be the acceptance rate besides the number of Kudoz. colleagues, users, agencies, et al. could sort translators according to one of these fields, the one they want, Kudoz or acceptance rate. They could toggle between one or another.
Many of those reading search results do not know there could be tricks behind KudoZ points, but they may be good customers.
Jussi Rosti wrote:
Although I agree with the idea, I find this much too complex. Moreover, the solution is not scalable, because the KudoZ languages are of different size. So, if in some very active language a low quality answer is often overridden with an answer of better quality, this may not be the case in such a language, where activity is so low that a competing answer is a rarity.
In one case, the hasty answerer will get bad PTA ratio, while in the other the PTA ratio may well be better than average.
We seldom need to compare people working on different pairs. We already have KudoZ points for each pair, so we can also have separate PTAs for each pair. We only need to check the differences between PTAs in the same pair. Usually we do not need to check PTA differences across pairs.
Most people work in active languages. So if PTA works, it can be beneficial for most members. This is a good enough reason. For some 'small' languages, PTAs still differ to some extent.
[Edited at 2008-10-29 19:13]
| | | | | There is a less complicated formula to use - just multiply | Oct 29, 2008 |
As it was pointed out before, everybody can see his/her own KudoZ statistics in his/her profile, the reliability ratio is there, it is called "acceptance rate" and it is a percentage figure (so 65.14 means 65.14%, or 0.6514).
If for directory ranking purposes, we want to use not only the number of KudoZ points but also the reliability ratio (how many answers were given in relation to the points gained), the easiest way is to MULTIPLY the two figures and use the result for ranking. <... See more As it was pointed out before, everybody can see his/her own KudoZ statistics in his/her profile, the reliability ratio is there, it is called "acceptance rate" and it is a percentage figure (so 65.14 means 65.14%, or 0.6514).
If for directory ranking purposes, we want to use not only the number of KudoZ points but also the reliability ratio (how many answers were given in relation to the points gained), the easiest way is to MULTIPLY the two figures and use the result for ranking.
I think this is the best of both worlds.
Using only the KudoZ point total omits the information about reliability. (It also gives unfair advantage to the aggressive, machine-gun style point-hunters, and disadvantageous for newcomers.)
Using only the reliability ratio omits the information about level of activity. (It would give unfair advantage to people that answer very few questions, and it may discourage people from answering questions.)
This combination has no problem for scaling (it can be used for any high number of points), and it works in a fair way, at least I found it fair. I calculated a few scenarios, and I liked what I saw.
If people want it, I can type up a few examples here, but I guess anyone can do it. ▲ Collapse
| | | | Yaotl Altan
Mexico
Local time: 13:54
Member (2006)
English to Spanish
+ ...
Katalin Horvath McClure wrote:
As it was pointed out before, everybody can see his/her own KudoZ statistics in his/her profile, the reliability ratio is there, it is called "acceptance rate" and it is a percentage figure (so 65.14 means 65.14%, or 0.6514).
If for directory ranking purposes, we want to use not only the number of KudoZ points but also the reliability ratio (how many answers were given in relation to the points gained), the easiest way is to MULTIPLY the two figures and use the result for ranking.
I think this is the best of both worlds.
Using only the KudoZ point total omits the information about reliability. (It also gives unfair advantage to the aggressive, machine-gun style point-hunters, and disadvantageous for newcomers.)
...
I like it, I have 4943 Kudoz pints with an acceptance rate of 42.2%.
4943 * 0.422 = 2085 "new" points. What about you?
So, we could have three ways to sort translators rankings according to everyone needs and desires: Kudoz total, acceptance rate and Katalin's proposal Kudoz total*acceptance rate.
It would be great if the default sort rankings are based according to Katalin's proposal.
[Edited at 2008-10-29 20:37]
| | | | dd dd
Local time: 03:54
English to Chinese
+ ...
| Ibone, you truly have "an attitude for research". :) | Oct 30, 2008 |
BTW, I am the Guinea pig used in sample 1.
As Ibone said: -
"Some people hide answers AFTER questions are closed. I suggest that they can hide answers, but the answers hidden AFTER questions are closed still be included in QCQs to discourage a possible rush of such actions."
I think this is something the Proz administrators probabaly need to pay attention to. In my opinion, the answers hidden (even before the question is closed, as long as the answerer ha... See more BTW, I am the Guinea pig used in sample 1.
As Ibone said: -
"Some people hide answers AFTER questions are closed. I suggest that they can hide answers, but the answers hidden AFTER questions are closed still be included in QCQs to discourage a possible rush of such actions."
I think this is something the Proz administrators probabaly need to pay attention to. In my opinion, the answers hidden (even before the question is closed, as long as the answerer has DECIDED to hit the "submit" button) should still be counted in the hit rate, PTA or any other such indexes.
[Edited at 2008-10-30 05:53] ▲ Collapse
| | |
|
|
|
Jussi Rosti
Finland
Local time: 22:54
Member (2005)
English to Finnish
+ ...
| The simple formula is good, submit a support ticket | Oct 30, 2008 |
lbone wrote: Yaotl Altan wrote:
complex We human do not need to calculate anything - ProZ.com can do it automatically. We just need to read results. We do not need to care how complex it is produced. The only thing we need to know is the final value of this quality index should be as accurate as possible.
But I'd say the formula used should be transparent enough for people to understand it.
I like Katalin's formula, it's quite understandable and it takes into account the two most important factors (activity of the translator and quality of the answers).
Maybe, Katalin, you would like to submit a support ticket to the site staff so that they can consider revising the sort order?
| | | | Jussi Rosti
Finland
Local time: 22:54
Member (2005)
English to Finnish
+ ...
| One important danger to consider | Oct 30, 2008 |
I just expressed my enthusiasm for Katalin's proposal. However, if not-chosen answers will lower your ranking this may lead to situation, where people are reluctant to answer if they for some reason fear that their answer will not be chosen. This would take a lot of good answers away.
One example. I see that there is a good answer already. I'm 90% sure it will be chosen, as it is good, but I know I have a reasonable other option. In current system I would post it with a comment "ano... See more I just expressed my enthusiasm for Katalin's proposal. However, if not-chosen answers will lower your ranking this may lead to situation, where people are reluctant to answer if they for some reason fear that their answer will not be chosen. This would take a lot of good answers away.
One example. I see that there is a good answer already. I'm 90% sure it will be chosen, as it is good, but I know I have a reasonable other option. In current system I would post it with a comment "another option to consider". If the calculation formula was changed, I would not post it at all.
Other examples are answers with low confidence, which may help the answerer, but would be "too risky" in the new ranking model.
Especially in low-activity pairs (all of my pairs, for example!) this might kill the low volume almost completely
So, this should not be changed hastily, without thinking. ▲ Collapse
| | | | lbone
China
Local time: 03:54
Member (2006)
English to Chinese
+ ...
TOPIC STARTER | KudoZ points still dominate | Oct 30, 2008 |
Jussi Rosti wrote: I just expressed my enthusiasm for Katalin's proposal. However, if not-chosen answers will lower your ranking this may lead to situation, where people are reluctant to answer if they for some reason fear that their answer will not be chosen. This would take a lot of good answers away. One example. I see that there is a good answer already. I'm 90% sure it will be chosen, as it is good, but I know I have a reasonable other option. In current system I would post it with a comment "another option to consider". If the calculation formula was changed, I would not post it at all. Other examples are answers with low confidence, which may help the answerer, but would be "too risky" in the new ranking model. Especially in low-activity pairs (all of my pairs, for example!) this might kill the low volume almost completely So, this should not be changed hastily, without thinking.
It is clear that some people knowingly use tricks to increase their quality index. Their abilities are not higher than those brighter answerers who do not use tricks and somewhat lower in the index. So I just suggest this index to be used as a reference only. A technical translator tends to get higher score in this index than a literary translator. But we seldom need to compare between two translators working in different fields. KudoZ point should still be the first factor. Actually, I know many outsourcers will read into the details of KudoZ answers directly.
This index can give outsourcers a fast reference, not a decision. A new choice is no harm and can be better than no choice. This is a good enough reason.
The reason I submit such a complex 'model' is a simple ratio as suggested by Katalin is not fair for newcomers. Newcomers usually have relatively too many open questions as against their answer total. So their ratios as decided by Katalin’s model are lower due to the big portion of open questions that contribute no points. But they need to be spotted early. Those listed in the first page can often be busy, or their rates are high. So a model to find good newcomers is meaningful.
As long as KudoZ points are used for ranking, people will answer questions. Because like it or not, KudoZ points decide where they are in the list.
| | | | | To second Jussi | Oct 30, 2008 |
Right, people could start avoiding entering their answers for fear of lowering their standing, should the Asker choose another answer for any reason. As I see it, the outcome would be two-fold:
1. The Asker has fewer options to choose from
2. People who won't take the risk and still want to assist the Asker in some way, will post their comments (answers, in fact) in any field they see on the page, thus violating the site rules. Do we want to encourage that?
| | |
|
|
|
Yaotl Altan
Mexico
Local time: 13:54
Member (2006)
English to Spanish
+ ...
| Already working | Oct 30, 2008 |
lbone wrote:
...A technical translator tends to get higher score in this index than a literary translator. But we seldom need to compare between two translators working in different fields. KudoZ point should still be the first factor. ...
But this is already working when you specify the proper field, isn't it?
| | | | megane_wang
Spain
Local time: 21:54
Member (2007)
English to Spanish
+ ...
| I agree with Iza | Oct 30, 2008 |
Iza Szczypka wrote:
Right, people could start avoiding entering their answers for fear of lowering their standing, should the Asker choose another answer for any reason. As I see it, the outcome would be two-fold:
1. The Asker has fewer options to choose from
2. People who won't take the risk and still want to assist the Asker in some way, will post their comments (answers, in fact) in any field they see on the page, thus violating the site rules. Do we want to encourage that?
I think that the most important element of KudoZ is to provide fast and accurate help to the askers.
I feel ike that sometimes people tend to think too much about points; this already has bad consequences, since there exist people who choose an answer from someone they know about, or someone from their country, or whatever, instead of checking the suggested answers and trying to figure out why an answer is ok or not for their own question. Sometimes, this is discouraging enough... to filter out that person and add a comment to the wrong entry in the glossary resulting of that attitude, hoping that someone that looks for help also finds the comments and uses better judgement.
If we give even more relevance to this acceptance ratio, I think that many good answerers would just not answer. And that would be a real pity for GOOD ASKERS too !
Ruth @ MW
| | | | | I am not sure about the effects, Jussi - perhaps one more factor is needed | Oct 30, 2008 |
Jussi Rosti wrote:
I just expressed my enthusiasm for Katalin's proposal. However, if not-chosen answers will lower your ranking
Yes they will, but it is proportional. The more you participate, the effect will be less and less (unless you are one of those point-hunters that just post any guesses they may come up with).
this may lead to situation, where people are reluctant to answer if they for some reason fear that their answer will not be chosen. This would take a lot of good answers away.
I don't think it will take lot of good answers away. Again, think about the math (try your own numbers, for example).
One example. I see that there is a good answer already. I'm 90% sure it will be chosen, as it is good, but I know I have a reasonable other option. In current system I would post it with a comment "another option to consider". If the calculation formula was changed, I would not post it at all.
Well, I see this situation a bit differently.
If you think the answer that is posted, is correct, fits the given context, then you would agree to it, right? Even if you have a slight modification that you would prefer in the chosen answer, you could also put that in the Agree field. If the suggestion you are about to make is a good answer in a different context, you may want to put it into the Reference field. Postings references has nothing to do with points (and rankings).
Other examples are answers with low confidence, which may help the answerer, but would be "too risky" in the new ranking model.
Well, this is an interesting point. One way of looking at it is this:
If I am not confident, that what I have to offer is helpful, I can do one of these things: 1. think about it/research more and develop an idea that I can present with more confidence (this is good for the asker), 2. post my findings as a reference (this can be still helpful for the asker), 3. refrain from posting (if my idea is so weak that I do not want to risk it, refraining from posting it may not be a bad thing).
Another idea would be to somehow combine the confidence level into the picture in a way that if answers posted with low-confidence are not chosen at the end, they would not be "penalized" to the same extent as if a confidence 5 is not chosen. This may actually encourage people to indicate their true confidence level. So perhaps the confidence level of each answer should be included in the formula. I will think about it.
Especially in low-activity pairs (all of my pairs, for example!) this might kill the low volume almost completely
Again, I am not sure.
Directory rankings are important but we should keep in mind that searches are almost always in a specific area of expertise, thus the effect of one or two answers not chosen is not that of a big deal. If my ranking changes from one day to the other slightly, I won't panic. It will change again in a few days, depending on my Kudoz activity.
So, this should not be changed hastily, without thinking.
Of course, not. That's why we are brainstorming here, right?
| | | | | Pages in topic: [1 2] > | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » 'hit rate' or 'points to closed answers ratio' (PTA) | Trados Business Manager Lite | Create customer quotes and invoices from within Trados Studio
Trados Business Manager Lite helps to simplify and speed up some of the daily tasks, such as invoicing and reporting, associated with running your freelance translation business.
More info » |
| | Wordfast Pro | Translation Memory Software for Any Platform
Exclusive discount for ProZ.com users!
Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value
Buy now! » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
