Pages in topic:   [1 2] >
Method of sorting used in ProZ.com directory to be updated
Thread poster: Jared Tabor

Jared Tabor
Local time: 07:29
SITE STAFF
Jun 2, 2010

Dear members,

After about ten years without a change, the way that translators are sorted in the ProZ.com translator directory is being evaluated for an update.

Several adjustments may ultimately be made, and searchers will be given the option of defining their preferred sort order. For now, the plan is to change the default sorting order to use the number of KudoZ PRO points earned over the past twelve months (total PRO points earned ever, which is used now, will be an optional secondary sorting choice).

Recommendations for further improvements are also being evaluated. If you would like to enter a suggestion for an improvement to the directory, please do so using a support request.

The translator directory is located under the "Jobs and directories" menu tab.

Regards,

Jared
Member services

Why change the way the directory works at all?

To keep it as relevant and useful as possible over time.

Why change from sorting by KudoZ points all time, to sorting by KudoZ points earned over the past year?

As things stand, to move toward the top of the directory in a given combination (ie. pair and field), a person who has been at ProZ.com for just one or two years would have to gain as many points in that time frame as colleagues who have been at ProZ.com for as many as 10 years. Obviously this is not fair, and the adjustment will level the playing field somewhat.

Does this mean that the points I earned previously are now worthless?

No. Secondary sort is still carried out according to points earned over all time.

What about the advantage in exposure that members have had over non-members?

That will not change. As always, when someone searches the directory, they will get two sets of results: paying members first, then non-members. What is being updated is the sort order used within each set.


Direct link
 

Damian Harrison
Germany
Local time: 11:29
German to English
Why? Jun 2, 2010

Jared wrote:

Several adjustments may ultimately be made, and searchers will be given the option of defining their preferred sort order. For now, the default sorting order will be changed to use the number of KudoZ PRO points earned over the past twelve months (total PRO points earned ever, which is used now, will be an optional secondary sorting choice).


Aside from the fact that the current system has now been in place for ten years, why do you plan to make this change?


Direct link
 

Jared Tabor
Local time: 07:29
SITE STAFF
TOPIC STARTER
Providing more relevant default search results Jun 2, 2010

Hi Damian,

A goal of this change is to provide default search results that are more relevant to the searcher (since those who have been active in the field more recently are shown), and more fair to translators who are active in KudoZ but who have joined more recently than others.

Regards,

Jared


Direct link
 

Giuseppina Gatta, MA (Hons)
Member (2005)
English to Italian
+ ...
Then... Jun 2, 2010

Jared wrote:

Hi Damian,

A goal of this change is to provide default search results that are more relevant to the searcher (since those who have been active in the field more recently are shown), and more fair to translators who are active in KudoZ but who have joined more recently than others.

Regards,

Jared


...you will be unfair to all the people that have joined this website a long time ago, and have contributed in much more significant ways than KudoZ, have a full-time job as freelancers and not much time to dedicate answering the KudoZ (btw answers are often wrong and often the wrong ones are awarded with points).
Anyway I wrote a ticket about this.


Direct link
 

gianfranco  Identity Verified
Brazil
Local time: 08:29
Member (2001)
English to Italian
+ ...
I suggest a weighted method based on various criteria Jun 2, 2010

Jason,

there is an alternative and fair method has been suggested many times and always ignored.

KudoZ points are quite unfair as they are a "cumulative" system. It means that any mediocre answered, provided with enough time and nothing better to do, can cumulate points and raise to the top of the directory, because even if he/she provides n-thousands incorrect answers, the score only shows and cumulates the correct ones.

The directory should be sorted by quality, not quantity, adding to the equation a factor that takes into account the ratio (chosen answers / total answers ).

Example:
800 chosen / 1000 total is an excellent ratio, 80%
considering that there are often other good answers, or the asker may even occasionally discard a perfectly good answer for any reason.

On the other hand
800 correct answers / 4000 total answers (a 20% ratio)
currently gives the same score and position, but it is actually obtained only by persistence and not much real skills.

I suggest to sort the directory, at least starting from a certain treshold (i.e. above 250-500 points) only showing the ratio and not the actual score, and get the most reliable answerer raise to the top.


Alternatively, possibly more balanced than the criterion illustrated above, could be to introduce a weighted system that may include one or more of the following:

1. total score
2. recent score (latest 1-2 years)
3. reliability ratio (as defined above)
4. other relevant contributions to the site

Factor 1. should not be removed as it would be unfair to long time members

Factor 2. can be introduced to give a higher visibility to the current active members, but not entirely unfair to the old timers

Factor 3. illustrated above, very important and identified a long time ago by many members

Factor 4. makes sense as activities beneficial to the site can be rewarded, in some measure, with some increased visibility in the directory.


None of these factors alone is perfect, but a good combination and a weighted system may be the least unfair overall. I would exclude a system consisting in alternative sorting methods because what happens is that everybody would stop to the default method, which may be heavily biased in one way or another.
I suggest a combined method, possibly assigning to each factor a fair weight, and not necessarily the same weight as the other factors.

Gianfranco


[Edited at 2010-06-02 21:37 GMT]


Direct link
 

Walter Landesman  Identity Verified
Uruguay
Local time: 07:29
Member (2005)
English to Spanish
+ ...
Disagree with suggested change Jun 2, 2010

Hi Jared,

I agree with Giuseppina. This would be unfair to those who abided by certain rules and fought our way up in the Kudoz system several years ago.

We also had to deal with old members with such a high number of kudoz we thought we could never reach. But we did, we invested time and efforts and research to answer questions and get those kudoz.

It is unfair that rules may change now when we are up on the scale. Should we start all over again? We might not have the time to contribute now, the time we had as new members and new freelancers and following your suggestions (Proz`s) we built a strong profile including getting many kudoz because you said that was the way to be among the firsts in a directory search.

I see it as a way of giving an incentive to new members as part of the current Membership Campaign and also to encourage people to participate more in the kudoz system.

We believed in Proz.com, we also invested our membership money to support the site all these years and abide to the rules valid at that time.

I really disagree with this proposal. I think new members or newcomers to Kudoz should fight their way up as we did several years ago.


Direct link
 

Konstantin Kisin  Identity Verified
United Kingdom
Local time: 10:29
Member (2004)
Russian to English
+ ...
Phase-in required to ensure fairness Jun 2, 2010

I think most of us have seen this coming. Clearly KudoZ participation has been suffering from a decline for several years now and the intent of this change is obvious: to encourage active involvement.

At the same time, I think it's important to amend the system in a way that meets this objective while also ensuring that past KudoZ contributions are recognised in some way and that all of us are treated fairly.

It's also essential to understand the kind of impact this may have on the basis of specific examples. A brief search of the Directory indicates just what a massive rebalancing of the value of 1 KudoZ point we are talking about here:

With this new system in place, answering 3-4 questions can take you from zero to hero in no time. While all of us, including those at the top of the rankings, would probably recognise that KudoZ points were never a great way of distinguishing between good translators and bad translators, it was some sort of indicator nonetheless.

If you were a terrible translator it used to be difficult (although not impossible as has been shown in my language pair by a number of "hard-working" individuals) to get to the top of the directory. Now, all you need to do is grab a few easy points and you're there. The outcome of this is that activity is rewarded, however I don't think that outsourcers searching the Directory are looking for people who have been active in the last 12 months - they're looking for people who can do the job they need done.

In addition, the fact that we (or at least I) had no warning of this impending change means that those of us who were, quite reasonably, acting in accordance with the old system are now at a sizeable disadvantage. Someone who has been active in the last 12 months has had a head start of... 12 months.

In view of these two things, I would suggest that a more balanced system should be based on the following ideas:

1. Historical contributions should be recognised in some (albeit significantly less influential) way.
2. Regardless of whether 1. is implemented, notifications of this change should be sent to all current and past KudoZ participants informing them that it will be made effective in 12 months. This gives everyone an opportunity to start from a level playing field, while the extra time allows the system to be refined and perfected.


Direct link
 

Mats Wiman  Identity Verified
Sweden
Local time: 11:29
Member (2000)
German to Swedish
+ ...

MODERATOR
A totally irrlevant change IMHO Jun 2, 2010

Hi Jared,

This seems to be a change for change's sake.

1. Why would it be more relevant with the last 12 months' contributions than all contributions?

2. Is it fair to strip someone of his/her achieved KudoZ points only because they were gained 2000-2009? (Yes - I am partial)
No outsourcer will agree on that unless they are only looking for hungry and active newbies.

A seasoned outsourcer would rather look for an experienced translator with lots of proven
competence (the older the better).

The option for the outsourcer to choose sorting order should IMHO depart from a default sorting order, i.e. the original one - by total KudoZ points in the pair.

As we have extensively discussed two years ago, a sorting by rate is IMHO a relevant and accceptable sorting method. (I maintained that being cheap is no advantage and many an outsourcer shares this view).

Sorting by years of experience is also a possibility.

Sorting by credentials is also a possibility.

Sorting by target language native speech is also relevant to outsourcers.

For interpreters, location is also a elevant sorting option

I sincerely hope your suggestion won't be implemented wihout dicussion among members.

It's too important to be a rush discussion withou consultation.

PS addition "intended to level the playing field".
Jared, This is not a playing field but the result of heavy investment, very relevant to the outsourser and very important to the freelancer.
All KudoZ points could have a note (like 2000-2007) indicating the time period when point were earned, thus putting a newbie at less disadvantage.
OTOH, having old points should not be rewarded with a disadvantage (see above)

Mats

[Edited at 2010-06-02 22:03 GMT]


Direct link
 

Natalie  Identity Verified
Poland
Local time: 11:29
Member (2002)
English to Russian
+ ...

MODERATOR
Hi Jared Jun 2, 2010

Some addicted KudoZers who provide tons of answers using machine (or Google) translator also gather points rapidly - so what will happen now? Will they be listed the top of the directory? I doubt that outsourcers will be really happy when the directory would suggest them such candidates...

Directory sorting by simple sum of points gathered rapidly within a short period of time (and 12 months is a very short period of time, believe me!) can be hardly considered as a serious parameter allowing high rankings in the directory. As it has been suggested many times in the past, rankings should be based on more complicated indices that take into account not only the sum of the points, but also the reliability of answerers. Lots of discussions, suggestions and propositions can be found in the archives - please just open those old threads and take a look at them. There were excellent suggestions of weighted scoring that would take into account reliability, period of time and many other factors.

Also, before introducing drastic changes it would be fair to announce them beforehand. You should give people a chance to become more active and improve their rankings.

Natalia


Direct link
 

Johanna Timm, PhD  Identity Verified
Canada
Local time: 02:29
Member (2002)
English to German
+ ...
reliability instead of quantity Jun 2, 2010

Natalie wrote:

As it has been suggested many times in the past, rankings should be based on more complicated indices that take into account not only the sum of the points, but also the reliability of answerers.


Precisely. Thanks, Natalia.

johanna


Direct link
 

Jabberwock  Identity Verified
Poland
Local time: 11:29
Member (2004)
English to Polish
Oh, really? Jun 2, 2010

A change which makes KudoZ a place for a more cutthroat competition? Where KudoZ chasers will be given even more prominence? Where the "winning" answer will have to be given within seconds? Where users who actually care about their ranking will claw and grasp at every conceivable question not to get ahead, but just to keep afloat?

You can't seriously think that the change will not affect the quality of KudoZ as such... Considering that in the current system you are never "punished" for giving wrong answers, it might be enough to answer as many questions as possible - there are enough clueless askers around.


Direct link
 

Mikhail Kropotov  Identity Verified
Russian Federation
Local time: 13:29
Member (2005)
English to Russian
+ ...
A Suggestion: Depreciating KudoZ Points Jun 2, 2010

Every community aims to encourage - and reward - active participation, and ours is no exception. I see this change as a means of achieving that objective. Moreover, now could just be the time to try something new and fresh.

Having said that, I can envision a simple mechanism that could take into account one's older accomplishments together with newer ones. And it would still serve the ultimate purpose, i.e. motivate people to stay active and keep contributing.

Many things in life depreciate with time, so why not KudoZ points? I propose a simple weighting method: use multipliers for all points based on when they were earned. For instance, 100 points from 5 years ago could be worth as much as 20 earned in the last thee months.

The exact math for this can be discussed, played with and tested to maximize the utility and fairness of the new ranking system. The key idea is to recognize past accomplishments and equate them with newer ones in a fair and balanced way. The specifics will follow.


Direct link
 

Erik Freitag  Identity Verified
Germany
Local time: 11:29
Member (2006)
Dutch to German
+ ...
reliability ratio Jun 2, 2010

Natalie wrote:
As it has been suggested many times in the past, rankings should be based on more complicated indices that take into account not only the sum of the points, but also the reliability of answerers. Lots of discussions, suggestions and propositions can be found in the archives - please just open those old threads and take a look at them. There were excellent suggestions of weighted scoring that would take into account reliability, period of time and many other factors.



I agree 100%! One example where this (and the reasons proz.com might have to not implement something like that) has been discussed at large can be found here.


Direct link
 

Katalin Horváth McClure  Identity Verified
United States
Local time: 05:29
Member (2002)
English to Hungarian
+ ...
Previous, repeated discussions and proposals Jun 2, 2010

Jared,
I am glad to hear that you are considering adding more options to the directory sorting. However, I have to agree with those that disagree with your proposed change. It does not accomplish the stated intended purpose: to provide more reliable information for outsourcers. Instead, please focus on the word: reliability.
That is what's important.

There were several discussions in the past about this, here is one, with what I wrote there:

http://www.proz.com/forum/kudoz/119297-hit_rate_or_points_to_closed_answers_ratio_pta.html#980033


As it was pointed out before, everybody can see his/her own KudoZ statistics in his/her profile, the reliability ratio is there, it is called "acceptance rate" and it is a percentage figure (so 65.14 means 65.14%, or 0.6514).

If for directory ranking purposes, we want to use not only the number of KudoZ points but also the reliability ratio (how many answers were given in relation to the points gained), the easiest way is to MULTIPLY the two figures and use the result for ranking.

I think this is the best of both worlds.
Using only the KudoZ point total omits the information about reliability. (It also gives unfair advantage to the aggressive, machine-gun style point-hunters, and disadvantageous for newcomers.)

Using only the reliability ratio omits the information about level of activity. (It would give unfair advantage to people that answer very few questions, and it may discourage people from answering questions.)

This combination has no problem for scaling (it can be used for any high number of points), and it works in a fair way, at least I found it fair. I calculated a few scenarios, and I liked what I saw.

If people want it, I can type up a few examples here, but I guess anyone can do it.


Katalin


Direct link
 

Angie Garbarino  Identity Verified
Local time: 11:29
Member (2003)
French to Italian
+ ...
Not fair in my opinion Jun 2, 2010

Hello Jared

I disagree with this change, Natalie has just provided a good point and I also agree with Giuseppina, a change like this will be unfair for old members who have little time to participate actively in KudoZ, not to mention the very wrong answers that are often selected.

Please reconsider

Regards

Angio


Direct link
 
Pages in topic:   [1 2] >


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Method of sorting used in ProZ.com directory to be updated

Advanced search






Wordfast Pro
Translation Memory Software for Any Platform

Exclusive discount for ProZ.com users! Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value

More info »
Anycount & Translation Office 3000
Translation Office 3000

Translation Office 3000 is an advanced accounting tool for freelance translators and small agencies. TO3000 easily and seamlessly integrates with the business life of professional freelance translators.

More info »



Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search