Pages in topic:   [1 2] >
How do you assess translation quality without looking at the language?
Thread poster: Phil Hand

Phil Hand  Identity Verified
China
Local time: 04:36
Chinese to English
Dec 29, 2015

Hello, Prozians. Merry Christmas to those who celebrate it, season's greetings to those who don't, and an early happy New Year to everyone!

I've been away because I have a new in-house job, but I'm back to seek the benefit of your collective wisdom. The title of my post is not a joke, believe it or not.

In my company we have a large team of in-house translators who are native speakers of Chinese, but who work mainly from Chinese to English. One of the things I would like to do is to try to improve quality across the whole team. One of the biggest problems they have had as a team is that for a long time there has been literally no-one in their organisation who is competent to assess quality. By this I mean no native English speakers, or any speakers with a good enough grasp to really read a document and grade its language. Another problem is far too much time spent thinking about the source text wording, and too little time thinking about communicating meaning in the target text.

My big idea is that I could kill these two birds with one stone by putting in place non-linguistic standards or assessment criteria. What I mean is ways of looking at the target text in use, to see whether it's fit for purpose.

Examples:
1) Customer satisfaction. This one is the most obvious, but it's problematic in our company, because many of our "customers" are company departments staffed by people whose English is not nearly as good as they seem to think it is. They just love to instruct us to use "their" English. Nevertheless, customer satisfaction is a vital measure.
2) Online comments. Some of our translated material is published for in-house consumption (it's a big company, over 100,000 people), or in customer-facing magazines. My thought is that unreadable material is unlikely to be read, so no-one will comment on it. Well-translated material should attract more readers, and more comments, so on average, the number of comments could be some kind of index of translation quality.
3) User feedback. Sometimes we translate computer system interfaces. User satisfaction might be correlated with translation quality (though of course there will be lots of other confounding factors as well).

Does anyone have any experience using this kind of non-linguistic quality measure? Any measures that you can recommend? Any comments on how to get useful feedback, or how to use quality indices once we've got them? Thanks for any comments you are able to offer.


Direct link Reply with quote
 

Balasubramaniam L.  Identity Verified
India
Local time: 02:06
Member (2006)
English to Hindi
+ ...
Some thoughts Dec 29, 2015

The problem with the yardsticks you have mentioned is that they will tell of problems much after the effects of a defective translation have done the damage, in other words, they are not preventive yardsticks. They won't be able to save you or your organization from translation-engineered disasters in time.

Since you are in a position to personally know the translators and become aware of their linguistic and educational backgrounds, as well the type of translation experience they have, one way would be to assess the translators, instead of the translations, and distribute translation tasks (if you have any control on this) in such a way that the skills required for a task and the skills possessed by the translator implementing the task are better matched. This will yield better results than randomly picking any translator for any job. For example, a person with education in engineering, should do technical manuals, and a person with better command over English should do marketing material, just to take an example.

As you get to know your translators more, you will be able to do this best-fit strategy more efficiently.

Simultaneously, you should institute skill improvement programmes for all your translators so that they gradually upgrade their linguistic skills (in English).

[Edited at 2015-12-29 17:08 GMT]


Direct link Reply with quote
 

Neptunia
Local time: 21:36
Italian to English
interesting problem Dec 29, 2015

The strategies you've described might work but I think the sample size would have to be vast to get real answers. Like you said - -too many possible confounding factors. The only suggestion I have would add more to the team's workload which is probably not desirable and I am not sure if it counts as "looking at the language." It involves some kind of a brief, informal, periodic Back-Translation Workshop in which you select a few lines or paragraphs from recently translated material and see if the group can come up with a consensus of what it means in Chinese (or alternatively, compare individual translated versions). If you can find "good" ambiguous or nonsensical texts then at least the discussion might demonstrate the extent of the problem and drive home the necessity of preserving the sense of the text in the translation. Even though the translations wouldn't have names attached, you might find that soon no one would want their texts used as workshop samples. ha ha ha.

Direct link Reply with quote
 

B D Finch  Identity Verified
France
Local time: 21:36
Member (2006)
French to English
+ ...
My take on your assessment criteria Dec 29, 2015

Merry Winter Solstice and a Happy New Year!

Phil Hand wrote:non-linguistic standards or assessment criteria. What I mean is ways of looking at the target text in use, to see whether it's fit for purpose.


Examples:
1) Customer satisfaction. Your customers who 'just love to instruct us to use "their" English' are trying to maintain their status, as self-proclaimed English-speakers, within their own company. Even if "customer satisfaction is a vital measure", beware; they might be pleased to see their own poor English repeated, but later become dissatisfied if their own customers raise complaints e.g. because of unclear or faulty assembly or safety instructions. What happens if customers are satisfied but don't realise that the material was badly translated?

2) Online comments. "Well-translated material should attract more readers, and more comments, so on average, the number of comments could be some kind of index of translation quality." Have you noticed the semi-literate standard of English produced by native speakers of English (especially trolls), on internet forums? This is less of a problem in fields where the readers a likely to have a high level of education and you are probably right for material relating to such fields. However, this might fail to identify major errors of translation, e.g. reversals of meaning or incorrect imputations of causality.

3) User feedback. I agree that there will be confounding factors as well as quality factors. One problem is that there is often a bias in favour of negative feedback unless feedback is positively invited. Clear and accurate instructions are often expected and not commented upon, while poor instructions are likely to give rise to accidents, product failures, complaints, ridicule or just queries.



[Edited at 2015-12-29 19:10 GMT]

[Edited at 2015-12-29 19:11 GMT]


Direct link Reply with quote
 

The Misha
Local time: 15:36
Russian to English
+ ...
I am afraid you are fighting the windmills here Dec 29, 2015

Who is your output intended for anyway? If it's for the company's internal (or, broader, in-country) consumption to satisfy some regulatory mandate, the issue is moot since no one would know any different, nor care. If it is indeed intended for actual English-speaking audiences, then you are vastly outnumbered against the onslaught of Chinglish your colleagues must be producing, and it will probably matter little what you do or don't do.

As a disclaimer, I know no Chinese, nor do I have any experience with any Chinese translations into English, aside from the anecdotal (such as fook yee restaurants or some unfortunate user manuals for cheap Chinese-made goods). However, if my extensive experience with translations into English produced by nonnative residents of Moscow and other such fine places is any indication, the predominant majority of them are not suitable for consumption by unsuspecting English-speaking audiences without major rewriting. And I would assume that Russian is much less "alien" to English than Chinese.

Long story short, I am afraid there's precious little you could do except for fixing one bad translation at a time, drop by drop into that bottomless bucket.

Regardless, good luck with your valiant effort.


Direct link Reply with quote
 

Katalin Horváth McClure  Identity Verified
United States
Local time: 15:36
Member (2002)
English to Hungarian
+ ...
Look at it backwards Dec 29, 2015

Phil,
Let's see you found an assessment method to look at the translations produced by your company.
What is the next step?
How are you going to improve it?
If there are no native English speakers on that in-house team, what good it is to find out exactly what the current quality is? It seems to me you are already suspecting it is not good, so what difference does it make whether it is 2 stars out of 10 or 5 out of 10? (I just made this up.)

I am not sure how many translators are involved, but I don't think there is a way around assessing their individual abilities. After all, they are the ones producing the translations, no matter what processes are around them. You could check random samples, or subject them to the same test, whatever, but I have a feeling that you need to take it down to the level of individuals.

And, if you know you would need to bring in native or native-level editors anyway, why don't you push for bringing them in now? They could participate in the assessments and building a better quality control framework.

So, what I am saying is that perhaps you should attack the problem from the "other end", take what you would do at the end, and do it first.

I hope I am making sense...


Direct link Reply with quote
 

Anton Konashenok  Identity Verified
Czech Republic
Local time: 21:36
English to Russian
+ ...
Mission (almost) impossible Dec 30, 2015

I, too, am trying to improve the quality of non-native translations by providing detailed feedback as an editor and by slowly writing a white paper on typical translation mistakes in my field and language pairs. However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team of translators, I will never engage anyone who offers translation into non-native language, even if the project in question does not involve that language pair, as I consider it a sure sign of professional incompetence.

Direct link Reply with quote
 

jyuan_us  Identity Verified
United States
Local time: 15:36
Member (2005)
English to Chinese
+ ...
No employers in China has that knowledge Dec 30, 2015

Anton Konashenok wrote:
However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team of translators, I will never engage anyone who offers translation into non-native language, even if the project in question does not involve that language pair, as I consider it a sure sign of professional incompetence.


No one out of the translation circle knows that translation should be done by those native in the target language.


Direct link Reply with quote
 

Dani Karuniawan  Identity Verified
Indonesia
Local time: 03:36
English to Indonesian
+ ...
Assess customer satisfaction Dec 30, 2015

Merry Christmas! (Not merry winter solstice)

Phil Hand wrote:
Does anyone have any experience using this kind of non-linguistic quality measure? Any measures that you can recommend?


I am a native Indonesian and live in Indonesia. I use below criteria to assess my own translation, beside linguistic measure.

1) Wait for user feedback and/ or critics and provide post-sale service, such as, free consultation and "adjustment" for 1 month after delivery. This step is not only to ensure customer satisfaction, but it is also useful to examine translation quality whether your translation is satisfying.

2) The same person uses your service for more than 3 individual occasions in the same year. Your translation is more useful when it retains your customers or it converts a client to be a customer with long relationship.

3) He or she becomes your regular customer (at least 5 years).

4) Key speaker of an International event, such as an International seminar or workshop, use your translation AND he or she is happy with his or her audience reaction. When your client happily tell you about their reaction, it is a good news for you.

5) Your translation is published by at least an International scientific journal head-quartered in English-speaking countries without many "adjustment". This is a sign that your translation meets International criteria of good translation.

I composed this criteria for my self a long time ago from my experience of rise and fall cycle in this field.

[Edited at 2015-12-30 03:16 GMT]


Direct link Reply with quote
 

Kevin Dias
Local time: 05:36
SITE STAFF
TAUS / Microsoft Dec 30, 2015

Hi Phil,

I would recommend looking into some of the research TAUS and Microsoft are doing. I think TAUS does a lot of work and research in the area of translation quality assessment (i.e. search 'taus translation quality assessment' or 'taus dynamic quality framework').

Also, your examples reminded me of a presentation I saw by Microsoft on P3 (Post-Publish Post-Edit). While not exactly the same as what you are doing (as you are not using MT), I think some of the principles are similar (i.e. looking at user feedback / use statistics) and might apply to what you are trying to do.

Kevin


[Edited at 2015-12-30 03:03 GMT]

[Edited at 2015-12-30 13:21 GMT]


Direct link Reply with quote
 

laniHK  Identity Verified
Hong Kong
Local time: 04:36
Member (2015)
English to Chinese
+ ...
If you pay peanuts, you will get monkeys Dec 30, 2015

jyuan_us wrote:

Anton Konashenok wrote:
However, I firmly believe that a truly professional translation requires a native speaker of the target language (or a native-equivalent one who has lived, studied and worked in the country of the target language for at least several years). By the same token, if I am to recruit a team of translators, I will never engage anyone who offers translation into non-native language, even if the project in question does not involve that language pair, as I consider it a sure sign of professional incompetence.


No one out of the translation circle knows that translation should be done by those native in the target language.


I think the major problem lies in extremely low pay for translators in Mainland China. There are plenty of talented Chinese on the mainland who have professional English proficiency that is more than good enough to handle the bulk of translation jobs. Unfortunately, those talents will not be appropriately compensated as being a translator. The industry in China fails to attract and retain the good ones. Hence forth, they end up with a house full of mediocre or even sub-standard producers. I once searched translation agency websites and found tones of mistakes there. I took the time to write one of the big ones to give some feedback but I think no one there even bothers.

Phil, I can definitely appreciate your situation. I bet the staff turnover rate is also very high. You have a tall order. I personally got a client from Mainland because he was not satisfied at all with the translation quality produced by mainland translation companies. He used his personal network and found me at the end. So, the cut-throat pricing strategy and less than minimum wage pay to translators is not a sustainable business model in the long run.


Direct link Reply with quote
 

Phil Hand  Identity Verified
China
Local time: 04:36
Chinese to English
TOPIC STARTER
Thank you Dec 30, 2015

Thanks, everyone. As ever, I am very impressed by the experience and knowledge that you all have to contribute. Particularly thanks to Kevin, for those links, and to Katalin for that very well put inversion.

As Katalin suggested, in the end I'm going to have to just get down to the hard slog of building up the skills of the team. I'm trying to look for some ways to ease and simplify that process because: (1) I'm a translator, not an educator. I'll do education if that's what it takes, but training is neither my top skill nor my first love. (2) My colleagues are already very well trained (most have an MA in translation), and very busy. More importantly, they are smart and hard-working, and I am pretty confident that if I can point them in the right direction, they'll get where we need to go themselves. At the moment it's a weird brew of institutional and cultural constraints that is holding them back.

So in part what I'm looking for is ways to empower my colleagues. At the moment, when a senior manager sends back their translation with a note saying, "I think this phrase should be 'talents cultivation'," they scuttle to comply. I'd like them to be in a position to say, actually, 'workforce development' is a much more standard phrase - and we know because our HR people found working with documents 20% easier when we write it like that.

That's the dream, though I'm far from sure that it's really possible. A TAUS document puts it rather well: "Quality is when the buyer or customer is satisfied, yet quality measurement in the translation industry...is managed by quality gatekeepers...who have specific evaluation models." When the translators are battling deeply flawed evaluators, it's hard to get anything done, so more than anything, I want to equip them with the tools to find and create their own models/standards, which they can take to the evaluators and use to persuade them. Because all the training in the world can't help when every piece of idiomatic English you try to deliver just gets thrown back in your face.


Direct link Reply with quote
 

Maxi Schwarz
Local time: 14:36
German to English
+ ...
Why not hire a trained professional to do a professional job? Dec 30, 2015

Surely the person who can assess the quality of translations is someone who is trained in that profession. I would think that you need someone who can also assess the translation itself because there are two main components of a translation: that all important meaning is rendered, and that it reads smoothly and comprehensively in the target language. Someone trained in translation, hopefully with experience, who has native quality mastery of the target language, should be doing this assessing. Otherwise you are going about it blindly, hoping that the output is ok, but without ever being able to be sure.

Direct link Reply with quote
 

TranslateThis  Identity Verified
Local time: 14:36
Spanish to English
+ ...
style guides and glossaries could be very helpful Dec 30, 2015

How about client-specific style guides and glossaries?

Direct link Reply with quote
 

Heinrich Pesch  Identity Verified
Finland
Local time: 22:36
Member (2003)
Finnish to German
+ ...
Mission impossible? Dec 31, 2015

The one thing I use when assessing a translation in a foreign language is Word spelling checker. If Word displays a lot of red lines under words of the source language this could be a sign for careless work. Then I test some of the offensive words, copy them into Google and whatch what it makes out of it.
Of course this is only part of the task you have in mind. But assessing the meaning of a translation without analysing the language is quite impossible. Back translation by MT is one approach, but that too requires someone to actually read the stuff.


Direct link Reply with quote
 
Pages in topic:   [1 2] >


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

How do you assess translation quality without looking at the language?

Advanced search







Protemos translation business management system
Create your account in minutes, and start working! 3-month trial for agencies, and free for freelancers!

The system lets you keep client/vendor database, with contacts and rates, manage projects and assign jobs to vendors, issue invoices, track payments, store and manage project files, generate business reports on turnover profit per client/manager etc.

More info »
SDL Trados Studio 2017 Freelance
The leading translation software used by over 250,000 translators.

SDL Trados Studio 2017 helps translators increase translation productivity whilst ensuring quality. Combining translation memory, terminology management and machine translation in one simple and easy-to-use environment.

More info »



Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search