Pages in topic:   [1 2] >
Agency analysing the quality of your work using a program
Thread poster: Paula Borges

Paula Borges  Identity Verified
Brazil
Local time: 18:25
Member (2010)
English to Portuguese
+ ...
Sep 6, 2010

This agency I've been working with for almost a year now introduced a new system of "quality control". They use this program to compare your translations X final text after proofreading.

I spent a few months getting these "congratulations" e-mails because the proofreader had changed less than 1% of the text.

Then a few days ago I started getting e-mails requiring explanations: ratio of change over 5%. It's an automated system but once in a while they'll take the time to say: "this is not the level of quality we normally get from you, please explain what happened".

Of course I'd open the files and all these changes had to do with style and vocabulary. Examples:

- Asking me to translate a text into Brazilian Portuguese, then sending it to an European proofreader and asking me why did it have so many "errors".
- A proofreader decides to capitalise certain words that weren't capitalised in the original, which the program claims to be a 10% change.

I couldn't find ONE proper "error" in any of them.
I explained, time after time, that these changes had to do with the proofreader's style and not any lack of quality but it seems that they aren't willing to understand that you cannot judge the quality of a translation using a machine.

They won't even open the files before filling your mailbox with "congratulations" and "explain this" e-mails. Does that happen to anyone else? It's getting really annoying and I do not have the time to explain everytime that I can't read the proofreader's mind.

It seems like a dangerous approach to me. How would you handle this?


 

Jaroslaw Michalak  Identity Verified
Poland
Local time: 21:25
Member (2004)
English to Polish
Services provided Sep 6, 2010

It is generally accepted that if your work has defects, you should remove them free of charge. However, if there are no actual errors, then for the time you spend on such tasks you should charge them your regular hourly rates.

I am not saying that you will be able to enforce it, but if you put it this way it might at least make them think over the whole procedure.

Automatic QA systems, as most other inventions, is both a boon and a curse for translators... If they end up in the hands of people with too much time, it's definitely a curse.

One of clients decided to unleash on me a program that matched numbers, counted words and compared capitalizations. The last feature was particularly hilarious as it prompted an overzealous editor(?)/PM(?) who didn't know a word of the target language to dutifully capitalize all "suspect" words (including months, weekdays, you get the idea...) in the translation. That little trick saved me hours of browsing through the QA report, as I could immediately tell them what they could do with their "corrected" version. Come to think of it, I do not know if they followed my advice...


 

Paula Borges  Identity Verified
Brazil
Local time: 18:25
Member (2010)
English to Portuguese
+ ...
TOPIC STARTER
Time-consuming nuisance Sep 7, 2010

They are not asking for discounts or expecting me to change the text, they are merely trying to save the money they'd spend on properly qualified staff to perform quality control. They probably think scaring translators with numbers is a good technique to keep a good level of quality.

It seems dangerous to try and deal with language as an exact science.

When I got all these "congratulations/less than 1% change" e-mails, it did not always mean the text couldn't have been improved or that there weren't any errors, it just meant the proofreader didn't do much. On the other hand, when I got all those "explain yourself/over 5% change" it didn't mean the texts had improved: sometimes the proofreader had a different style, changed some words and did some tweaking, but sometimes there were errors added. So, the system is obviously flawed.

If they aren't willing to employ any humans to check quality, I'm afraid quality tends to get worse.

I'm not saying that every owner should be a translator, but they should at least understand the basics or hire someone who does.

I've been getting rid of clients/ PMs who don't even know Portuguese but attempt to proofread and question my work as well. It seems too surreal to be true.









[Edited at 2010-09-07 00:19 GMT]


 

Philippe Etienne  Identity Verified
Spain
Local time: 21:25
Member
English to French
shout Sep 7, 2010

I have had a few experiences like the one you describe (although not as procedural), receiving reports of so-called QA checks, from SDLX, Trados or other in-house QA tools.
I go through the list (or don't even open it if I already did myself my own automated QA WITH MY OWN SETTINGS) and reply they are all false positives, that I know what automated QA checks consist of and how irrelevant most of their findings (numbers, uppercase, punctuation, target-to-source word ratio, even terminology) usually are when applied without care.
They usually expect that I go through each entry, so usually I shout.

QA checks from CAT tools are useful if used sensibly, but applying brute force with them tells a lot about how people think languages work. Every language has their own specifics and thinking that QA checks always spots misses from the translator/editor is like thinking that MT can replace a translator. Either you use machines from start to finish (and leave translators to the real work) or you leverage their knowledge of languages to ensure quality. Using machines as the ultimate layer of quality/security works well in most industries, but not so in translation.

Those agencies would rather start TRUSTING their translators and ask THEM to use QA tools in CAT programs to do the work, rather than thinking they can add a layer of quality control without any target-speaking people in-house or hired.
And it's time for them to think that we sell our time, and we don't have spare hours to spend on such useless tasks that bring strictly nothing to the end product.

Fortunately, there are also many agencies who know that translators are an asset and not some unexhaustable resource in some workflow diagram.

Philippe


 

Laurent KRAULAND (X)  Identity Verified
France
Local time: 21:25
French to German
+ ...
Clueless people should not be allowed... Sep 7, 2010

to use machines, be they cars, chainsaws or computers.

My question to this agency would be:
If you can document such statistical changes, why can't you track and document all possible causes of these changes?


 

Samuel Murray  Identity Verified
Netherlands
Local time: 21:25
Member (2006)
English to Afrikaans
+ ...
The value of such automated checks Sep 7, 2010

Philippe Etienne wrote:
QA checks from CAT tools are useful if used sensibly, but applying brute force with them tells a lot about how people think languages work.


True. I think QA tools are very valuable because they sometimes catch things that humans can't (unless they have super brains), but I also think that one should realise the limitation of these tools. Some types of "errors" will be more frequent in some languages (e.g. capitalisation or end-of-segment punctuation). It is a mistake to use one QA tool for all languages unless the QA tool is able to adjust its checking based on the language.

One can also make an effort to reduce the types of "errors" caught by the QA tool yourself. For example, most of these tools check for doubled spacing, or for extra spaces at the start or end of a segment. Some translators believe that careless typing doesn't matter if it's only an extra space, but for some clients this is important, and translators should learn that accuracy relates not only to meaning but also to typing.

A smart client will realise that QA tools have their limitations and should allow the translator to select a bunch of "error" and mark them as "ignore", and accept the translator's word for it.

Proofreaders should be told that it is okay to disagree with the QA system. Client should learn to tell their proofreaders whether they want an "objective errors only" type of proofreading or an "improve the style" type of proofreading.


 

T F F  Identity Verified
Italy
Local time: 21:25
English to Italian
It's a long way... Sep 7, 2010

I quote every single word written here...

End client must trust its (selected) Language Service Provider...
Language Service Provider must trust its (selected) translators...
Translators must accept they are humans and welcome technology that helps spotting human errors...
Technology must accept that it has been created by humans so that its approach should always be "Is this a false positive?" and never "This is wrong!"...
...This is a long way made of trust, respect and intelligence!

When trust is missing, it always turns into "having control"...And strange things happen!
I never heard about these "Congratulations: 1% of change, you earned a teddy bear" I suppose it has costed a lot to this agency (- at least buying "teddy bears"!), but I don't think this could be a sound innovation...
It's not about numbers or statistics! Surely, they can help, but they turn into enemies when trust is missing: if the proofreader has edited the translation the way it markets his/her skills, a system like this one will double the problems of the agency using it!

Again: it's about trust and professionalism! Are you searching for innovation? Well, try to find a mean to have only "really honest proofread"!

IMMO, unfortunately there is nothing you can do, Paula: you have already done all you could do...Try to ask your client to double check with the proofreader that the changes are "preferential modifies" and not"errors"...But if the client has invested its trust (and money!!!!) into dishonest humans and/or self-opinionated systems, there is nothing you can do for it! Just let it go...

My 2 cents.
RR

[Edited at 2010-09-07 12:01 GMT]


 

Krzysztof Kajetanowicz (X)  Identity Verified
Poland
Local time: 21:25
English to Polish
+ ...
I guess a similar thing happened to me just a week ago. Sep 7, 2010

1. Sent an EN-PL test translation to an agency.

2. Got an early reply (the next day!), according to which my work didn't meet their standards. They were encouraging me to come back when I've improved my skills (yes, this was the kind of wording they used).

3. Now, I simply know that what I'd sent them was very good. I am not a lunatic. The source text was uncomplicated and I know my Polish. It was even above my usual quality, which – if you excuse me – is above substandard.

4. The only viable explanation I have come up with is that they they compared my work to some sort of a master in order to obtain an output in per cent (note the short turnaround time). The machine must have told them how different the two translations were. Seeing as how many agencies are producing Polglish junk nowadays, my English-Polish translation of this idiomatic text was apparently too non-literal and too Polish-sounding.

[Edited at 2010-09-07 12:51 GMT]


 

Paula Borges  Identity Verified
Brazil
Local time: 18:25
Member (2010)
English to Portuguese
+ ...
TOPIC STARTER
Krzysztof Sep 7, 2010

It used to be common sense that if you send the same text to 10 people, you'll get 10 different translations. Different proofreaders have different methods: some only fix mistakes and others like to rewrite a lot of the text.

In this particular case, it was a very short text. There were no errors, so nothing really for a proofreader to do, so he/she chose one specific word and replaced it with a synonym. It made no difference whatsoever to the meaning in general, it was like we say in Brazil "exchanging 6 for half a dozen". Since it was a very short text, the "ratio of change" was above 5%.

I did shout, and got an apologetic e-mail: "this is our method to ensure quality, we weren't questioning your work", but asking me "why did the proofreader change that word then?" what am I supposed to say: because he/she thinks this one sounds nicer while I prefer another word?

Technology is great, but will we have to continue fighting people who deal with language like it's an exact science? The fact certain agencies have been ignoring different languages have different rules about capitalisation, or even attempting to check languages they don't even know seems to indicate something is fundamentally wrong in their approach.

I've had clients use MT to translate back to English and then ask me why it sounds so different. I feel like asking: if you trust MT so much, why did you bother hiring my services?


 

Laurent KRAULAND (X)  Identity Verified
France
Local time: 21:25
French to German
+ ...
Where is the logic? Sep 7, 2010

Paula Borges wrote:
I did shout, and got an apologetic e-mail: "this is our method to ensure quality, we weren't questioning your work", but asking me "why did the proofreader change that word then?" what am I supposed to say: because he/she thinks this one sounds nicer while I prefer another word?


Where is the logic?

They ask you why an unknown colleague changed a word or possibly reformulated a sentence?

My reply to this would be "Well, ladies and gentlemen, your question should go to the person who made the changes!"...


 

Paula Borges  Identity Verified
Brazil
Local time: 18:25
Member (2010)
English to Portuguese
+ ...
TOPIC STARTER
I don't know where logic is hiding these days... Sep 7, 2010

I'm guessing in this case the answer would be: "well, I had to change something to justify my rates"

The only thing that seems obvious to me is that the PM doesn't know the first thing about translation or language. So I guess I'll just come up with my own automated machine response to these e-mails? Any suggestions?

They have told me some of their clients were so happy with my work they request all of their work to be done by me. So I guess 1% or 5% won't make them stop coming.


 

Oliver Walter  Identity Verified
United Kingdom
Local time: 20:25
Member (2005)
German to English
+ ...
That's what I thought Sep 7, 2010

Paula Borges wrote:
The only thing that seems obvious to me is that the PM doesn't know the first thing about translation or language.

That's what I thought when I read your first posting on this topic.
So I guess I'll just come up with my own automated machine response to these e-mails? Any suggestions?

Well, not really. I thought you could tell them (email or phone) that they don't understand what translation and language really are, and either they must allow you to use your understanding of these matters or they might lose you as a valuable translator.
They have told me some of their clients were so happy with my work they request all of their work to be done by me. So I guess 1% or 5% won't make them stop coming.

Tell them, that is the evidence that your translations are of good quality and their QA system isn't!
(other post:)
what am I supposed to say: because he/she thinks this one sounds nicer while I prefer another word?
Yes, I thinks that's exactly what you should say. Tell them they are wasting their and your time if their QA process suggests changes that are not improvements, or even changes that make the translation worse. They need to learn how to use QA for real improvement, not just give a machine free rein for its spurious "error" reporting.

If all this only started a few days ago, you could say you appreciate they're trying to be innovative and efficient, but the system, or their use of it, is not at present satisfactory.

Oliver


 

Paula Borges  Identity Verified
Brazil
Local time: 18:25
Member (2010)
English to Portuguese
+ ...
TOPIC STARTER
Thank you Oliver! Sep 7, 2010

I just sent an e-mail saying all these things, I guess I needed to hear from a fellow translator that my decision to stick up for myself is not too extreme.

 

Laurent KRAULAND (X)  Identity Verified
France
Local time: 21:25
French to German
+ ...
It is extreme,... Sep 7, 2010

Paula Borges wrote:

I just sent an e-mail saying all these things, I guess I needed to hear from a fellow translator that my decision to stick up for myself is not too extreme.


but on the low sideicon_wink.gif - best of luck to you for future assignments, maybe the PM's will get the gen!


 

Pablo Bouvier  Identity Verified
Local time: 21:25
German to Spanish
+ ...
that matched numbers? well, may be.... Sep 7, 2010

[quote]Jabberwock wrote:


One of clients decided to unleash on me a program that matched numbers...

[quote]

I would had ask the client on how a QA system will match numbers like 10 ºC and 50 ºF (wich is perfectly valid) in a translation or somethig like this sentence begins with letter T in another language like German (D), French (C) or Spanish (E)...


 
Pages in topic:   [1 2] >


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Agency analysing the quality of your work using a program

Advanced search







Protemos translation business management system
Create your account in minutes, and start working! 3-month trial for agencies, and free for freelancers!

The system lets you keep client/vendor database, with contacts and rates, manage projects and assign jobs to vendors, issue invoices, track payments, store and manage project files, generate business reports on turnover profit per client/manager etc.

More info »
BaccS – Business Accounting Software
Modern desktop project management for freelance translators

BaccS makes it easy for translators to manage their projects, schedule tasks, create invoices, and view highly customizable reports. User-friendly, ProZ.com integration, community-driven development – a few reasons BaccS is trusted by translators!

More info »



Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search