Comparing MT engines, which MT should I choose?
Thread poster: Philippe Locquet

Philippe Locquet  Identity Verified
Portugal
Local time: 19:52
English to French
+ ...
Aug 10

I have been browsing and I could not find a comparison for MT output. So, here’s a thread about it.
When a translator or LSP considers Machine Translation, it is likely that, among other things, they goal is to reduce their effort and speed up a little their workflow (maintaining quality through human language skills off-course).
Since there are now many, many options, the big question becomes:
Which MT should I choose?
This seems like a good question, but which criter
... See more
I have been browsing and I could not find a comparison for MT output. So, here’s a thread about it.
When a translator or LSP considers Machine Translation, it is likely that, among other things, they goal is to reduce their effort and speed up a little their workflow (maintaining quality through human language skills off-course).
Since there are now many, many options, the big question becomes:
Which MT should I choose?
This seems like a good question, but which criteria should be used to discriminate MTs?
Here I propose a method, feel free to test it and comment/add your results.

Method:
These are the criteria you need to look at. In my opinion, price should be last, because only when you have tested and compared you will then have an idea if using an MT will really help you and how much will help you. Then you will be able to work out the value of the service for your specific needs.
Here are the criterias:
A/Language pair (not all MT perform the same and not all language pairs are supported)
B/Type of documents (not all MT perform the same in legal, news, medical, marketing etc.)
C/Access method (how you want to use the MT: pretranslate a document outside a CAT, pretranslate an xliff, use it inside your CAT etc. Not all MTs can connect equally)
D/Price (and method of payment: yearly, monthly, mileage (per character consumed) etc.)

Procedure for testing:
1/When starting your comparison, you need to determine for which language pair you want to use the MT most. Then pick up a news article about a common topic and talking about very famous people. This is the easiest for most MTs. Create a table worksheet and start copy-pasting the outputs of all the MTs you are comparing. Review and then, select 3 or 4 that performed the best.
2/Now you need to decide what is the document type you will use the MT most for and in which domain (automotive, legal, medical etc.). Now you can make a new table comparing the 3 or 4 engines that came ahead in the previous test. Feed a type of text represents the type of doc + domain you want to use the MT for. Pick the 2 winners.
3/Using criterias C and D decide who is your winner. (Or another conclusion could be that in your language pair + domain, using MT will waste you time instead of helping.)

In the next posts, I will upload 2 comparisons made from the same source text. These will be in EN2FR and EN2CZ. EN2FR is a language pair where MT perform well since they have a lot of resources available for training and EN2CZ has far less resources.
Then moving from English as a source, results differ a lot. For example, in DE2FR usually DeepL performs better.
Feel free to upload your findings. If you want to compare with the same text, it’s from https://www.bbc.com/news/business-52504187
I will put it below so you can copy-paste. This text has different levels of difficulty in each paragraph so it’s a very interesting sample.
My bests

Source text:
Tesla boss Elon Musk wiped $14bn (£11bn) off the carmaker's value after tweeting its share price was too high.

It also knocked $3bn off Mr Musk's own stake in Tesla as investors promptly bailed out of the company.

"Tesla stock price too high imo," he said in one of several tweets that included a vow to sell his possessions.

In other tweets, he said his girlfriend was mad at him, while another simply read: "Rage, rage against the dying of the light of consciousness."
Collapse


Monica Carneiro
Morano El-Kholy
 

Philippe Locquet  Identity Verified
Portugal
Local time: 19:52
English to French
+ ...
TOPIC STARTER
EN2FR Aug 10

Here goes the snapshot.
Difficult to have 10 columns in the same picture... I had to remove manually some returns so it could fit
9 engines tested here.



My overall winner IMO is "ModernMT"

Engines give different results some are better than others in different aspects.


 

Philippe Locquet  Identity Verified
Portugal
Local time: 19:52
English to French
+ ...
TOPIC STARTER
EN2CZ Aug 10

So here this language pair is more taxing on MTs and as you see the 9 MT providers are a bit different since not all MT providers provide services for the same language pairs.



I don't speak Czech, but a colleague told me that "ModernMT" came on top here again.

Let me know what you think and who's your winner in your languape pair + field.

... See more
So here this language pair is more taxing on MTs and as you see the 9 MT providers are a bit different since not all MT providers provide services for the same language pairs.



I don't speak Czech, but a colleague told me that "ModernMT" came on top here again.

Let me know what you think and who's your winner in your languape pair + field.

My bests
Collapse


 

TranslateWithMe
Poland
Local time: 20:52
English to Polish
+ ...
My opinion Aug 10

Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I hav
... See more
Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I haven't paid a penny for it).

https://cloud.google.com/free

The information how to get API for Google Translate can be found online.

Cheers
K.
Collapse


Philippe Locquet
 

Philippe Locquet  Identity Verified
Portugal
Local time: 19:52
English to French
+ ...
TOPIC STARTER
Switching MTs ? Aug 12

TranslateWithMe wrote:

Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I haven't paid a penny for it).

https://cloud.google.com/free

The information how to get API for Google Translate can be found online.

Cheers
K.


While thinking about this, it is a fact that because of language pair or translation domain, one MT is better than the other.

Fort such reason do you subscribe to two or more MTs and switch between them or is it not something to consider because of cost issues?

My bests 😊


 

Philippe Locquet  Identity Verified
Portugal
Local time: 19:52
English to French
+ ...
TOPIC STARTER
Comparative MT study available! Aug 20

Hi all, a comparative study report has just been published by Intento.

https://try.inten.to/mt_report_2020?

They did a comparative test study between 15 MT engines across 14 language pairs. They made it specific too to 16 industry sectors.

If you are considering getting an MT engine, I strongly suggest you check it out! Once you spot which is best for you, make
... See more
Hi all, a comparative study report has just been published by Intento.

https://try.inten.to/mt_report_2020?

They did a comparative test study between 15 MT engines across 14 language pairs. They made it specific too to 16 industry sectors.

If you are considering getting an MT engine, I strongly suggest you check it out! Once you spot which is best for you, make a few tests of your own with that engine and compare it to two or three others. That will make for a very educated purchase!!!

They explain their methodology, scoring etc. (BERT, BLEURT…). This can be foreign concept to some translators, so I suggest you jump to these pages:

P 25 for ranking by industry sectors
P 29 for ranking by content types

There is also a call for volunteers rewarded with some free access.
I’ll open a different thread about this.

Hope this is useful!
Collapse


Morano El-Kholy
 

Samuel Murray  Identity Verified
Netherlands
Local time: 20:52
Member (2006)
English to Afrikaans
+ ...
The report is a KEY file Aug 21

Philippe Locquet wrote:
https://try.inten.to/mt_report_2020?


FWIW, the report is in KEY format. This converter worked for me:
https://cloudconvert.com/key-to-pdf


Philippe Locquet
 

TranslateWithMe
Poland
Local time: 20:52
English to Polish
+ ...
I use one MT Aug 22

Philippe Locquet wrote:

TranslateWithMe wrote:

Hi,

I have used some MTs over the years and I would also choose MTModern as the best. A few months ago, it was offered for free due to lockdowns, so I used it at that time, and it is really good.

But, if you need an equally good MT for free, you can use Google Translate API. When you register in Google Cloud (Console) for the first time, you get 300 USD of credit to be used for their services, including their MT, for one year, which is usually enough for one year (I haven't paid a penny for it).

https://cloud.google.com/free

The information how to get API for Google Translate can be found online.

Cheers
K.


While thinking about this, it is a fact that because of language pair or translation domain, one MT is better than the other.

Fort such reason do you subscribe to two or more MTs and switch between them or is it not something to consider because of cost issues?

My bests 😊


From my experience, I use only Google Translate API now, as it is simply the best in both my language pairs: EN-PL, PL-EN. It does not matter if the text is very simple or technical. With technical texts, there is more "proofreading".


Philippe Locquet
 


To report site rules violations or get help, contact a site moderator:


You can also contact site staff by submitting a support request »

Comparing MT engines, which MT should I choose?

Advanced search






TM-Town
Manage your TMs and Terms ... and boost your translation business

Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.

More info »
SDL Trados Business Manager Lite
Create customer quotes and invoices from within SDL Trados Studio

SDL Trados Business Manager Lite helps to simplify and speed up some of the daily tasks, such as invoicing and reporting, associated with running your freelance translation business.

More info »



Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search