(I'm taking these quotes out of another thread in this forum, just to make a new topic from it.)
I wrote:
When comparing printers people often use the famous Dr. Grauert letter:
http://de.wikipedia.org/wiki/Dr.-Grauert-Brief Why not create such a benchmark text ourselves in order to test the quality of fuzzy matching and subsegment leverage, in order to compare our CAT tools?
What would be a good language combination for this?
And Michael B. responded:
Interesting idea about the 'Dr. Hans letter'. I suppose you would have to have one in every language someone might want to test. However, once one existed, MOUSE tool vendors could of course cheat by optimising their tool to handle that one page particularly well... Also, there would be the problem of selecting a type of text; technical, legal, literary, etc. Good luck!
I didn't say it would be simple . But I think it is time to take comparisons one step further. It is fascinating to know that CAT tool A has nicer buttons, a crispier font and a stunning grid than CAT tool B. But at the end of the day, what really matters is how well does a CAT tool perform with term recognition and TM matching.
And what could be against CAT tool vendors optimising their tools for a certain set of example documents? We can always make new version, e.g. to include new features of a certain DTP or word processor software.
Compliance with de facto standards as SDLXLIFF is a starting point, not the final goal. BTW, talking about this, I just read a horrible report from Duncan Bell in the Déjà Vu user list at Yahoo Groups. One quote I'd like to present here, because it describes a horrible scenario:
It's no pleasure to me at all to have to say that "very simple,
step-by-step explanations" are not possible in this case! It's a
complicated matter! The need to handle Studio files has added
considerably to our admin effort and time spent not translating but doing necessary peripheral work.
I remember that I once was very enthusiastic about the inter-CAT tool compatibility XLIFF would allow ...