ProZ.com Translation Forums

TMX fixer | Hmm

Michael Beijer — Sun, 05 Oct 2014 12:52:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Michael Beijer
Post title: Hmm

I admit I know nothing about AutoIT and I use AHK mainly for hotkeys and keyboard remapping, so I'm probably biased.

Incidentally, it looks like the AHK is currently splitting off into two groups:

• The original one: [url removed]
+
• A new one (based on AutoHotkey_L, which is the one I use): [url removed]

TMX fixer | AHK vs AU3

Samuel Murray — Sun, 05 Oct 2014 12:17:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: AHK vs AU3

[quote]Michael Beijer wrote:
It's a shame it's not written in AHK. [/quote]

AutoIt is superior to AutoHotKey in all things except hotkeys and keyboard remapping.

TMX fixer | Thanks Samuel!

Michael Beijer — Sun, 05 Oct 2014 11:09:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Michael Beijer
Post title: Thanks Samuel!

It's a shame it's not written in AHK.

Michael

TMX fixer | @Michael

Samuel Murray — Sat, 04 Oct 2014 08:58:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: @Michael

[quote]Michael Beijer wrote:
[quote]Samuel Murray wrote:
Here is my small TMX fixer (very basic):
[url removed] [/quote]
Can I have a new link? [/quote]

Sure... [url removed]

I haven't used it in a while, but it worked for me (mostly). You need to install AutoIt to make it work.

TMX fixer | @Samuel:

Michael Beijer — Fri, 03 Oct 2014 22:14:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Michael Beijer
Post title: @Samuel:

[quote]Samuel Murray wrote:

Here is my small TMX fixer (very basic):
[url removed]

The link will last 90 days. Requires Windows. Assumes TMX is UTF8. Removes or marks up invalid characters and fixes or marks up broken entities. Comments welcome.
[/quote]

Can I have a new link?

Michael

TMX fixer | Yup

Michael Beijer — Fri, 03 Oct 2014 21:51:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Michael Beijer
Post title: Yup

[quote]Julio A. Juncal wrote:

[quote]Michael Beijer wrote:

I seem to remember running a few TMs (.tmx) through Xbench, which fixed them in the sense that they could then be imported into a program that had previously balked.

Michael [/quote]

Thanks, Michael! Just used Xbench to fix a defective .tmx file. [/quote]

Just had to repair another TMX file and Xbench is the quickest way to do it. Just run it through Xbench, which will clean it for you in the process.

Second easiest method is to use the import command in Olifant and note down the line numbers of the errors in the error thingee and clean them manually in a text editor.

Michael

TMX fixer | Xbench does fix .tmx files

Julio A. Juncal — Sat, 15 Mar 2014 01:13:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Julio A. Juncal
Post title: Xbench does fix .tmx files

[quote]Michael Beijer wrote:

I seem to remember running a few TMs (.tmx) through Xbench, which fixed them in the sense that they could then be imported into a program that had previously balked.

Michael [/quote]

Thanks, Michael! Just used Xbench to fix a defective .tmx file.

TMX fixer | But Okapi Rainbow is free

Piotr Bienkowski — Sun, 15 Apr 2012 18:35:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Piotr Bienkowski
Post title: But Okapi Rainbow is free

[quote]Jan Sundström wrote:

Hi!

Sorry for the late reply, maybe this can still help someone.
[url removed]

Functionality, among others:
"ability to easily clean and repair translation memory files to ensure a well maintained translation memory library free of corrupt characters and codes"

/Jan [/quote]

And does exactly what the topic starter asked.

Regards,

Piotr

TMX fixer | Heartsome TMX

Jan Sundström — Fri, 13 Apr 2012 12:26:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Jan Sundström
Post title: Heartsome TMX

Hi!

Sorry for the late reply, maybe this can still help someone.
[url removed]

Functionality, among others:
"ability to easily clean and repair translation memory files to ensure a well maintained translation memory library free of corrupt characters and codes"

/Jan

TMX fixer | TMX Fixer basic

Samuel Murray — Sat, 04 Jun 2011 20:34:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: TMX Fixer basic

Here is my small TMX fixer (very basic):
[url removed]

The link will last 90 days. Requires Windows. Assumes TMX is UTF8. Removes or marks up invalid characters and fixes or marks up broken entities. Comments welcome.

TMX fixer | Limited number of entities...

Samuel Murray — Tue, 31 May 2011 20:46:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: Limited number of entities...

[quote]FarkasAndras wrote:
TMX only allows a very limited set of character entities anyway (amp, lt, gt, quot, apos and maybe their numerical versions)...[/quote]

Oh, yeah, I seem to remember something like that now, yes. Well, that is good news then. Makes my life a little simpler. My own TMX fixer will probably just delete misbehaving segments outright, to keep the quality of the TM as high as possible.

TMX fixer | The list of XML entities is short

Didier Briel — Tue, 31 May 2011 20:45:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Didier Briel
Post title: The list of XML entities is short

[quote]Samuel Murray wrote:
In my case, since my files are already TMXes, I'm going to have to figure out how to check if something is a valid entity if it starts with an ampersand. This means finding out what characters are not allowed in an entity.
[/quote]
In addition to the list I gave, the only one missing was '
Since it is TMX, no other entities can be defined, so you just have to deal with pure XML entities, as defined in the standard:
[url= [url removed] #sec-predefined-ent]XML pre-defined entities[/url]

[quote] FarkasAndras wrote:
You can just use the character entities in such situations.
So you can write & and it comes through as &
[/quote]
Thank you for the reminder.
(I knew, but I was being lazy.)

Didier

TMX fixer | That works, too

FarkasAndras — Tue, 31 May 2011 20:36:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: FarkasAndras
Post title: That works, too

[quote]Samuel Murray wrote:

[quote]s/&/& a m p ;/g;[/quote]

In my case, since my files are already TMXes, I'm going to have to figure out how to check if something is a valid entity if it starts with an ampersand. This means finding out what characters are not allowed in an entity.

[/quote]
You don't necessarily need to check for the validity of character entities. Replacing all & with && "fixes" illegal character entities by converting them to an ampersand entity followed by some random characters. It should make your file pass the import filter, but of course it will also break any correct character entities that might be in the file. At that point, you might as well do s/&\S+//, of course.
If there are clear trends in [i]how[/i] your entities are broken, you could try more refined solutions, or you could just pass the file through some filter that converts entities to literal characters, then remove corrupted character entities and convert back. TMX only allows a very limited set of character entities anyway (amp, lt, gt, quot, apos and maybe their numerical versions), so this would be the correct solution.

TMX fixer | Unusual characters in forum

FarkasAndras — Tue, 31 May 2011 20:23:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: FarkasAndras
Post title: Unusual characters in forum

[quote]Didier Briel wrote:

I have added a space between the & and the entity name, otherwise the forum code interprets it. I had also to use a name for the lesser and greater than characters.
[/quote]

You can just use the character entities in such situations.
So you can write &amp; and it comes through as &
Similarly, you can do &lt; and get < (in this post, I added yet another amp; to make these show up correctly in the forum).
Thanks for the list, though. Perhaps I'll make a simple TMX fixer .exe for the uninitiated among us who might need it. With your list, it'll take 5 minutes.

TMX fixer | @Didier

Samuel Murray — Tue, 31 May 2011 19:52:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: @Didier

[quote]Didier Briel wrote:
From bligner source code:
s/[\x00-\x08]|\x0B|\x0C|[\x0E-\x1F]//g;
[/quote]

Thanks, this is useful.

[quote]s/&/& a m p ;/g;[/quote]

In my case, since my files are already TMXes, I'm going to have to figure out how to check if something is a valid entity if it starts with an ampersand. This means finding out what characters are not allowed in an entity.

TMX fixer | The list is short

Didier Briel — Tue, 31 May 2011 15:55:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Didier Briel
Post title: The list is short

[quote]Samuel Murray wrote:

Alternatively, does anyone know of a list of the most commonly encountered invalid/illegal XML characters? If I can have such a list, then I can write my own checker. [/quote]

From bligner source code:

s/&/& amp;/g;
s/lesser_than_char/& lt;/g;
s/greater_than_char/& gt;/g;
s/"/& quot;/g;
s/[\x00-\x08]|\x0B|\x0C|[\x0E-\x1F]//g;

I have added a space between the & and the entity name, otherwise the forum code interprets it. I had also to use a name for the lesser and greater than characters.

(In some circumstances " might not be invalid, for instance, but let's make it simple.)

Note that Rainbow will already clean these, as Piotr has written.

Didier

[Edited at 2011-05-31 15:56 GMT]

TMX fixer | Alternatively

Samuel Murray — Tue, 31 May 2011 14:24:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: Alternatively

Alternatively, does anyone know of a list of the most commonly encountered invalid/illegal XML characters? If I can have such a list, then I can write my own checker.

TMX fixer | @Nathalie

Samuel Murray — Tue, 31 May 2011 13:30:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: @Nathalie

[quote]Natalie wrote:
I use ver.3.0.8 ... The steps are described above (this is a quotation from Olifant help).[/quote]

Thanks, I didn't realise the quoted text was the steps.

[quote]You can look through your TM and decide if you wish to replace these letters or just delete the segments.[/quote]

Thanks, but I don't want to "look through the TM". I want to spend as close to zero time on TM admin and most of my time on actual translation.

TMX fixer | Manually is not an option

Samuel Murray — Tue, 31 May 2011 12:35:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: Manually is not an option

[quote]Piotr Bienkowski wrote:
...for example UltraEdit does parse XML (and hence TMX) files, and takes you directly to the place where the problem is, so that you can fix it manually.[/quote]

Imagine a file with 15000 segments, with 1000 problem segments. Manual editing is not an option, sorry.

TMX fixer | Okapi Rainbow?

Piotr Bienkowski — Tue, 31 May 2011 12:32:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Piotr Bienkowski
Post title: Okapi Rainbow?

[url removed]

It has Fix illegal characters under XML utilities, and since TMX is XML it should work for TMX too.

As for unterminated entities, not sure whether Rainbow can fix it, but for example UltraEdit does parse XML (and hence TMX) files, and takes you directly to the place where the problem is, so that you can fix it manually.

Regards,

Piotr

TMX fixer | --

Natalie — Tue, 31 May 2011 12:14:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Natalie
Post title: --

I use ver.3.0.8 and quite recently opened tmx files with 78900 and 92K segments without any problems; I haven't heard about any size limitations for tmx files.

The steps are described above (this is a quotation from Olifant help). You can look through your TM and decide if you wish to replace these letters or just delete the segments.

TMX fixer | Just a wild guess, but

Michael Beijer — Tue, 31 May 2011 12:10:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Michael Beijer
Post title: Just a wild guess, but

I seem to remember running a few TMs (.tmx) through Xbench, which fixed them in the sense that they could then be imported into a program that had previously balked.

Michael

TMX fixer | Steps?

Samuel Murray — Tue, 31 May 2011 12:03:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray
Post title: Steps?

[quote]Natalie wrote:
Olifant can do this for you.[/quote]

Can you please tell me which version of Olifant, and what the steps are?

Because I have used Olifant myself (version 308) and it isn't really helpful. In fact, I have a WF TM of 15000 segments, but Olifant only imports 5000 of the segments. And I have checked some of the segments that were no imported -- they contain no errors.

TMX fixer | Why not Olifant?

Natalie — Tue, 31 May 2011 11:58:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Natalie
Post title: Why not Olifant?

Hi Samuel, Olifant can do this for you:

[quote]
How to open a TM that contains invalid XML characters?

Some TMX files may come with control characters that are invalid in XML document (often found in RTF-generated style sheet section). When one or more characters is in the TMX file, an error "hexadecimal value 0xHH is an invalid character" occurs when opening or importing the TMX file.

To open such TMX file in Olifant:
Select the Import command from the File menu.
Select the path of the TMX file to import.
Click Open.
An TMX Import Options dialog box opens.
Make sure the option Check for invalid characters is set.
Click OK.

A temporary copy of the file is created, where each invalid XML character will be replaced by _#xHHHH_ where HHHH is the Unicode hexadecimal value of the character. Then the temporary file is open. Note that these characters are left like this after: Olifant does not convert them back automatically when saving the file. You have to decide yourselves what you want to do with them.

You should use this option only when needed, as it increases the time it takes to open the file.
[/quote]

TMX fixer

Samuel Murray — Tue, 31 May 2011 11:52:00 +0000

Forum: CAT Tools Technical Help
Topic: TMX fixer
Poster: Samuel Murray

G'day everyone

I'm looking for a program that can fix a non-compliant TMX file for me in a jiffy. I'm using Windows XP Pro SP2.

My TMX file contains illegal characters and it contains unterminated entities in some segments. The program in which I want to use the TMX file isn't intelligent enough to simply ignore these errors -- no, the program stops *(and dies) at every error.

What I need is a program that will fix the TMX. In other words, it should seek out illegal characters and simply delete them (or replace them with placeholder characters), and it should seek out unterminated entities and simply delete or dummy-fix them (e.g. by changing a lone ampersand to an entitised ampersand). Or... it should simply delete the segments that are non-conformant.

Do you know of such a program?

Thanks
Samuel

[Edited at 2011-05-31 11:54 GMT]