This site uses cookies. Some of these cookies are essential to the operation of the site, while others help to improve your experience by providing insights into how the site is being used. For more information, please see the ProZ.com privacy policy.
Cookie settings

Technical forums » OmegaT support »
snowball vs. lucene tokenizers
Thread poster: Deborah Kolosova

Deborah Kolosova  Identity Verified
United States
Russian to English
+ ...
Nov 3, 2011

The instructions on the OmegaT site for installing tokenizers say you should select the appropriate tokenizer from the list. For my source language, Russian, there are two listed: the SnowballRussianTokenizer and the LuceneRussianTokenizer. What is the difference, and which one is the best to use? Or do they each have their own advantages?
 

Susan Welsh  Identity Verified
United States
Local time: 21:08
Member (2008)
Russian to English
+ ...
I use lucene Nov 3, 2011

My recollection of past discussions is that lucene has a "stop word" function that snowball does not (meaning it ignores little irrelevant words like "and" and "the" when matching segments). Someone will probably correct me if I'm wrong. You can try them both and see what you like.

I translate from Russian, and lucene works great for me.

Susan
 


There is no moderator assigned specifically to this forum.
To report site rules violations or get help, please contact site staff »


snowball vs. lucene tokenizers

Advanced search






SDL MultiTerm 2019
Guarantee a unified, consistent and high-quality translation with terminology software by the industry leaders.

SDL MultiTerm 2019 allows translators to create one central location to store and manage multilingual terminology, and with SDL MultiTerm Extract 2019 you can automatically create term lists from your existing documentation to save time.

More info »
PerfectIt consistency checker
Faster Checking, Greater Accuracy

PerfectIt helps deliver error-free documents. It improves consistency, ensures quality and helps to enforce style guides. It’s a powerful tool for pro users, and comes with the assurance of a 30-day money back guarantee.

More info »

Recent posts | FAQ | Rules | Moderators | Article knowledgebase

Your current localization setting

English

Select a language

More languages...
Close search
Forums
  • All of ProZ.com
  • Term search
  • Jobs
  • Forums
  • Multiple search