Tokenization

Definition

Tokenization splits text into the units a search or analysis system works with, usually words or word pieces. It sounds trivial but decides what can match: in languages written without spaces, such as Chinese, finding the word boundaries is a task in its own right.

In plain terms

Even in English there are choices: is “state-of-the-art” one token or four, is “U.S.” one, what happens to “don’t” and to numbers like “3.14”? Each choice changes which queries match which documents. Rules such as lower-casing, and later steps such as reducing words to a stem, come after tokenization.

For Chinese, Japanese and Thai there are no spaces between words, so systems use dictionaries, statistical models, or simply overlapping character sequences. What matters most is that indexing and querying use the same procedure: a query tokenized differently from the documents will miss them.

Example

The Chinese sentence 我喜欢自然语言处理 (“I like natural language processing”), tokenized two ways:

A search for 语言 (“language”) finds this sentence only under the second tokenizer, unless the system also indexes character sequences. English has smaller versions of the same issue, such as “state-of-the-art”.

Common misunderstandings

• “Splitting on spaces is enough.” It fails for languages without spaces and mishandles hyphens, punctuation and numbers.
• “The tokenizer is a detail.” It determines what can be matched and counted, and so affects every later result.
• “Documents and queries may be tokenized differently.” They must use the same procedure or matches are lost.