Understanding Analyzers

Character Filters remove undesirable characters from input: for example, the html character filter removes HTML tags, and indexes HTML text-content alone.

Tokenizers split input-strings into individual tokens, which together are made into a token stream. The nature of the decision-making whereby splits are made differs across tokenizers.

Token Filters are chained together, with each performing additional post-processing on each token in the stream provided by the tokenizer. This may include reducing tokens to the stems of the dictionary-based words from which they were derived, removing any remaining punctuation from tokens, and removing certain tokens deemed unnecessary.

asciifolding: Converts characters that are in the first 127 ASCII characters (basic latin unicode block) into their ASCII equivalents.

html: Removes html elements such as <p>.

regexp: Uses a regular expression to match characters which should be replaced with the specified replacement string.

zero_width_spaces: Substitutes a regular space-character for each zero-width non-joiner space.

Letter: Creates tokens by breaking input-text into subsets that consist of letters only: characters such as punctuation-marks and numbers are omitted. Creation of a token ends whenever a non-letter character is encountered. For example, the text Reqmnt: 7-element phrase would return the following tokens: Reqmnt, element, and phrase.

Single: Creates a single token from the entirety of the input-text. For example, the text in each place would return the following token: in each place. Note that this may be useful for handling URLs or email-addresses, which can thus be prevented from being broken at punctuation or special-character boundaries. It may also be used to prevent multi-word phrases (for example, placenames such as Milton Keynes or San Francisco) from being broken up due to whitespace; so that they become indexed as a single term.

Unicode: Creates tokens by performing Unicode Text Segmentation on word-boundaries, using the segment library. For examples, see Unicode Word Boundaries.

Web: Creates tokens by identifying and removing html tags. For example, the text <h1>Introduction<\h1> would return the token Introduction.

Whitespace: Creates tokens by breaking input-text into subsets according to where whitespace occurs. For example, the text in each place would return the following tokens: in, each, and place.

apostrophe: Removes all characters after an apostrophe, and the apostrophe itself. For example, they’ve becomes they.

camelCase: Splits camelCase text to tokens.

dict_compound: Allows user-specification of a dictionary whose words can be combined into compound forms, and individually indexed.

edge_ngram: From each token, computes n-grams that are rooted either at the front or the back.

elision: Identifies and removes characters that prefix a term and are separated from it by an apostrophe. For example, in French, l’avion becomes avion.

keyword_marker: Identifies keywords and marks them as such. These are then ignored by any downstream stemmer.

length: Removes tokens that are too short or too long for the stream.

to_lower: Converts all characters to lower case.

ngram: From each token, computes n-grams. There are two parameters, which are the minimum and maximum n-gram length.

reverse: Simply reverses each token.

shingle: Computes multi-token shingles from the token stream. For example, the token stream the quick brown fox, when configured with a shingle minimum and a shingle maximum length of 2, produces the tokens the quick, quick brown, and brown fox.

stemmer_porter: Transforms the token stream as per the porter stemming algorithm.

stemmer_snowball: Uses libstemmer to reduce tokens to word-stems.

stop_tokens: Removes from the stream tokens considered unnecessary for a Full Text Search. For example, and, is, and the. For example, HTML becomes html.

truncate: Truncates each token to a maximum-permissible token-length.

normalize_unicode: Converts tokens into Unicode Normalization Form.

unique: Only indexes unique tokens during analysis.

keyword: Creates a single token representing the entire input, which is otherwise unchanged. This forces exact matches, and preserves characters such as spaces.

simple: Analysis by means of the Unicode tokenizer and the to_lower token filter.

standard: Analysis by means of the Unicode tokenizer, the to_lower token filter, and the stop token filter.

web: Analysis by means of the Web tokenizer and the to_lower token filter.