For AI agents: a documentation index is available at https://www.mongodb.com/docs/llms.txt — markdown versions of all pages are available by appending .md to any URL path.
Docs Menu
Docs Home
/ /
Custom
Docs Home
/ /
Analyzers
/
Custom
Docs Home
/
Development
/
Search
/
Queries & Indexes
/
Index Reference
/
Analyzers
/
Custom

Tokenizers for Custom Search Analyzers

A custom analyzer's tokenizer determines how MongoDB Search splits up text into discrete chunks for indexing. Tokenizers require a type field, and some take additional options as well.

Syntax
"tokenizer": {
  "type": "<tokenizer-type>",
  "<additional-option>": "<value>"
}

The sample index definitions and queries on this page use the sample collection named minutes. To try these examples, load the minutes collection on your cluster and navigate to the Create a Search Index page in the Atlas UI following the steps in the Create a MongoDB Search Index tutorial. Then, select the minutes collection as your data source, and follow the example procedure to create an index from the Atlas UI or using mongosh.

edgeGram

The edgeGram tokenizer tokenizes input from the left side, or "edge", of a text input into n-grams of given sizes. You can't use a custom analyzer with edgeGram tokenizer in the analyzer field for synonym or autocomplete field mapping definitions.

Attributes

The edgeGram tokenizer has the following attributes:

Note

The edgeGram tokenizer yields multiple output tokens per word and across words in input text, producing token graphs.

Because autocomplete field type mapping definitions and analyzers with synonym mappings only work when used with non-graph-producing tokenizers, you can't use a custom analyzer with edgeGram tokenizer in the analyzer field for autocomplete field type mapping definitions or analyzers with synonym mappings.

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be edgeGram.

minGram

integer

yes

Number of characters to include in the shortest token created.

maxGram

integer

yes

Number of characters to include in the longest token created.

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named edgegramExample. It uses the edgeGram tokenizer to create tokens (searchable terms) between 2 and 7 characters long starting from the first character on the left side of words in the message field.

The following query searches the message field in the minutes collection for text that begin with tr.

MongoDB Search returns documents with _id: 1 and _id: 3 in the results because MongoDB Search created a token with the value tr using the edgeGram tokenizer for the documents, which matches the search term. If you index the message field using the standard tokenizer, MongoDB Search would not return any results for the search term tr.

The following table shows the tokens that the edgeGram tokenizer and by comparison, the standard tokenizer, create for the documents in the results:

Tokenizer
Token Outputs

standard

try, to, sign, in

edgeGram

tr, try, try{SPACE}, try t, try to, try to{SPACE}

keyword

The keyword tokenizer tokenizes the entire input as a single token. MongoDB Search doesn't index string fields that exceed 32766 characters using the keyword tokenizer.

Attributes

The keyword tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be keyword.

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named keywordExample. It uses the keyword tokenizer to create a token (searchable terms) on the entire field as a single term.

The following query searches the message field in the minutes collection for the phrase try to sign-in.

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value try to sign-in using the keyword tokenizer for the documents, which matches the search term. If you index the message field using the standard tokenizer, MongoDB Search returns documents with _id: 1, _id: 2 and _id: 3 for the search term try to sign-in because each document contains some of the tokens the standard tokenizer creates.

The following table shows the tokens that the keyword tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Tokenizer
Token Outputs

standard

try, to, sign, in

keyword

try to sign-in

nGram

The nGram tokenizer tokenizes into text chunks, or "n-grams", of given sizes. You can't use a custom analyzer with nGram tokenizer in the analyzer field for synonym or autocomplete field mapping definitions.

Attributes

The nGram tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be nGram.

minGram

integer

yes

Number of characters to include in the shortest token created.

maxGram

integer

yes

Number of characters to include in the longest token created.

Example

The following index definition indexes the title field in the minutes collection using a custom analyzer named ngramExample. It uses the nGram tokenizer to create tokens (searchable terms) between 4 and 6 characters long in the title field.

The following query searches the title field in the minutes collection for the term week.

MongoDB Search returns the document with _id: 1 in the results because MongoDB Search created a token with the value week using the nGram tokenizer for the documents, which matches the search term. If you index the title field using the standard or edgeGram tokenizer, MongoDB Search would not return any results for the search term week.

The following table shows the tokens that the nGram tokenizer and by comparison, the standard and edgeGram tokenizer create for the document with _id: 1:

Tokenizer
Token Outputs

standard

The, team's, weekly, meeting

edgeGram

The{SPACE}, The t, The te

nGram

The{SPACE}, The t, The te, he t, ... , week, weekl, weekly, eekl, ..., eetin, eeting, etin, eting, ting

regexCaptureGroup

The regexCaptureGroup tokenizer matches a Java regular expression pattern to extract tokens.

Tip

To learn more about Java regular expression syntax, see the Pattern class in the Java documentation.

Attributes

The regexCaptureGroup tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be regexCaptureGroup.

pattern

string

yes

Regular expression to match against.

group

integer

yes

Index of the character group within the matching expression to extract into tokens. Use 0 to extract all character groups.

Example

The following index definition indexes the page_updated_by.phone field in the minutes collection using a custom analyzer named phoneNumberExtractor. It uses the following:

  • mappings character filter to remove parenthesis around the first three digits and replace all spaces and periods with dashes

  • regexCaptureGroup tokenizer to create a single token from the first US-formatted phone number present in the text input

The following query searches the page_updated_by.phone field in the minutes collection for the phone number 123-456-9870.

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value 123-456-7890 using the regexCaptureGroup tokenizer for the documents, which matches the search term. If you index the page_updated_by.phone field using the standard tokenizer, MongoDB Search returns all of the documents for the search term 123-456-7890.

The following table shows the tokens that the regexCaptureGroup tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Tokenizer
Token Outputs

standard

123, 456.9870

regexCaptureGroup

123-456-9870

regexSplit

The regexSplit tokenizer splits tokens with a Java regular-expression based delimiter.

Tip

To learn more about Java regular expression syntax, see the Pattern class in the Java documentation.

Attributes

The regexSplit tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be regexSplit.

pattern

string

yes

Regular expression to match against.

Example

The following index definition indexes the page_updated_by.phone field in the minutes collection using a custom analyzer named dashDotSpaceSplitter. It uses the regexSplit tokenizer to create tokens (searchable terms) from one or more hyphens, periods and spaces on the page_updated_by.phone field.

The following query searches the page_updated_by.phone field in the minutes collection for the digits 9870.

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value 9870 using the regexSplit tokenizer for the documents, which matches the search term. If you index the page_updated_by.phone field using the standard tokenizer, MongoDB Search would not return any results for the search term 9870.

The following table shows the tokens that the regexCaptureGroup tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Tokenizer
Token Outputs

standard

123, 456.9870

regexSplit

(123), 456, 9870

standard

The standard tokenizer tokenizes based on word break rules from the Unicode Text Segmentation algorithm.

Attributes

The standard tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be standard.

maxTokenLength

integer

no

Maximum length for a single token. Tokens greater than this length are split at maxTokenLength into multiple tokens.

Default: 255

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named standardExample. It uses the standard tokenizer and the stopword token filter.

The following query searches the message field in the minutes collection for the term signature.

MongoDB Search returns the document with _id: 4 because MongoDB Search created a token with the value signature using the standard tokenizer for the documents, which matches the search term. If you index the message field using the keyword tokenizer, MongoDB Search would not return any results for the search term signature.

The following table shows the tokens that the standard tokenizer and by comparison, the keyword analyzer, create for the document with _id: 4:

Tokenizer
Token Outputs

standard

write, down, your, signature, or, phone

keyword

write down your signature or phone №

uaxUrlEmail

The uaxUrlEmail tokenizer tokenizes URLs and email addresses. Although uaxUrlEmail tokenizer tokenizes based on word break rules from the Unicode Text Segmentation algorithm, we recommend using uaxUrlEmail tokenizer only when the indexed field value includes URLs and email addresses. For fields that don't include URLs or email addresses, use the standard tokenizer to create tokens based on word break rules.

Attributes

The uaxUrlEmail tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be uaxUrlEmail.

maxTokenLength

int

no

Maximum number of characters in one token.

Default: 255

Example

Basic Example

Create a simple index using the uaxUrlEmail tokenizer.

The following index definition indexes the page_updated_by.email field in the minutes collection using a custom analyzer named basicEmailAddressAnalyzer. It uses the uaxUrlEmail tokenizer to create tokens (searchable terms) from URLs and email addresses in the page_updated_by.email field.

The following query searches the page_updated_by.email field in the minutes collection for the email lewinsky@example.com.

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value lewinsky@example.com using the uaxUrlEmail tokenizer for the documents, which matches the search term. If you index the page_updated_by.email field using the standard tokenizer, MongoDB Search returns all the documents for the search term lewinsky@example.com.

The following table shows the tokens that the uaxUrlEmail tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Tokenizer
Token Outputs

standard

lewinsky, example.com

uaxUrlEmail

lewinsky@example.com

Advanced Example

Create an index using the uaxUrlEmail tokenizer with autocomplete tokenization strategy.

The following index definition indexes the page_updated_by.email field in the minutes collection using a custom analyzer named emailAddressAnalyzer. It uses the following:

The following query searches the page_updated_by.email field in the minutes collection for the term exam.

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value lewinsky@example.com using the uaxUrlEmail tokenizer for the documents, which matches the search term. If you index the page_updated_by.email field using the standard tokenizer, MongoDB Search returns all the documents for the search term lewinsky@example.com.

The following table shows the tokens that the uaxUrlEmail tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Tokenizer
MongoDB Search Field Type
Token Outputs

standard

autocomplete edgeGram

le, lew, lewi, lewin, lewins, lewinsk, lewinsky, lewinsky@, lewinsky, ex, exa, exam, examp, exampl, example, example., example.c, example.co, example.com

uaxUrlEmail

autocomplete edgeGram

le, lew, lewi, lewin, lewins, lewinsk, lewinsky, lewinsky@, lewinsky@e, lewinsky@ex, lewinsky@exa, lewinsky@exam, lewinsky@examp, lewinsky@exampl

whitespace

The whitespace tokenizer tokenizes based on occurrences of whitespace between words.

Attributes

The whitespace tokenizer has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this tokenizer type. Value must be whitespace.

maxTokenLength

integer

no

Maximum length for a single token. Tokens greater than this length are split at maxTokenLength into multiple tokens.

Default: 255

Example

The following index definition indexes the message field in the minutes collection using a custom analyzer named whitespaceExample. It uses the whitespace tokenizer to create tokens (searchable terms) from any whitespaces in the message field.

The following query searches the message field in the minutes collection for the term SIGN-IN.

MongoDB Search returns the document with _id: 3 in the results because MongoDB Search created a token with the value sign-in using the whitespace tokenizer for the documents, which matches the search term. If you index the message field using the standard tokenizer, MongoDB Search returns documents with _id: 1, _id: 2 and _id: 3 for the search term sign-in.

The following table shows the tokens that the whitespace tokenizer and by comparison, the standard tokenizer, create for the document with _id: 3:

Tokenizer
Token Outputs

standard

try, to, sign, in

whitespace

try, to, sign-in

Back

Character Filters

Next

Token Filters

On this page