Standard Analyzer

The standard analyzer is the default for all MongoDB Search indexes and queries. It divides text into terms based on word boundaries, which makes it language-neutral for most use cases. It converts all terms to lower case and removes punctuation. It provides grammar-based tokenization that recognizes email addresses, acronyms, Chinese-Japanese-Korean characters, alphanumerics, and more.

If you select Refine Your Index, the Atlas UI displays a section titled View text analysis of your selected index configuration within the Index Configurations section. If you expand this section, the Atlas UI displays the index and search tokens that the standard analyzer generates for each sample string. You can see the tokens that the standard analyzer creates for a built-in sample document and query string when you create or edit an index in the Atlas UI Visual Editor.

Important

MongoDB Search won't index string fields where analyzer tokens exceed 32766 bytes in size. If using the keyword analyzer, string fields which exceed 32766 bytes will not be indexed.

Example

The following example index definition specifies an index on the title field in the sample_mflix.movies collection using the standard analyzer. To follow along with this example, load the sample data on your cluster and either use mongosh or navigate to the Create a Search Index page in the Atlas UI following the steps in the Create a MongoDB Search Index tutorial.

Then, using the movies collection as your data source, follow the example procedure to create an index from mongosh or the Atlas UI Visual Editor or JSON editor.

➤ Use the Select your language drop-down menu to set the interface for the example on this page.

Interface

The following query searches the title field for the term action and limits the output to two results.

MongoDB Search returned these documents because it matched the query term action to the token action for the documents, which MongoDB Search created by doing the following for the text in the title field using the lucene.standard analyzer:

Convert the text to lowercase.
Split the text based on word boundaries and create separate tokens.

The following table shows the tokens (searchable terms) that MongoDB Search creates using the Standard Analyzer and, by contrast, the tokens that MongoDB Search creates for the Keyword Analyzer and Whitespace Analyzer for the documents in the results:

Title	Standard Analyzer Tokens	Keyword Analyzer Tokens	Whitespace Analyzer Tokens
`Action Jackson`	`action`, `jackson`	`Action Jackson`	`Action`, `Jackson`
`Class Action`	`class`, `action`	`Class Action`	`Class`, `Action`

If you index the field using the:

Keyword Analyzer, MongoDB Search wouldn't match the documents in the results for the query term action because the keyword analyzer matches only documents in which the search term matches the entire contents of the field (Action Jackson and Class Action) exactly.
Whitespace Analyzer, MongoDB Search wouldn't match the documents in the results for the query term action because the whitespace analyzer tokenizes the title field value in its original case (Action) and the query term has the lowercase action, which doesn't match the whitespace analyzer token.

Back

Analyzers

Simple