For AI agents: a documentation index is available at https://www.mongodb.com/docs/llms.txt — markdown versions of all pages are available by appending .md to any URL path.
Docs Menu
Docs Home
/ /
Custom
Docs Home
/ /
Analyzers
/
Custom
Docs Home
/
Development
/
Search
/
Queries & Indexes
/
Index Reference
/
Analyzers
/
Custom

Character Filters

Character filters examine text one character at a time and perform filtering operations. Character filters require a type field, and some take additional options as well.

Syntax
"charFilters": [
  {
    "type": "<filter-type>",
    "<additional-option>": <value>
  }
]

The sample index definitions and queries on this page use the sample collection named minutes. To try these examples, load the minutes collection on your cluster and navigate to the Create a Search Index page in the Atlas UI following the steps in the Create a MongoDB Search Index tutorial. Then, select the minutes collection as your data source, and follow the example procedure to create an index from the Atlas UI or using mongosh.

➤ Use the Select your interface drop-down menu to set the method to run the examples on this page.

Character Filter Types

MongoDB Search supports the following types of character filter:

htmlStrip

The htmlStrip character filter strips out HTML constructs.

Attributes

The htmlStrip character filter has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this character filter type. Value must be htmlStrip.

ignoredTags

array of strings

yes

List that contains the HTML tags to exclude from filtering.

Example

The following index definition example indexes the text.en_US field in the minutes collection using a custom analyzer named htmlStrippingAnalyzer. The custom analyzer specifies the following:

The following query looks for occurrences of the string head in the text.en_US field of the minutes collection.

MongoDB Search doesn't return the document with _id: 1 because the string head is part of the HTML tag <head>. The document with _id: 3 contains HTML tags, but the string head is elsewhere so the document is a match. The following table shows the tokens that MongoDB Search generates for the text.en_US field values in documents _id: 1, _id: 2, and _id: 3 in the minutes collection using the htmlStrippingAnalyzer.

Document ID
Output Tokens

_id: 1

This, page, deals, with, department, meetings

_id: 2

The, head, of, the, sales, department, spoke, first

_id: 3

We'll, head, out, to, the, conference, room, by, noon

icuNormalize

The icuNormalize character filter normalizes text with the ICU Normalizer. It is based on Lucene's ICUNormalizer2CharFilter.

Attributes

The icuNormalize character filter has the following attribute:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this character filter type. Value must be icuNormalize.

Example

The following index definition example indexes the message field in the minutes collection using a custom analyzer named normalizingAnalyzer. The custom analyzer specifies the following:

  • Normalize the text in the message field value using the icuNormalize character filter.

  • Tokenize the words in the field based on occurrences of whitespace between words using the whitespace tokenizer.

The following query searches for occurrences of the string no (for number) in the message field of the minutes collection.

MongoDB Search matched document with _id: 4 to the query term no because it normalized the numero symbol № in the field using the icuNormalize character filter and created the token no for that typographic abbreviation of the word "number". MongoDB Search generates the following tokens for the message field value in document _id: 4 using the normalizingAnalyzer:

Document ID
Output Tokens

_id: 4

write, down, your, signature, or, phone, no

mapping

The mapping character filter applies user-specified normalization mappings to characters. It is based on Lucene's MappingCharFilter.

Attributes

The mapping character filter has the following attributes:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this character filter type.

Value must be mapping.

mappings

object

yes

Object that contains a comma-separated list of mappings. A mapping indicates that one character or group of characters should be substituted for another, in the format <original> : <replacement>.

Example

The following index definition example indexes the page_updated_by.phone field in the minutes collection using a custom analyzer named mappingAnalyzer. The custom analyzer specifies the following:

  • Remove instances of hyphen (-), dot (.), open parenthesis ((), close parenthesis ( )), and space characters in the phone field using the mapping character filter.

  • Tokenize the entire input as a single token using the keyword tokenizer.

The following query searches the page_updated_by.phone field for the string 1234567890.

The MongoDB Search results contain one document where the numbers in the phone string match the query string. MongoDB Search matched the document to the query string even though the query doesn't include the parentheses around the phone area code and the hyphen between the numbers because MongoDB Search removed these characters using the mapping character filter and created a single token for the field value. Specifically, MongoDB Search generated the following token for the phone field in document with _id: 1:

Document ID
Output Tokens

_id: 1

1234567890

MongoDB Search would also match document with _id: 1 for searches for (123)-456-7890, 123-456-7890, 123.456.7890, and so on because for How to Index String Fields fields, MongoDB Search also analyzes search query terms using the index analyzer (or if specified, using the searchAnalyzer). The following table shows the tokens that MongoDB Search creates by removing instances of hyphen (-), dot (.), open parenthesis ((), close parenthesis ( )), and space characters in the query term:

Query Term
Output Tokens

(123)-456-7890

1234567890

123-456-7890

1234567890

123.456.7890

1234567890

persian

The persian character filter replaces instances of zero-width non-joiner with the space character. This character filter is based on Lucene's PersianCharFilter.

Attributes

The persian character filter has the following attribute:

Name
Type
Required?
Description

type

string

yes

Human-readable label that identifies this character filter type. Value must be persian.

Example

The following index definition example indexes the text.fa_IR field in the minutes collection using a custom analyzer named persianCharacterIndex. The custom analyzer specifies the following:

  • Apply the persian character filter to replace non-printing characters in the field value with the space character.

  • Use the whitespace tokenizer to create tokens based on occurrences of whitespace between words.

The following query searches the text.fa_IR field for the term صحبت.

MongoDB Search returns the _id: 2 document that contains the query term. MongoDB Search matches the query term to the document by first replacing instances of zero-width non-joiners with the space character and then creating individual tokens for each word in the field value based on occurrences of whitespace between words. Specifically, MongoDB Search generates the following tokens for document with _id: 2:

Document ID
Output Tokens

_id: 2

ابتدا, رئیس, بخش, فروش, صحبت, کرد

Learn More

To see additional index definitions and queries that use the mapping character filter, see the following reference page examples:

Back

Custom

Next

Tokenizers

On this page