Limitations

Atlas Stream Processing supports only at-least-once processing.

Atlas Stream Processing doesn't support horizontal scaling.

Atlas Stream Processing uses a single core for transformation pipeline stages, except for stages that can specify a parallelism value.

The state.stateSize of a stream processor can't exceed 80% of the RAM available for its pod. For example, the maximum size of a stream processor in the SP30 tier, which has 8GB of RAM, is 6.4GB. If the state.stateSize of any of your stream processors is approaching 80% of its available RAM, consider stopping the processor and restarting it on a higher tier. If your stream processor already runs at the maximum tier enabled for your stream processing workspace, consider adjusting your stream processing workspace configuration to enable higher-tier stream processors.

When a stream processor crosses the 80% RAM threshold, it fails with a Worker out of memory error. You can view the state.stateSize value of each stream processor with the sp.processor.stats() command. See View Statistics of a Stream Processor to learn more.

An Atlas Stream Processing pipeline definition cannot exceed 16 MB.

Only users with the Organization Stream Processing Admin, Project Owner, Project Stream Processing Owner or Atlas admin roles can use Atlas Stream Processing.

You must use mongosh version 2.3.4 or later to override an option on an existing stream processor using a mongosh method. For example, using sp.processor.start() to specify a tier for the processor you are starting.

See Develop Stream Processors to learn more about managing a stream processor using mongosh.

Atlas Stream Processing supports a subset of the Aggregation Pipeline Stages available in Atlas, allowing you to perform many of the same operations on streaming data that you can perform on data-at-rest. For a full list of supported Aggregation Pipeline Stages, see the Stream Aggregation documentation.

Atlas Stream Processing doesn't support the aggregation variables $$NOW, $$CLUSTER_TIME, $$USER_ROLES, and $SEARCH_META.

Atlas Stream Processing doesn't support writing BSON documents larger than 125 MB using the $emit stage to an AWS S3 bucket.

For Apache Kafka $source stages, if the Apache Kafka topic acting as $source to the running processor adds a partition, the starting offset is determined by the auto_offset_reset property.

For Apache Kafka $source stages are subject to stream processing workspace tier partition limits. If you exceed the partition limit of your stream processing workspace tier, the affected stream processor will fail. You must upgrade your stream processing workspace to support additional partitions.

Atlas Stream Processing doesn't support connecting to an Apache Kafka consumer in the same consumer group as other consumers. Ensure any Kafka consumer used with Atlas Stream Processing runs in a dedicated consumer group.

For Apache Kafka connections, Atlas Stream Processing currently supports only the following security protocols:

• SASL_PLAINTEXT

• SASL_SSL

• SSL

For SASL, Atlas Stream Processing supports the following mechanisms:

• PLAIN

• SCRAM-SHA-256

• SCRAM-SHA-512

• OAUTHBEARER

For SSL, you must provide the following assets for your Apache Kafka system mutual TLS authentication with Atlas Stream Processing:

• a Certificate Authority (if you are using one other than the default Apache Kafka CA)

• a client TLS certificate

• a TLS keyfile, used to sign your TLS certificate

initialSync supports collections whose _id values are of any type except array, regular expression, or JavaScript code, which MongoDB doesn't support as _id values. All _id values in a collection must be of the same type. Atlas Stream Processing doesn't support initialSync on collections that contain mixed _id types.

If your collection's _id values are default generated ObjectId values or ordered int or long values, initialSync achieves optimal performance. For other _id types, initialSync might take longer to complete because the values aren't stored in a predictable order. Increase your oplog size before you begin a longer initialSync operation so that the processor's resume token remains on the oplog. If the resume token is no longer on the oplog when initialSync completes, the processor enters a failed state to prevent data loss, and initialSync runs again.

Atlas Stream Processing can insert new documents into a partition while initialSync copies it. A network failure that interrupts a query on a growing partition can prolong the collection-copy phase.

initialSync might insert duplicate documents if it reads a change event during the collection-copy or catch-up phase. Atlas Stream Processing's at-least-once processing guarantee covers this behavior.

Atlas Stream Processing supports failover processors only for processors with the following configurations:

• Atlas source and sink

• Atlas source and Apache Kafka sink

Atlas Stream Processing supports failover processors only for processors of tier SP10 or higher.

For processors configured with failover processors, only one processor can be active at any given time. You can edit only the active processor.