Export Cloud Backup Snapshots to Object Storage

Note

This page applies to both Atlas Infinite and Atlas Core.

Note

This feature is available for M10 clusters or higher. To learn more about which features are available for Free clusters, see Atlas Free Cluster Limits, and for Flex clusters, see Atlas Flex Limitations.

Atlas lets you export your Cloud Backup snapshots to an object storage service.

To learn how to restore an exported snapshot back into an Atlas cluster, see Import Archived Snapshots into Atlas.

To learn how to manage automated backup policies and schedules, see Manage Backup Policies.

Supported Storage Services

Atlas currently supports the following object storage services:

AWS S3 buckets
Azure Blob Storage
Google Cloud Storage buckets

Additionally, Atlas supports exporting snapshots to AWS S3 buckets using PrivateLink.

How Atlas Exports Snapshots

You can manually export individual snapshots or set up an export policy for automatic export of your snapshots. For steps specific to your object storage service, see your cloud provider's export page.

To set up automatic exports, add an export policy to your Cloud Backup schedule and set its frequency type to daily, weekly, monthly, or yearly. When a backup policy item with the same frequency type creates a snapshot, Atlas exports that snapshot to your specified object store.

Atlas uploads an empty file to the /exported_snapshots/.permissioncheck path on your object store when you add a new object store for export and when you start an export. As the export progresses, you may see partial results in your object storage service.

Atlas persists documents in snapshots irrespective of Time to Live settings. You can access these documents from your snapshot past their Time to Live deadline.

To learn about snapshot export pricing, see Snapshot Export Costs.

Exported Data Files

Each Cloud Backup snapshot export job uploads the snapshot data to the object store in gzip-compressed Extended JSON v2 documents. These files are not uploaded in any particular order. Each file contains a chunk of documents from a single collection in the snapshot. For sharded clusters Atlas creates a separate export job per replica set (shard), and specifies the replica set name in the path of the uploaded files.

Important

As the export progresses, you may see partial results in your object storage service. If an export job fails:

Atlas doesn't automatically try to export again.
Atlas doesn't remove any partial data in your object store.

Atlas uploads each data file to the following path on your object store:

/exported_snapshots/<orgName>/<projectName>/<clusterName>/<initiationDateOfSnapshot>/<timestamp>/<dbName>/<collectionName>/<shardName>.<increment>.json.gz

Atlas replaces each placeholder in the path with these values:

`<orgName>`	Name of your Atlas organization.
`<projectName>`	Name of your Atlas project.
`<clusterName>`	Name of your Atlas cluster.
`<initiationDateOfSnapshot>`	Date when snapshot was taken.
`<timestamp>`	Timestamp when the export job was created.
`<dbName>`	Name of the database in the Atlas cluster.
`<collectionName>`	Name of the Atlas collection.
`<shardName>`	Name of the replica set. For sharded collections, this is the name of the primary shard.
`<increment>`	Count that is incremented as chunks are uploaded. Starts at `0`.

Exported Metadata Files

After Atlas finishes exporting the snapshot data to your object store, Atlas also uploads a metadata file named .complete and a metadata file named metadata.json for each collection:

For each collection in the snapshot export job, Atlas uploads a metadata file named .complete to the following path on your object store:

/exported_snapshots/<orgUUID>/<projectUUID>/<clusterName>/<initiationDateOfSnapshot>/<timestamp>/

Note

By default, Atlas uses organization and project UUIDs in the path for the metadata files. To use organization and project names instead of UUIDs, set the useOrgAndGroupNamesInExportPrefix flag to true via the API. Atlas replaces any spaces with underscores (_) and removes any characters that might require special handling and characters to avoid from the organization and project names in the path.

The .complete metadata file is in JSON format and contains the following fields:

Request Body Field	Description
`orgId`	Unique 24-hexadecimal digit string that identifies the Atlas organization.
`orgName`	Name of the Atlas organization.
`groupId`	Unique 24-hexadecimal digit string that identifies the project in the Atlas organization.
`groupName`	Name of the Atlas project.
`clusterUniqueId`	Unique 24-hexadecimal digit string that identifies the Atlas cluster.
`clusterName`	Name of the Atlas project.
`snapshotInitiationDate`	Date when snapshot was taken.
`totalFiles`	Total number of files uploaded to the object store.
`labels`	Labels of the cluster whose snapshot was exported.
`customData`	Custom data, if any, that you specified when creating the export job.

Example

{
  "orgId": "60512d6f65e4047fe0842095",
  "orgName": "org1",
  "groupId": "60512dac65e4047fe084220f",
  "groupName": "group1",
  "clusterUniqueId": "60512dac65e4047fe0842212",
  "clusterName": "cluster0",
  "snapshotInitiationDate": "2020-04-03T05:50:29.321Z"
  "totalFiles": 23,
  "labels": [
    {
      "key": "key1",
      "value": "xyz"
    },
    {
      "key": "key2",
      "value": "xyzuio"
    }
  ],
  "customData": [
    {
      "key": "key1",
      "value": "xyz"
    },
    {
      "key": "key2",
      "value": "xyzuio"
    }
  ]
}

For each collection in the snapshot export job, Atlas uploads a metadata file named metadata.json to the following path on your object store:

/exported_snapshots/<orgUUID>/<projectUUID>/<clusterName>/<initiationDateOfSnapshot>/<timestamp>/<dbName>/<collectionName>/metadata.json

Note

By default, Atlas uses organization and project UUIDs in the path for the metadata files. To use organization and project names instead of UUIDs, set the useOrgAndGroupNamesInExportPrefix flag via the API to true. Atlas replaces any spaces with underscores (_) and removes any characters that might require special handling and characters to avoid from the organization and project names in the path.

The metadata file is in JSON format and contains the following fields:

Request Body Field	Description
`collectionName`	Human-readable label that identifies the collection.
`indexes`	List of all the indexes on the collection in the format returned by db.collection.getIndexes command.
`options`	Configuration options defined on the collection. To learn more about the options, see db.createCollection() command.
`type`	(Optional) Type of collection. This field is only supported for time series collections, with a value of `timeseries`. Leave this field unset for standard collections. Atlas doesn't support export of `view` type collections.
`uuid`	Collection's UUID. To learn more about UUID, see UUID.

Example

{
  "options":{
    "viewOn":"othercol",
    "pipeline":[{"$project":{"namez":"$name"}}]
  },
  "indexes":[],
  "collectionName":"viewcol",
  "type":"view"
}

{
  "options":{
    "timeseries":{
      "timeField":"timestamp",
      "granularity":"seconds",
      "bucketMaxSpanSeconds":{"$numberInt":"3600"}
    }
  },
  "indexes":[],
  "collectionName":"timeseriescol",
  "type":"timeseries"
}

{
  "indexes": [
    {
      "v":{"$numberInt":"2"},
      "key":{
        "_id":{"$numberInt":"1"}
      },
      "name":"_id_"
    }
  ],
  "uuid":"342c40a937c34c478bab03de8ce44f3e",
  "collectionName":"somecol"
}

Limitations

You can't perform the following actions:

Export fallback snapshots.
Have more than one active export per snapshot.
Export view collections, or system collections, except for <database>.system.js collections.
Export snapshots from clusters in an Atlas project that has Encryption at Rest using Customer Key Management enabled with network access restricted to specific IP addresses. This restriction doesn't apply if Atlas reaches your KMS through a private endpoint. To learn how to configure a private endpoint to your KMS, see the AWS instructions or the Azure instructions.
Export snapshots over Private Endpoints (private exports) for Azure or Google Cloud clusters.
Export snapshots taken from MongoDB versions that have reached end of life (EOL). See the MongoDB Software Lifecycle Schedules for version EOL dates.
Export snapshots to an Azure storage account if its firewall rules block public access.

Required Access

To create or manage a Cloud Backup export schedule or configure a Cloud Backup export bucket in Atlas, you must have Project Backup Manager or Project Owner to the project.
To create or manage a Cloud Backup export job, you must have Project Backup Export Operator or Project Owner access to the project.
To create or manage a Cloud Backup object storage private endpoint, you must have Project Owner access to the project.

Export to Your Cloud Provider

The steps to export Cloud Backup snapshots depend on your object storage service. To set up Atlas access, configure an export bucket, and export snapshots, see the page for your cloud provider:

Export snapshots to AWS S3 buckets, including over PrivateLink.

AWS S3

Export snapshots to Azure Blob Storage containers.

Azure Blob Storage

Export snapshots to Google Cloud Storage buckets.

Google Cloud Storage