/ /

Administration

Manage Configuration

Docs Home

Development

Data Federation

Administration

Manage Configuration

`createStore`

The createStore command creates a store in the federated database instance storage configuration. Click on the tab below to learn more about creating a federated database instance store for the data source.

This tab contains the syntax and parameters for creating a federated database instance store for data in AWS S3 bucket.

Important

Atlas Data Federation doesn't support federated queries across different cloud providers. You can only create federated database instances that map to stores on a single cloud provider.

Syntax

db.runCommand({ createStore: <store-name>, provider: <storage-provider>, region: <region-name>, bucket: <bucket-name>, additionalStorageClasses: [ <storage-classes> ], delimiter: <delimiter>, public: true|false })

db.runCommand({ createStore: <store-name>, provider: <storage-provider>, region: <region-name>, serviceURL: <service-account-URL>, containerName: <storage-container-name>, delimiter: <delimiter>, public: true|false })

db.runCommand({ createStore: <store-name>, provider: <storage-provider>, region: <region-name>, bucket: <bucket-name>, delimiter: <delimiter>, prefix: <prefix> })

db.runCommand({ createStore: <store-name>, provider: <storage-provider>, clusterName: <cluster-name>, projectId: <project-id>, readPreference: {mode: <read-preference-mode>, tagSets: [[{name: <name>, value: <value>},...],...], maxStalenessSeconds: <number-of-seconds>}, readConcern: {level: <level>} })

db.runCommand({ createStore: <store-name>, provider: <storage-provider>, allowInsecure: true|false, urls: [ <url> ], defaultFormat: <file-extension> })

db.runCommand({ createStore: <store-name>, provider: "dls:aws", region: <online-archive-region> })

Parameters

Parameter	Type	Description	Required?
`createStore`	string	Name of the new federated database instance store. The federated database instance store name must be unique.	yes
`provider`	string	Name of the service where the data is stored. Value can be one of the following: `s3` for an AWS S3 bucket. `azure` for Azure Blob Storage container. `atlas` for Atlas cluster. `http` for files hosted at publicly accessible URLs. `dls:<subtype>` for online archives. Atlas Data Federation supports the following subtypes: `aws` - for data stored on AWS. Value must be `dls:aws`. `azure` - for data stored on Azure. Value must be `dls:azure`.	yes

`region`	string	Region in which the `bucket` is hosted. For a list of valid region names, see Amazon Web Services (AWS).	yes
`bucket`	string	Name of the bucket in which data is stored. Must exactly match the name of an S3 bucket that Data Federation can access using AWS IAM credentials.	yes
`additionalStorageClasses`	array of strings	Array of AWS S3 storage classes. Atlas Data Federation will include the files in these storage classes in the query results. Valid values are: `INTELLIGENT_TIERING` to include files in the Intelligent Tiering storage class. `STANDARD_IA` to include files in the Standard-Infrequent Access storage class. Files in the Standard storage class are supported by default.	no
`delimiter`	string	Character used to separate path segments in the federated database instance store. If ommitted, defaults to `"/"`.	no
`public`	boolean	Flag that indicates whether or not the bucket is public. Valid values are: `true` to not use the AWS IAM role to access the bucket `false` to require the AWS IAM role to access the bucket If omitted, defaults to `false`.	no

Parameter	Type	Description	Required?
`createStore`	string	Name of the new federated database instance store. The federated database instance store name must be unique.	yes
`provider`	string	Name of the service where the data is stored. Value can be one of the following: `s3` for an AWS S3 bucket. `azure` for Azure Blob Storage container. `atlas` for Atlas cluster. `http` for files hosted at publicly accessible URLs. `dls:<subtype>` for online archives. Atlas Data Federation supports the following subtypes: `aws` - for data stored on AWS. Value must be `dls:aws`. `azure` - for data stored on Azure. Value must be `dls:azure`.	yes

region

string

Region in which the container is hosted. For a list of valid region names, see Microsoft Azure.

yes

bucket

string

URL of the Azure Blob Storage account that contains your blob containers. For example:

https://<azure-storage-account-name>.blob.core.windows.net/

yes

containerName

string

Name of the Azure Blob Storage container that contains the files. For more information, see Azure documentation for Naming and Referencing Containers.

yes

delimiter

string

Character used to separate path segments in the federated database instance store. If omitted, defaults to "/".

public

boolean

Flag that indicates whether or not the Azure Blob Storage container is public. Valid values are:

true to not use the Azure Service Principal to access the bucket
false to require the Azure Service Principal to access the bucket

If omitted, defaults to false.

Parameter	Type	Description	Required?
`createStore`	string	Name of the new federated database instance store. The federated database instance store name must be unique.	yes
`provider`	string	Name of the service where the data is stored. Value can be one of the following: `s3` for an AWS S3 bucket. `azure` for Azure Blob Storage container. `atlas` for Atlas cluster. `http` for files hosted at publicly accessible URLs. `dls:<subtype>` for online archives. Atlas Data Federation supports the following subtypes: `aws` - for data stored on AWS. Value must be `dls:aws`. `azure` - for data stored on Azure. Value must be `dls:azure`.	yes

region

string

Region in which the bucket is hosted. For a list of valid region names, see Google Cloud Platform (GCP).

yes

bucket

string

Name of the bucket in which data is stored. Must exactly match the name of an Google Cloud Storage bucket that Data Federation can access using the configured Google Cloud IAM credentials.

yes

delimiter

string

Character used to separate path segments in the federated database instance store. If ommitted, defaults to "/".

prefix

boolean

Prefix Atlas Data Federation applies when searching for files in the Google Cloud Storage bucket. For example, consider a Google Cloud Storage bucket metrics with the following structure:

metrics
  |--hardware
  |--software
    |--computed

The federated database instance store prepends the value of prefix to databases.[n].collections.[n].dataSources.[n].path to create the full path for files to ingest. Setting prefix to /software restricts any databases objects using the federated database instance to only subpaths of /software.

Defaults to the root of the Google Cloud Storage bucket, retrieving all files.

Parameter	Type	Description	Required?
`createStore`	string	Name of the new federated database instance store. The federated database instance store name must be unique.	yes
`provider`	string	Name of the service where the data is stored. Value can be one of the following: `s3` for an AWS S3 bucket. `azure` for Azure Blob Storage container. `atlas` for Atlas cluster. `http` for files hosted at publicly accessible URLs. `dls:<subtype>` for online archives. Atlas Data Federation supports the following subtypes: `aws` - for data stored on AWS. Value must be `dls:aws`. `azure` - for data stored on Azure. Value must be `dls:azure`.	yes

`clusterName`	string	Name of the Atlas cluster on which the store is based.	yes
`projectId`	string	Unique identifier of the project that contains the Atlas cluster. Atlas Data Federation only supports the `projectId` of the project where the federated database instance exists. If not specified, Atlas Data Federation assumes that the cluster is located in the same project as the federated database instance.	no
`readPreference`	document	Cluster read preference, which describes how to route read requests to the cluster.	no
`readPreference` `.mode`	string	Read preference mode that specifies which replica set member to route the read requests to. Value can be one of the following: `primary` - to route all read requests to the replica set primary `primaryPreferred` - to route all read requests the replica set primary and to secondary members only if `primary` is unavailable `secondary` - to route all read requests to the secondary members of the replica set `secondaryPreferred` - to route all read requests to the secondary members of the replica set and the primary on sharded clusters only if `secondary` members are unavailable `nearest` - to route all read requests to random eligible replica set member, irrespective of whether that member is a primary or secondary	no
`readPreference` `.tagSets`	array of strings	Arrays of tag sets or tag specification documents that contain name and value pairs for the replica set member. If specified, Atlas Data Federation routes read requests to replica set member or members that are associated with the specified tags. To learn more, see Read Preference Tag Sets. Atlas Data Federation doesn't support this parameter for sharded clusters.	no
`readPreference` `.maxStalenessSeconds`	int	Maximum replication lag, or "staleness", for reads from secondaries. To learn more about `maxStalenessSeconds`, see Read Preference maxStalenessSeconds.	no

Parameter	Type	Description	Required?
`createStore`	string	Name of the new federated database instance store. The federated database instance store name must be unique.	yes
`provider`	string	Name of the service where the data is stored. Value can be one of the following: `s3` for an AWS S3 bucket. `azure` for Azure Blob Storage container. `atlas` for Atlas cluster. `http` for files hosted at publicly accessible URLs. `dls:<subtype>` for online archives. Atlas Data Federation supports the following subtypes: `aws` - for data stored on AWS. Value must be `dls:aws`. `azure` - for data stored on Azure. Value must be `dls:azure`.	yes

`allowInsecure`	boolean	Validates the scheme in the specified URLs. Value can be one of the following: `true` to allow insecure HTTP scheme `false` to only allow secure HTTPS scheme (default) If true, Atlas Data Federation: Does not verify the server's certificate chain and hostname. Accepts any certificate with any hostname presented by the server. WARNING: If you set this to `true`, your data might become vulnerable to a man-in-the-middle attack, which can compromise the confidentiality and integrity of your data. Set this to `true` only for testing and getting started with Atlas Data Federation. If omitted, defaults to `false`.	no
`urls`	array of strings or an empty array	One or more publicly accessible URLs. You can't specify URLs that require authentication.	no
`defaultFormat`	string	Format that Data Federation defaults to if it encounters a file without an extension while querying the federated database instance store. The following values are valid: `.json`, `.json.gz`, `.bson`, `.bson.gz`, `.avro`, `.avro.gz`, `.orc`, `.tsv`, `.tsv.gz`, `.csv`, `.csv.gz`, `.parquet` If omitted, Data Federation attempts to detect the file type by processing a few bytes of the file. If included, the specified format only applies to the URLs in the store.	no

Parameter	Type	Description	Required?
`createStore`	string	Name of the new federated database instance store. The federated database instance store name must be unique.	yes
`provider`	string	Name of the service where the data is stored. Value can be one of the following: `s3` for an AWS S3 bucket. `azure` for Azure Blob Storage container. `atlas` for Atlas cluster. `http` for files hosted at publicly accessible URLs. `dls:<subtype>` for online archives. Atlas Data Federation supports the following subtypes: `aws` - for data stored on AWS. Value must be `dls:aws`. `azure` - for data stored on Azure. Value must be `dls:azure`.	yes

region

string

Region of your online archive. If you have multiple archives in different regions, you must add a store for each region to map data in that region to virtual databases and collections in federated database instances.

yes

Output

The command prints the following output if it succeeds. If the command fails, see Troubleshoot Errors for recommended solutions.

{
  "ok": 1,
  "store": {
    "name": "<store-name>",
    "region": "<region-name>",
    "bucket": "<bucket-name>",
    "additionalStorageClasses": ["<storage-classes>"]
    "delimiter": "<delimiter>",
    "provider": "<storage-provider>"
  }
}

{
  "ok": 1,
  "store": {
    "name": "<store-name>",
    "region": "<region-name>",
    "serviceURL": "<service-account-URL>",
    "containerName": "<storage-container-name>"
    "delimiter": "<delimiter>",
    "provider": "<storage-provider>"
  }
}

{
  "ok": 1,
  "store": {
    "name": "<store-name>",
    "region": "<region-name>",
    "bucket": "<storage-container-name>"
    "delimiter": "<delimiter>",
    "prefix": "<prefix>",
    "provider": "<storage-provider>"
  }
}

{
        "ok" : 1,
        "store" : {
                "name" : "<store-name>",
                "provider" : "<storage-provider>",
                "clusterName" : "<cluster-name>",
                "projectId" : "<project-id>",
                "readPreference" : {
                       "mode" : "<read-preference-mode>",
                       "tagSets" : [[{"name": "<name>", "value": "<value>"},...],...],
                       "maxStalenessSeconds" : <number-of-seconds>
                }
        }
}

{
        "ok" : 1,
        "store" : {
                "name" : "<store-name>",
                "provider" : "<storage-provider>",
                "allowInsecure" : true|false,
                "urls" : [
                        "<url>"
                ],
                "defaultFormat: "<file-extension>"
        }
}

{
  ok: 1,
  store: {
    name: '<store-name>',
    provider: 'dls:aws',
    region: '<online-archive-region>'
  }
}

Example

The following example uses the createStore command to create a new federated database instance store called myStore.

use sample
db.runCommand({ createStore: "myStore", provider: "s3", region: "us-east-1", bucket: "my-bucket", "additionalStorageClasses" : ["STANDARD_IA","INTELLIGENT_TIERING"] })

{
  "ok": 1,
  "store": {
    "name": "myStore",
    "region": "us-east-1",
    "bucket": "my-bucket",
    "additionalStorageClasses" : [
                        "STANDARD_IA",
                        "INTELLIGENT_TIERING"
                ],
    "delimiter": "/",
    "provider": "s3"
  }
}

use sample
db.runCommand({ createStore: "myStore", provider: "azure", region: "eastus2", serviceURL: "https://mystorage.blob.core.windows.net/", "containerName" : "my-data-container" })

{
  "ok": 1,
  "store": {
    "name": "myStore",
    "region": "eastus2",
    "serviceURL": "https://mystorage.blob.core.windows.net/",
    "containerName" : "my-data-container",
    "delimiter": "/",
    "provider": "azure"
  }
}

use sample
db.runCommand({ createStore: "myStore", provider: "azure", region: "us-central1", "bucket" : "my-data-container" })

{
  "ok": 1,
  "store": {
    "name": "myStore",
    "region": "us-central1",
    "bucket" : "my-data-container",
    "delimiter": "/",
    "provider": "gcp"
  }
}

use sample
db.runCommand({ createStore: "myStore", provider: "atlas", clusterName: "myTestCluster", projectId: "<project-id>", "readPreference": {"mode": "secondary", "tagSets": [[{"name": "provider", "value": "AWS" }, {"name": "region", "value": "US_EAST_1"}]], "maxStalenessSeconds": 120} })

{
        "ok" : 1,
        "store" : {
                "name" : "myStore",
                "provider" : "atlas",
                "clusterName" : "myTestCluster",
                "projectId" : "<project-id>",
                "readPreference" : {
                        "mode" : "secondary",
                        "tagSets": [[{"name": "provider", "value": "AWS" }, {"name": "region", "value": "US_EAST_1"}]],
                        "maxStalenessSeconds" : 120
                }
        }
}

use sample
db.runCommand({ createStore: "myStore", provider: "http", urls: ["https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews","http://example.mongodb.com/sampleData"], allowInsecure: true, defaultFormat: ".json" })

{
        "ok" : 1,
        "store" : {
                "name" : "http-store",
                "provider" : "http",
                "urls" : [
                        "https://atlas-data-lake.s3.amazonaws.com/json/sample_airbnb/listingsAndReviews",
                        "http://example.mongodb.com/sampleData"
                ],
                "defaultFormat" : ".json"
                "allowInsecure" : true
        }
}

use sample
db.runCommand({ createStore: "myStore", provider: "dls:aws", region: "us-east-1" })

{
  ok: 1,
  store: {
    name: 'myStore',
    provider: 'dls:aws',
    region: 'us-east-1'
  }
}

Troubleshoot Errors

If the command fails, it returns one of the following errors.

Reason: a federated database instance store with the name specified in createStore already exists.

{
  "ok": 0,
  "errmsg": "store <store-name> already exists",
  "code": 2,
  "codeName": "BadValue"
}

Solution: Specify a unique name for the federated database instance store.

Reason: The specified provider isn't recognized.

{
  "ok": 0,
  "errmsg": "unrecognized store provider <storage-provider>",
  "code": 2,
  "codeName": "BadValue"
}

Solution: Ensure that you specify a valid storage provider.

Reason: One or more given URLs contain insecure HTTP scheme.

{
        "ok" : 0,
        "errmsg" : "store 'httpStore': the insecure HTTP scheme is not supported by default - please add a 'allowInsecure: true' flag to the store or datasource to query from such URLs, correlationID = 16332c6eddf7b32776fde638",
        "code" : 72,
        "codeName" : "InvalidOptions"
}

Solution: Specify URLs with the secure HTTPS scheme. If the specified URLs have the insecure HTTP scheme, set the allowInsecure flag to true. Note that setting the allowInsecure flag to true leaves your data vulnerable to man-in-the-middle attacks.

Back

Manage Configuration

List Stores