Overview
An alert configuration defines the conditions that trigger an alert and the alert's notification methods. This tutorial describes how to create and manage the alert configurations for a specified project. To create and manage global alert configurations, see Manage Global Alerts.
Considerations
SMS Delivery
Many factors may affect alert delivery, including do not call lists, caps for messages sent or delivered, delivery time of day, and message caching.
Check with your telephone service contract for the costs associated with receiving text messages.
If you choose SMS, Ops Manager sends alert text messages to all users in the project who have a mobile number associated with their user account.
Alert Intervals
You can create multiple alert configurations with different frequencies. The minimum frequency for an alert is 5 minutes.
The time between re-notifications increases by the frequency amount every alert cycle up to a maximum of 24 hours. For example, if the frequency amount is 5 minutes, and the alert condition is first triggered at 9am, subsequent alerts occur at 9:05am, 9:15am, 9:30am, and so on.
You can set the time to elapse before Ops Manager sends an alert after an alert condition occurs. This helps eliminate false positives.
Default Alert Configurations
Ops Manager provides no default alerts for organizations.
Ops Manager creates the following alert configurations for a project automatically upon creation of the project:
User joined the project (
JOINED_GROUP)Host's SSL certificate will expire within 30 days (
HOST_SSL_CERTIFICATE_STALE)Monitoring is down (
MONITORING_AGENT_DOWN)
If you enable Backup, Ops Manager creates the following alert configurations for the project, if they do not already exist:
Backup oplog is behind (
OPLOG_BEHIND)Backup requires a resync (
RESYNC_REQUIRED)Cluster is missing an active mongos (
CLUSTER_MONGOS_IS_MISSING)Host has security recommendations
Query Targeting: Scanned Objects / Returned is
Note
The default configuration sends an alert only if the ratio of documents scanned to documents returned meets or exceeds the specified threshold for at least 10 minutes.
Recommended Alert Configurations
To effectively monitor your deployment, consider configuring the following alerts:
Recommended Host Alerts
Host is recovering: Triggers when a MongoDB instance enters RECOVERING mode.
Replication Lag is: Monitors the lag time of secondaries behind the primary.
Tickets Available: Writes/Reads: Flags potential performance issues for the WiredTiger storage engine.
Replication Oplog: Monitors the oplog window size.
Host is down: Triggers when a host becomes unreachable.
Queues: Readers/Writers is: Tracks operations waiting on locks.
Page faults is: Investigate if consistently high (> 1000 avg/sec).
Host is exposed to the public Internet: Triggers when authentication and firewall protection are missing.
Host does not have the latest version: Triggers when the MongoDB version is outdated by two or more revisions.
Recommended Replica Set Alerts
Replica set elected a new primary: Triggers during an election event.
Number of unhealthy members > n: Triggers when more than n members are unhealthy.
Recommended Sharded Cluster Alerts
- Cluster missing an active mongos
Recommended Agent Alerts
Monitoring Agent is down
Backup Agent is down
Recommended Backup Alerts
When you use the Backup component, configure the following alerts:
Backup oplog is behind
Backup requires a resync
Note
Tune these alerts for your specific workload and usage patterns.
Default Settings for Notification Options
You can configure default settings for the following notification options:
Ops Manager Organization
Ops Manager User
Email
SMS
HipChat
Slack
PagerDuty
Datadog
VictorOps
Opsgenie
You can configure default settings for the following notification options:
Ops Manager Organization
Ops Manager User
Email
SMS
HipChat
Slack
PagerDuty
Datadog
VictorOps
Opsgenie
To configure default settings for one of the above notification options, click Integrations under Projects in the sidebar. Locate the notification option and enter the default values.
Ops Manager fills in the default values automatically when a user selects that option when creating an alert configuration. If the key, token, or URL that is used to send the notification becomes invalid, Ops Manager sends an email to the project owner and eventually removes the key, token, or URL.
Create an Alert Configuration
When you create a new alert, you can clone an existing alert.
Select the Notification Method.
Under the Add Notification Method heading, click the button for the particular notification method you want to use.
Notification Option | User Alerts | Billing Alerts | Description |
|---|---|---|---|
Ops Manager Organization | Sends the alert by email or text message to users with specific roles in the Organization.
| ||
Ops Manager User | Sends the alert to a Ops Manager user, either by email or text message.
| ||
Sends the alert to any email address you provide. | |||
Mobile Number | Sends the alert to a phone number. Ops Manager removes all letters and punctuation except the For international (non-U.S.) phone numbers, format phone numbers using the E.164 standard. For example, for New Zealand, enter Ops Manager uses the U.S.-based Twilio to send text messages. If you want to use a non-U.S. phone number, try using a Google Voice phone number. Configure Ops Manager for Twilio integration to use SMS. | ||
Slack | Sends the alert to a Slack channel in the authorized Slack workplace for the organization.
To learn more about Bot users in Slack, see the Slack documentation. | ||
PagerDuty | Sends the alert to a PagerDuty account. Enter only the PagerDuty integration key. Define escalation rules and alert assignments directly in PagerDuty. Acknowledge PagerDuty alerts from the PagerDuty dashboard. All new PagerDuty keys use their Events API v2. If you have an Events API v1 key, you can continue to use that key with Ops Manager. | ||
Datadog | Sends the alert to a Datadog account as a Datadog event. When the alert first opens, Ops Manager sends the alert as an "error" event. Subsequent updates are sent as "info" events. When the alert closes, Ops Manager sends a "success" event. If prompted, enter your Datadog API key under API Key and click Validate Datadog API Key. Find your Datadog API Key in your Datadog account. | ||
VictorOps | Sends the alert to a Splunk On-Call (formerly VictorOps) account.
This option displays for alerts that require acknowledgement. You can use this notification method on information alerts. These alerts include ones like Acknowledge VictorOps alerts from the VictorOps dashboard. | ||
Opsgenie | Sends the alert to an Opsgenie account.
This option displays for alerts that require acknowledgement. You can use this notification method on information alerts. These alerts include ones like Acknowledge Opsgenie alerts from the Opsgenie dashboard. |
Navigate to the Alerts page for your project.
If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
Click the Project Alerts icon in the navigation bar, or click Alerts in the sidebar.
Choose whether to create a new alert setting or clone an existing one.
To create a new alert without cloning an existing setting:
Click Add.
Select New Alert.
To clone an existing alert setting:
Click the Alert Settings tab.
Locate the alert setting you want to clone.
Click then Clone in that alert setting's row.
Select the condition that triggers the alert.
In the Alert if section, select the target component and the condition. To learn more about alert conditions, see Alert Event Types.
Select the alert recipients and delivery methods.
In the Send to section:
(Optional) To help reduce the false positives that Ops Manager sends, in the send if condition lasts at least field, specify the minutes to elapse before the condition sends the alert. The next alert check that runs after the specified time elapses sends the alert.
(Optional) In the resend after field, specify the minutes to elapse before the condition resends the alert.
Click Add and select from the options described in the following table.
Which alert notification methods can be set depend on the scope of the alert:
- Project alerts
- Apply to one or more individual Organizations and Projects only.
- Global alerts
- Apply to all Organizations and Projects.
- System alerts
- Apply to the health of Ops Manager and its backing databases.
The alert notifications methods are as follows:
Notification MethodProjectGlobalSystemDescriptionOps Manager Project
Sends the alert by email or text message to users with specific roles in the Project.
Select the Project roles that receive the alerts from the Select Role(s) check boxes or select All Roles for all users in the Project to receive the alert.
Select SMS to send these alerts to the mobile number configured for each Ops Manager Project user in their Account page.
Select Email to send these alerts to the email address configured for each Ops Manager Project user in their Account page. Email is checked by default.
Ops Manager Organization
Sends the alert by email or text message to users with specific roles in the Organization.
Select the Organization roles that receive the alerts from the Select Role(s) check boxes or select All Roles for all users in the Organization to receive the alert.
Select SMS to send these alerts to the mobile number configured for each Ops Manager Organization user in their Account page.
Select Email to send these alerts to the email address configured for each Ops Manager Organization user in their Account page. Email is checked by default.
Ops Manager User
Sends the alert to a Ops Manager user, either by email or text message.
Select SMS to send these alerts to the mobile number configured for the Ops Manager user's account.
Select Email to send these alerts to the email address configured for the Ops Manager user's account. Email is checked by default.
Ops Manager Team
Sends the alert by email or text message to a specified Ops Manager team.
This option appears only after at least one team is created.
Select SMS to send these alerts to the mobile number configured for each Ops Manager Team user in their account page.
Select Email to send these alerts to the email address configured for each Ops Manager Team user in their account page. Email is checked by default.
SNMP Host
Specify the hostname that will receive the v2c trap on standard port
162. The MIB file for SNMP is available for download.Email
Sends the alert to a specified email address.
Sends the alert to a HipChat room message stream. Enter the HipChat room name and API token.
Sends the alert to a Slack channel in the authorized Slack workplace for the organization.
Enter the channel name and either an API token or a Bot token.
To create an API token, see the API page in your Slack account.
To learn more about Bot users in Slack, see the Slack documentation.
Sends the alert to a PagerDuty account. Enter only the PagerDuty integration key. Define escalation rules and alert assignments directly in PagerDuty.
Acknowledge PagerDuty alerts from the PagerDuty dashboard.
All new PagerDuty keys use their Events API v2. If you have an Events API v1 key, you can continue to use that key with Ops Manager.
Webhook
Sends an HTTP POST request to an endpoint for programmatic processing. The request body contains a JSON document that uses the same format as the Ops Manager API Alerts resource.
You can authenticate webhook requests with a webhook secret, OAuth 2.0 client credentials, or both.
For detailed information about webhook request headers, request body structure, templates, authentication, and limitations, see Integrate with Webhooks.
To configure this option, configure the Webhook settings on the Project Settings page.
To use this method at the Global level:
Navigate to the Ops Manager Config Miscellaneous tab of the Administration console.
Update the
Webhook URL,Webhook Secret,Webhook OAuth2 Token URL,Webhook OAuth2 Client ID, andWebhook OAuth2 Client Secretsettings.
Ops Manager adds a request header called
X-MMS-Eventto distinguish between various alert states. The possible values for this header are:alert.openThe alert was just opened.
alert.closeThe alert was resolved.
alert.updateA previously opened alert is still open.
alert.acknowledgeThe alert was acknowledged.
alert.cancelThe alert became invalid and was canceled.
alert.informRepresents an informational alert, which is a point-in-time event, such as "Primary Elected."
If you specify a key in the Webhook Secret field, MongoDB Ops Manager adds the
X-MMS-Signaturerequest header. This header contains the base64-encoded HMAC-SHA-1 signature of the request body. MongoDB Ops Manager creates the signature using the provided secret.If you configure OAuth 2.0 client credentials, MongoDB Ops Manager adds the
Authorizationrequest header. This header contains the bearer access token that MongoDB Ops Manager obtains from your OAuth 2.0 token endpoint.Sends the alert to a Datadog account as a Datadog event.
When the alert first opens, Ops Manager sends the alert as an "error" event. Subsequent updates are sent as "info" events. When the alert closes, Ops Manager sends a "success" event.
If prompted, enter your Datadog API key under API Key and click Validate Datadog API Key.
Find your Datadog API Key in your Datadog account.
Administrators
Sends the alert to the email address specified in the Admin Email Address field in the Ops Manager configuration options.
Global Alerts Summary Email
Sends a summary email of all global alerts to the specified email address.
Delete an Alert Configuration
Choose the alert setting to delete.
On the row for a specific alert, click then Delete.
Click Delete in the Delete Alert modal or click Cancel to leave the alert as configured.
Note
If you have not logged in within the five minutes of pushing Delete, you need to re-enter your password, then click Delete again.
Navigate to the Alert Settings tab.
If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
Click the Project Alerts icon in the navigation bar, or click Alerts in the sidebar.
- Click the Alert Settings tab.
If you delete an alert setting, Ops Manager cancels active alerts related to the setting. A deleted alert setting does not remain visible.
Disable or Enable an Alert Configuration
Navigate to the Alert Settings tab.
If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
Click the Project Alerts icon in the navigation bar, or click Alerts in the sidebar.
- Click the Alert Settings tab.
If you disable an alert setting, Ops Manager cancels active alerts related to the setting. A disabled alert setting remains visible but unavailable and can be later re-enabled.
If you need to disable an alert only for a period of time, you can suspend alerts.
View the History of Changes to an Alert Configuration
You can view all alerts, alert settings, and deleted alerts on the Organization Alerts page. To learn more, see Alerts Workflow.
You can view open alerts, closed alerts, and alert settings on the Project Alerts page. Ops Manager sends notifications for all alerts that appear on the Open tab. To learn more, see Alerts Workflow.
Navigate to the Alert Settings tab.
If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
Click the Project Alerts icon in the navigation bar, or click Alerts in the sidebar.
- Click the Alert Settings tab.
Test Alert Configurations
You can test your alert configurations to ensure they work correctly before deploying them in production. This applies especially to webhook notifications with custom templates. Not all notification methods provide a way to test the configuration.
For webhook notifications, the test includes:
Sample alert data rendered through your custom templates
Validation that your webhook endpoint is reachable
Verification that custom headers and body templates render correctly
Global alert tests use sample data that represents the type of alerts that would be generated across multiple projects.