Azure CosmosDB Key Rotation: Monitoring and Alerts

Problem Statement

Azure Cosmos DB account keys are used by applications for authentication to perform read/write operations.

In the event that a Cosmos account key (either primary or secondary) is rotated without application team’s knowledge, it would result in authentication failures leading to outages.

The impact of such incidents can be mitigated by configuring an alerting mechanism that notifies the stakeholders via email every time a key is rotated.

Furthermore, the events could be persisted in storage accounts to track and maintain history of key rotation. This would enable teams to proactively and routinely rotate keys of Cosmos accounts and improve the security posture.

Solution

Using CosmosDB Alerts available to trigger a notification only when a key (either primary or secondary) is refreshed. This notification can be sent out as email, SMS etc and the alert can be pushed to Logic Apps, Azure Functions or Event Hubs for downstream processing, persistence, and analytics.

For the purpose of this article, we’re pushing the alerts to Azure Event Hubs and persisting in Storage Account containers.

Pre-requisites

1. Azure Storage Account
2. Event Hubs namespace and instance

Note:

• Minimum Storage tier should be Standard (Basic tier doesn’t support the Capture feature on Event Hubs)
• For Event Hubs, Capture is enabled to push the alerts/messages to persistent storage. Options available:
  - Azure Storage Account
  - Azure Data Lake Service account
• Do not emit empty files when no events occur is enabled for brevity.

Event Hubs Setup

Steps

1. Under the identified CosmosDB account, navigate to Alerts:

CosmosDB Alerts

2. Create an action group

Action Groups on Alerts

and define the actions that will be invoked when this action group is linked with an alert.

In the screenshot below, we have set up notification by email and writing the alert itself to Event Hubs instance created in the previous step

Action Group Setup — 1

Action Group Setup— 2

3. Create a new Alert Rule

Define the scope

• Resource type as CosmosDB accounts
• For this article, we’ve chosen the scope as the subscription. This encapsulates ALL CosmosDB accounts part of the subscription.
• By checking “Include all future resources”, any new CosmosDB accounts created in the future will also be included in the scope.

Alert Rule Setup

NOTE:

• If the scope is defined at the subscription or resource group level, the only available signal type is Activity Log (Bottom right corner in the above screenshot).
• If the scope is defined at a SINGLE CosmosDB account, Metrics and Log are also available as signal type in addition to Activity Log. The “Account Keys Updated” signal is available under metric signal type.

4. Under Condition, choose the signal name as “Rotate Keys”

Specify

• Event Level as “All Selected”
• Status as “All selected”
• Event initiated by as “All services and users”

5. Under Actions, add the Action Group created in Step 2 above:

6. Under Details, specify the rule name and description. This will appear in the alerts whenever they are generated.

7. [Optional but recommended]: Define the tags

8. Review and create the rule

…and we’re done! Now let’s see if this works!

Test the solution

1. Rotated the primary key of the Cosmos Account A

2. Alert triggered and published to Event Hub:

3. Alert raw message is persisted in AVRO in Blob Storage, containing the ID of the event initiator and impacted CosmosDB account.

Sample raw message file:

4. Email alert received with the details of the event initiator and impacted CosmosDB account:

Hope this was useful. Feel free to get in touch!

References:

Create alerts for Azure Cosmos DB using Azure Monitor | Microsoft Docs

Monitor your Azure Cosmos DB account for key updates and key regeneration | Microsoft Docs

Pricing — Event Hubs | Microsoft Azure

Azure Storage Blobs Pricing | Microsoft Azure

Pricing — Azure Monitor | Microsoft Azure