Edit

Share via

Azure Kubernetes Service Communication Manager

The Azure Kubernetes Service (AKS) Communication Manager streamlines notifications for all your AKS maintenance tasks by using Azure Resource Notifications and Azure Resource Graph frameworks. This tool enables you to closely monitor your upgrades because it provides you with timely alerts on event triggers and outcomes. If maintenance fails, it notifies you with the reasons for the failure, reducing operational hassles related to observability and follow-ups. You can set up notifications for all types of autoupgrades that utilize maintenance windows by following these steps.

Prerequisites

Note

After Communication Manager is set up, it sends advance notices one week before maintenance starts and one day before maintenance starts. It also sends you timely alerts during the maintenance operation.

Set up Communication Manager

  1. Go to the resource, select Monitoring, select Alerts, and then select Alert Rules.

  2. On the Condition tab, for Signal name, select Custom log search.

    Screenshot that shows the custom log search in the alert rule pane.

  3. In the Search query box, paste one of the following custom queries and then select the Review+Create button.

    The following query is for cluster autoupgrade notifications:

    | where type == "microsoft.containerservice/managedclusters/scheduledevents"
| where id contains "/subscriptions/subid/resourcegroups/rgname/providers/Microsoft.ContainerService/managedClusters/clustername"
| where properties has "eventStatus"
| extend status = substring(properties, indexof(properties, "eventStatus") + strlen("eventStatus") + 3, 50)
| extend status = substring(status, 0, indexof(status, ",") - 1)
| where status != ""
| where properties has "eventDetails"
| extend upgradeType = case(
                           properties has "K8sVersionUpgrade",
                           "K8sVersionUpgrade",
                           properties has "NodeOSUpgrade",
                           "NodeOSUpgrade",
                           status == "Completed" or status == "Failed",
                           case(
    properties has '"type":1',
    "K8sVersionUpgrade",
    properties has '"type":2',
    "NodeOSUpgrade",
    ""
),
                           ""
                       )
| where properties has "lastUpdateTime"
| extend eventTime = substring(properties, indexof(properties, "lastUpdateTime") + strlen("lastUpdateTime") + 3, 50)
| extend eventTime = substring(eventTime, 0, indexof(eventTime, ",") - 1)
| extend eventTime = todatetime(tostring(eventTime))
| where eventTime >= ago(2h)
| where upgradeType == "K8sVersionUpgrade"
| project
    eventTime,
    upgradeType,
    status,
    properties
| order by eventTime asc

    The following query is for Node OS autoupgrade notifications:

    | where type == "microsoft.containerservice/managedclusters/scheduledevents"
| where id contains "/subscriptions/subid/resourcegroups/rgname/providers/Microsoft.ContainerService/managedClusters/clustername"
| where properties has "eventStatus"
| extend status = substring(properties, indexof(properties, "eventStatus") + strlen("eventStatus") + 3, 50)
| extend status = substring(status, 0, indexof(status, ",") - 1)
| where status != ""
| where properties has "eventDetails"
| extend upgradeType = case(
                           properties has "K8sVersionUpgrade",
                           "K8sVersionUpgrade",
                           properties has "NodeOSUpgrade",
                           "NodeOSUpgrade",
                           status == "Completed" or status == "Failed",
                           case(
    properties has '"type":1',
    "K8sVersionUpgrade",
    properties has '"type":2',
    "NodeOSUpgrade",
    ""
),
                           ""
                       )
| where properties has "lastUpdateTime"
| extend eventTime = substring(properties, indexof(properties, "lastUpdateTime") + strlen("lastUpdateTime") + 3, 50)
| extend eventTime = substring(eventTime, 0, indexof(eventTime, ",") - 1)
| extend eventTime = todatetime(tostring(eventTime))
| where eventTime >= ago(2h)
| where upgradeType == "K8sVersionUpgrade"
| project
    eventTime,
    upgradeType,
    status,
    properties
| order by eventTime asc

  4. The interval should be 30 minutes, and the threshold should be 1.

  5. Make sure that an action group with the correct email address exists, so that you can receive the notifications.

  6. Make sure to give the Read role to the resource group and to the subscription to the managed identity of the log search alert rule.

  7. Go to the alert rule: Settings > Identity > System assigned managed identity > Azure role assignments > Add role assignment.

  8. Select the Reader role and assign it to the resource group. Repeat Add role assignment for the subscription.

Verification

To upgrade the cluster, wait for the autoupgrader to start. Then verify that you promptly receive notices on the email configured to receive notices.

Check the Azure Resource Graph database for the scheduled notification record. Each scheduled event notification should be listed as one record in the containerserviceeventresources table.

Screenshot that shows how to look up Azure Resource Graph.

Related content