Edit

Share via

Use Deployment Safeguards to enforce best practices in Azure Kubernetes Service (AKS)

This article shows you how to use Deployment Safeguards to enforce best practices on an Azure Kubernetes Service (AKS) cluster.

Overview

Note

Deployment Safeguards is turned on by default in AKS Automatic.

Throughout the development lifecycle, it's common for bugs, issues, and other problems to arise if the initial deployment of your Kubernetes resources includes misconfigurations. To ease the burden of Kubernetes development, Azure Kubernetes Service (AKS) offers Deployment Safeguards. Deployment Safeguards enforce Kubernetes best practices in your AKS cluster through Azure Policy controls.

Deployment Safeguards offer two levels of configuration:

  • Warn: Displays warning messages in the code terminal to alert you of any noncompliant cluster configurations but still allows the request to go through.
  • Enforce: Enforces compliant configurations by denying and mutating deployments if they don't follow best practices.

After you configure Deployment Safeguards for 'Warn' or 'Enforce', Deployment Safeguards programmatically assess your Kubernetes resources at creation or update time for compliance. Deployment Safeguards also display aggregated compliance information across your workloads at a per resource level via Azure Policy's compliance dashboard in the Azure portal or in your CLI or terminal. Running a noncompliant workload indicates that your cluster isn't following best practices and that workloads on your cluster are at risk of experiencing issues caused by your cluster configuration.

Prerequisites

Note

Cluster admins don't need Azure Policy permissions to enable or disable Deployment Safeguards. However, it's required to have the Azure Policy add-on installed.

Deployment Safeguards policies

The following table lists the policies that become active and the Kubernetes resources they target when you enable Deployment Safeguards. You can view the currently available Deployment Safeguards in the Azure portal as an Azure Policy definition or at Azure Policy built-in definitions for Azure Kubernetes Service. The intention behind this collection is to create a common and generic list of best practices applicable to most users and use cases.

Deployment safeguard policy Mutation outcome if available
Cannot Edit Individual Nodes N/A
Kubernetes cluster containers CPU and memory resource limits shouldn't exceed the specified limits Sets CPU resource limits to 500m if not set and sets memory limits to 500Mi if no path is present
Must Have Anti Affinity Rules or topologySpreadConstraintsSet N/A
No AKS Specific Labels N/A
Kubernetes cluster containers should only use allowed images N/A
Reserved System Pool Taints Removes the CriticalAddonsOnly taint from a user node pool if not set. AKS uses the CriticalAddonsOnly taint to keep customer pods away from the system pool. This configuration ensures a clear separation between AKS components and customer pods and prevents eviction of customer pods that don't tolerate the CriticalAddonsOnly taint.
Ensure cluster containers have readiness or liveness probes configured N/A
Kubernetes clusters should use Container Storage Interface (CSI) driver StorageClass N/A
Kubernetes cluster services should use unique selectors N/A
Kubernetes cluster container images should not include latest image tag N/A

If you want to submit an idea or request for Deployment Safeguards, open an issue in the AKS GitHub repository and add [Deployment Safeguards request] to the beginning of the title.

Enable Deployment Safeguards

Note

Using the Deployment Safeguards Enforce level means you're opting in to deployments being blocked and mutated. Please consider how these policies might work with your AKS cluster before enabling Enforce.

Enable Deployment Safeguards on an existing cluster

Enable Deployment Safeguards on an existing cluster that has the Azure Policy add-on enabled using the az aks safeguard create command with the --level flag. If you want to receive noncompliance warnings, set the --level to Warn. If you want to deny or mutate all noncompliant deployments, set it to Enforce.

az aks safeguards create --resource-group <resource-group-name> --name <cluster-name> --level Enforce

You can also enable Deployment Safeguards by using the --cluster flag and specifying the cluster resource ID.

az aks safeguards create --cluster <ID> --level Enforce

If you want to update the Deployment Safeguards level of an existing cluster, run the following command with the new value for --level.

az aks safeguards update --resource-group <resource-group-name> --name <cluster-name> --level Warn

Excluding namespaces

You can also exclude certain namespaces from Deployment Safeguards. When you exclude a namespace, activity in that namespace is unaffected by Deployment Safeguards warnings or enforcement.

For example, to exclude the namespaces ns1 and ns2, use a space separated list of namespaces with the --excluded-ns flag, as shown in the following example:

az aks safeguards update --resource-group <resource-group-name> --name <cluster-name> --level Warn --excluded-ns ns1 ns2

Update your deployment safeguard version

Deployment Safeguards adhere to the AKS addon versioning scheme. Each new version of a deployment safeguard will be released as a new minor version in AKS. These updates will be communicated through the AKS GitHub release notes and reflected in the "Deployment Safeguards Policies" table in our documentation.

To learn more about AKS versioning and addons, refer to the following documentation: aks-component-versions and aks-versioning-for-addons.

Verify compliance across clusters

After deploying your Kubernetes manifest, you see warnings or a potential denial message in your CLI or terminal if the cluster isn't compliant with Deployment Safeguards, as shown in the following examples:

Warn

$ kubectl apply -f deployment.yaml
Warning: [azurepolicy-k8sazurev1antiaffinityrules-ceffa082711831ebffd1] Deployment with 2 replicas should have either podAntiAffinity or topologySpreadConstraints set to avoid disruptions due to nodes crashing
deployment.apps/simple-web created

Enforce

With deployment safeguard mutations, the Enforce level mutates your Kubernetes resources when applicable. However, your Kubernetes resources still need to pass all safeguards to deploy successfully. If any safeguard policies fail, your resource is denied and won't be deployed.

$ kubectl apply -f deployment.yaml 
Error from server (Forbidden): error when creating "deployment.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev1antiaffinityrules-ceffa082711831ebffd1] Deployment with 2 replicas should have either podAntiAffinity or topologySpreadConstraints set to avoid disruptions due to nodes crashing

If your Kubernetes resources comply with the applicable mutation safeguards and meet all other safeguard requirements, they will be successfully deployed, as shown in the following example:

$ kubectl apply -f deployment.yaml
deployment.apps/simple-web created

Verify compliance across clusters using the Azure Policy dashboard

To verify Deployment Safeguards have been applied and to check on your cluster's compliance, navigate to the Azure portal page for your cluster and select Policies, then select go to Azure Policy.

From the list of policies and initiatives, select the initiative associated with Deployment Safeguards. You see a dashboard showing compliance state across your AKS cluster.

Note

To properly assess compliance across your AKS cluster, the Azure Policy initiative must be scoped to your cluster's resource group.

Disable Deployment Safeguards

To disable Deployment Safeguards on your cluster, use the delete command.

az aks safeguards delete --resource-group <resource-group-name> --name <cluster-name>

FAQ

Can I create my own mutations?

No. If you have an idea for a safeguard, open an issue in the AKS GitHub repository and add [Deployment Safeguards request] to the beginning of the title.

Can I pick and choose which mutations I want in Enforcement?

No. Deployment Safeguards is all or nothing. Once you turn on Warn or Enforce, all safeguards will be active.

Why did my deployment resource get admitted even though it wasn't following best practices?

Deployment Safeguards enforce best practice standards through Azure Policy controls and has policies that validate against Kubernetes resources. To evaluate and enforce cluster components, Azure Policy extends Gatekeeper. Gatekeeper enforcement also currently operates in a fail-open model. As there's no guarantee that Gatekeeper will respond to our networking call, we make sure that in that case, the validation is skipped so that the deny doesn't block your deployments.

To learn more, see workload validation in Gatekeeper.

Next steps