Enable secret encryption on an AKS Edge Essentials cluster

Following Kubernetes security best practices, it's recommended that you encrypt the Kubernetes secret store on AKS Edge Essentials clusters. You can perform this encryption by activating the Key Management Service (KMS) provider for AKS Edge Essentials, which enables encryption at rest for secrets stored in the etcd key-value store. It enables this encryption by generating a Key Encryption Key (KEK) and automatically rotating it every 30 days. The KEK is protected with administrator credentials and is accessible only to administrators.

For more detailed information about using KMS, see the official KMS provider documentation.

This article describes how to activate the KMS provider for AKS Edge Essentials clusters.

Prerequisites

The KMS provider is supported for all AKS Edge Essentials clusters, version 1.10.868.0 and later.

Enable the KMS provider

In your aksedge-config.json file, in the Init section, set Init.KmsPlugin.Enable to true:

"Init": {
 "KmsPlugin": {
     "Enable": true
  }
}

The following output is displayed during deployment, showing that the KMS provider is enabled:

Preparing to install kms-plugin as encryption provider...

For deployment instructions, see Single machine deployment.

Verify that the KMS provider is enabled

To verify that the KMS provider is enabled, run the following command and ensure that the health status of kms-providers is OK:

kubectl get --raw='/readyz?verbose'

[+]ping ok
[+]Log ok
[+]etcd ok
[+]kms-providers ok
[+]poststarthook/start-encryption-provider-config-automatic-reload ok

To create secrets in AKS Edge Essentials clusters, see Managing Secrets using kubectl in the Kubernetes documentation.

If you encounter errors, see the Troubleshooting section.

How to update your secrets after KEK is rotated

The KEK is automatically updated every 30 days. At that point, each secret remains encrypted with the KEK that was in use when you created it. When you next update the secret, it's re-encrypted with the current KEK. If you don't regularly update your secret values as part of your regular processes, then consider re-writing them (with the same value) every 30 days anyway. This ensures you are following Kubernetes best practices and that every secret is encrypted with the latest KEK. For larger clusters, consider updating each namespace's secrets in turn, or developing a script or other automation to streamline the process:

kubectl get secrets --all-namespaces -o json | kubectl replace -f - 

Troubleshooting

If there are errors with the KMS provider, follow this procedure:

1. Check that the AKS version is 1.10.868.0 or later. Use the following command to check the current version of AKS Edge Essentials:

   Get-Command -Module AKSEdge | Format-Table Name, Version
   

   If the version is older, upgrade to the latest version. For more information, see Upgrade an AKS cluster.

2. View the readyz API. If the problem persists, verify that the KMS provider is enabled. See the Verify that the KMS provider is enabled section.

   If you receive "[-]" before the kms-providers field, collect diagnostic logs for debugging. For more information, see Get kubelet logs from cluster nodes.

3. If there are still errors, the machine running the AKS Edge Essentials cluster might be paused or turned off for an extended period of time (over 30 days). To get KMS back into a healthy state, you can use the Repair-Kms command to restore any necessary tokens:

   Repair-AksEdgeKms
   

4. If you still encounter errors, contact Microsoft Customer Support and collect logs.

Next steps

• Overview
• Uninstall AKS cluster