Manual backup and recovery of guest state protection keys for Trusted launch Azure Local VMs enabled by Azure Arc

On a secure computer using PowerShell 7, generate a wrapping key of size 2048:

$rsa = [System.Security.Cryptography.RSA]::Create(2048) 

$privateKeyPem = $rsa.ExportPkcs8PrivateKeyPem() 

$privateKeyPem | Out-File -FilePath .\private.pem 

$publicKeyPem = $rsa.ExportRSAPublicKeyPem() 

$publicKeyPem | Out-File -FilePath .\public.pem 

Make a note of the wrapping key as you need it later.

Copy the .\public.pem file to your Azure Local instance.

Copy VM guest state protection keys from the local key vault of your Azure Local instance to a folder that is backed up periodically:

1. Download the TvmBackupUtils.psm1 script on GitHub to your Azure Local instance.

2. Run the following:

   import-module .\TvmBackupUtils.psm1 -force
   
   Backup-TVMKeys -WrappingKeyPath <path to public.pem> -BackupRootPath <path to backup root folder where the timestamped backup folder is stored>
   

   Here's a sample output after running this command:

   Backing up TVM Vault keys to .\<backup root folder>\20250722192116
   Backing up key 11111111-1111-1111-1111-111111111111 to AES folder
   Backing up key 7fb16fe7-00a0-476f-92b3-ccb98fd9525a to AES folder
   Backing up key AzureStackTvmAKRootKey to RSA folder
   

Make note of the timestamped backup folder created under the backup root folder. You need this later during recovery. For example, backup folder named "2025-06-12-20-53-55" with the format "yyyy-MM-dd-HH-mm-ss".

Copy both private and public key files for the wrapping key that you created previously to the Azure Local instance.

Copy the timestamped backup folder to the Azure Local instance. Pick the folder under the backup root folder with the latest timestamp as that folder will have the most recent copy. Don't modify the backup folder.

Import the wrapping key that you created previously to the Azure Local instance:

1. Download the TvmBackupUtils.psm1 script on GitHub to your Azure Local instance.

2. Run the following commands.

   Note

   Make sure to create a unique name for the WrappingKeyName. Otherwise, this causes a failure during import:

   Import-Module .\TvmBackupUtils.psm1 -force 
   
   Import-TvmWrappingKeyFromPem -KeyName <WrappingKeyName> -PublicKeyPath <path to public.pem> -PrivateKeyPath <path to private.pem> -KeySize 2048
   

   Here's sample output:

   Generating import JSON for key <WrappingKeyName> at temporary ___location C:\Users\HCIDeploymentUser\AppData\Local\Temp\tmpD383.tmp... 
   Importing key <WrappingKeyName> into the vault...
   Key <WrappingKeyName> successfully imported into the vault.
   Temporary file C:\Users\HCIDeploymentUser\AppData\Local\Temp\tmpD383.tmp has been cleaned up.
   

Do this step only if you're restoring the VM to the same Azure Local instance where the VM resided before failure. Delete AzureStackTvmAKRootKey as follows:

Remove-MocKey -name  AzureStackTvmAKRootKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault

Restore the keys from backup:

Import-TVMKeys -WrappingKeyName <WrappingKeyName> -BackupPath <path to timestamped backup folder>

Here's sample output:

Importing TVM  keys from .\tvm_keys_backup_root\20250722192116\

Importing key 11111111-1111-1111-1111-111111111111 with size 256 from AES folder path = .\tvm_keys_backup_root\20250722192116\AES\11111111-1111-1111-1111-111111111111_256.json

Importing key 7fb16fe7-00a0-476f-92b3-ccb98fd9525a with size 256 from AES folder path = .\tvm_keys_backup_root\20250722192116\AES\7fb16fe7-00a0-476f-92b3-ccb98fd9525a_256.json

Importing key AzureStackTvmAKRootKey with size 4096 from RSA folder path = .\tvm_keys_backup_root\20250722192116\RSA\AzureStackTvmAKRootKey_4096.json

If the local key vault of the Azure Local instance already has a VM guest state protection key with the same name or already has an AzureStackTvmAKRootKey, you receive an InvalidVersion error for that key. You can ignore this, as the key is already in the key vault.

Here's sample output showing this error:

Importing TVM  keys from .\tvm_keys_backup_root\20250722192116\
Importing key 11111111-1111-1111-1111-111111111111 with size 256 from AES folder path = .\tvm_keys_backup_root\20250722192116\AES\11111111-1111-1111-1111-111111111111_256.json
Import-TVMKeys : Error Importing Key: C:\Program Files\AksHci\mocctl.exe --cloudFqdn
s-cluster.v.masd.stbtest.microsoft.com  security keyvault key import --group "AzureStackHostAttestation" --key-size
"256" --vault-name "AzureStackTvmKeyVault" --key-type "AES" --key-file-path
".\tvm_keys_backup_root\20250722192116\AES\11111111-1111-1111-1111-111111111111_256.json" --name
"11111111-1111-1111-1111-111111111111" --wrapping-key-name "WrappingKey" System.Collections.Hashtable.generic_non_zero
1 [Error: Keys Import failed:  Type[Key] Vault[AzureStackTvmKeyVault] Name[11111111-1111-1111-1111-111111111111]:
InvalidVersion]
+ CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-TVMKeys

Importing key 7fb16fe7-00a0-476f-92b3-ccb98fd9525a with size 256 from AES folder path = .\tvm_keys_backup_root\20250722192116\AES\7fb16fe7-00a0-476f-92b3-ccb98fd9525a_256.json
Import-TVMKeys : Error Importing Key: C:\Program Files\AksHci\mocctl.exe --cloudFqdn
s-cluster.v.masd.stbtest.microsoft.com  security keyvault key import --group "AzureStackHostAttestation" --key-size
"256" --vault-name "AzureStackTvmKeyVault" --key-type "AES" --key-file-path
".\tvm_keys_backup_root\20250722192116\AES\7fb16fe7-00a0-476f-92b3-ccb98fd9525a_256.json" --name
"7fb16fe7-00a0-476f-92b3-ccb98fd9525a" --wrapping-key-name "WrappingKey" System.Collections.Hashtable.generic_non_zero
1 [Error: Keys Import failed:  Type[Key] Vault[AzureStackTvmKeyVault] Name[7fb16fe7-00a0-476f-92b3-ccb98fd9525a]:
InvalidVersion]
+ CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-TVMKeys

Importing key AzureStackTvmAKRootKey with size 4096 from RSA folder path = .\tvm_keys_backup_root\20250722192116\RSA\AzureStackTvmAKRootKey_4096.json
Import-TVMKeys : Error Importing Key: C:\Program Files\AksHci\mocctl.exe --cloudFqdn
s-cluster.v.masd.stbtest.microsoft.com  security keyvault key import --group "AzureStackHostAttestation" --key-size
"4096" --vault-name "AzureStackTvmKeyVault" --key-type "RSA" --key-file-path
".\tvm_keys_backup_root\20250722192116\RSA\AzureStackTvmAKRootKey_4096.json" --name "AzureStackTvmAKRootKey"
--wrapping-key-name "WrappingKey" System.Collections.Hashtable.generic_non_zero 1 [Error: Keys Import failed:
Type[Key] Vault[AzureStackTvmKeyVault] Name[AzureStackTvmAKRootKey]: InvalidVersion]
+ CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Import-TVMKeys

Clean up files and keys:

1. Delete both public.pem and private.pem files from the Azure Local instance.

   Important

   Remove the wrapping key from the key vault of the Azure Local instance using Remove-MocKey. This helps avoid collisions later.

   Remove-MocKey -name WrappingKeyName -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault
   

On the Azure Local system with the backup key vault, run the following commands on the Azure Local system with the backup key vault:

1. Create a wrapping key in the backup key vault. Make note of the name as you need it later:

   New-MocKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type RSA -size 2048
   

2. Download the Privacy Enhanced Mail (PEM) file:

   Get-MocKeyPublicKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -outputFile wrappingKey.pem
   

On the Azure Local system where the VM resides, run the following commands on the Azure Local system:

1. Copy the PEM file to the Azure Local system.

2. Confirm the owner node of the VM by running:

   Get-ClusterGroup <VM name>
   

3. Run the following cmdlet on the owner node to determine the VM ID:

   (Get-VM -Name <VM name>).vmid
   

4. Export the VM guest state protection key:

   Export-MocKey -name <VM ID> -wrappingKeyName wrappingKey -wrappingPubKeyFile wrappingKey.pem -outFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -size 256
   

On the Azure Local system with the backup key vault, run the following steps:

1. Copy the VM ID and VM ID.json file to the Azure Local system.

2. Import the VM guest state protection key to the backup key vault:

   Import-MocKey -name <VM ID> -importKeyFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type AES -size 256
   

On the source Azure Local system where the VM needs to be restored, run the following commands:

1. Create a wrapping key in the key vault:

   New-MocKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type RSA -size 2048
   

2. Download the PEM file:

   Get-MocKeyPublicKey -name wrappingKey -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -outputFile wrappingKey.pem
   

On the Azure Local system with the backup key vault, run the following commands:

1. Copy the PEM file to the Azure Local system.

2. Get the VM ID from the VM files stored on disk. There's a VM config file (.xml) that has the VM ID as its name. You can also use the following command to obtain the VM ID if you know the VM name. Perform this step on a Hyper-V host that has the VM files:

   (Get-VM -Name <VM name>).vmid
   

3. Export the VM guest state protection key:

   Export-MocKey -name <VM ID> -wrappingKeyName wrappingKey -wrappingPubKeyFile wrappingKey.pem -outFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -size 256
   

On the Azure Local system where the VM needs to be restored, run the following commands from the target Azure Local system:

1. Copy the VM ID and VM ID.json file to the Azure Local system.

2. Import the VM guest state protection key:

   Import-MocKey -name <VM ID> -importKeyFile <VM ID>.json -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault -type AES -size 256
   

   Note

   Restore the VM guest state key (complete the preceding steps) before you start the VM on the Azure Local instance where the VM needs to be restored. This ensures that the VM uses the restored VM guest state protection key. Otherwise, the VM creation fails, and a new VM guest state protection key is created by the system. If this happens by mistake (human error), delete the VM guest state protection key and then repeat the steps to restore the VM guest state protection key.

   Remove-MocKey -name <vm id> -group AzureStackHostAttestation -keyvaultName AzureStackTvmKeyVault