Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Pod sandboxing on AKS uses Kata Containers to run each sandboxed pod in a lightweight virtual machine (VM) with its own guest kernel. This compute boundary isolates the workload from the host kernel and workloads in other pod VMs. To learn about the architecture, benefits, and use cases, see Overview of Pod sandboxing in Azure Kubernetes Service (AKS).
In this article, you deploy Pod sandboxing on a new or existing Azure Linux AKS cluster, deploy an application with the Kata runtime, and optionally verify kernel isolation.
Deploying Pod sandboxing by using Kata Containers is similar to the standard runc workflow to deploy containers. Clusters with Pod sandboxing enabled come with a specific runtime class that you can reference in a pod manifest, runtimeClassName: kata-vm-isolation (GA) or runtimeClassName: kata-vm-isolation-v2 (preview). For more information about runtime class defaults and customization, see Considerations for Pod sandboxing.
Limitations
The following constraints apply to Pod sandboxing with kata-vm-isolation:
- Kata containers might not reach the IOPS performance limits that traditional containers can reach on Azure Files and high-performance local SSD.
- Microsoft Defender for Containers doesn't support assessing Kata runtime pods.
- Kata host-network access isn't supported. You can't directly access host networking configuration from within the pod VM.
- CPU and memory allocation differs from
runc. Each Kata pod runs in a pod VM whose size comes from pod limits: - Pod sandboxing is supported only on Azure Linux. Ubuntu, Windows, and Azure Container Linux aren't supported. For more information, see Azure Linux container host for AKS.
- You can't enable Federal Information Processing Standard (FIPS) or Trusted Launch on a node pool that uses Pod sandboxing.
- Arm64 architecture isn't supported on a node pool that uses Pod sandboxing.
Prerequisites
- The Azure CLI version 2.80.0 or later. Run
az --versionto find the version of your Azure CLI, and runaz upgradeto upgrade. For more details, see the steps at Install Azure CLI. - To manage a Kubernetes cluster, use the Kubernetes command-line client
kubectl. Azure Cloud Shell comes withkubectl. You can installkubectllocally using theaz aks install-clicommand.
Limitations
The following constraints apply to Pod sandboxing with kata-vm-isolation-v2:
kata-vm-isolation-v2is in preview. You must register theKataVMIsolationV2Previewfeature flag and use Azure Linux node image version202610.30.1or later. Regional rollout can vary by region. For current availability, see the AKS release tracker.- Kata containers might not reach the IOPS performance limits that traditional containers can reach on Azure Files and high-performance local SSD.
- Microsoft Defender for Containers doesn't support assessing Kata runtime pods.
- Kata host-network access isn't supported. You can't directly access host networking configuration from within the pod VM.
- Runtime-class migration is an explicit workload change. Don't assume behavior is identical to
kata-vm-isolationacross all workload and storage patterns. - For
emptyDirvolumes, explicitly setmedium: MemoryandsizeLimitin the pod spec. For more information, see Pod sandboxing considerations. - CPU and memory sizing behavior can differ from
kata-vm-isolationandkata-vm-isolation-v2when you initialize the template VM with the DaemonSet in Create sandboxed pods from a paused, preinitialized template VM (kata-vm-isolation-v2preview). After the template is initialized, each v2 pod receives a 2 GiB, 2 vCPU guest regardless of pod limits. For memory sizing and overhead details, see RuntimeClass memory overhead. - Pod sandboxing is supported only on Azure Linux. Ubuntu, Windows, and Azure Container Linux aren't supported. For more information, see Azure Linux container host for AKS.
- You can't enable Federal Information Processing Standard (FIPS) or Trusted Launch on a node pool that uses Pod sandboxing.
- Arm64 architecture isn't supported on a node pool that uses Pod sandboxing.
Use kata-vm-isolation-v2 (preview)
Important
AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:
The kata-vm-isolation-v2 runtime class is available in preview. For its benefits and a comparison with v1, see Runtime class v2 improvements. For memory sizing and overhead details, see RuntimeClass memory overhead.
Important
The node pool still uses --workload-runtime KataVmIsolation for both runtime classes. The v1/v2 choice is made in each workload manifest through runtimeClassName.
Prerequisites for kata-vm-isolation-v2
Before you use kata-vm-isolation-v2, confirm all of the following prerequisites:
- The Azure CLI version 2.80.0 or later.
kubectlis installed and configured for your cluster.- The
KataVMIsolationV2Previewfeature is registered in your subscription. - The cluster runs Kubernetes version 1.34.0 or later.
- The target Pod sandboxing node pool uses Azure Linux node image version
202610.30.1or later. Regional rollout can vary by region. For current availability, see the AKS release tracker. kata-vm-isolation-v2supports v6 and v7 VM SKUs in these families: Dasv6, Dadsv6, Dalsv6, Daldsv6, Easv6, Eadsv6, Falsv6, Fasv6, and Famsv6.
Register the KataVMIsolationV2Preview feature flag
Register the
KataVMIsolationV2Previewfeature flag by using theaz feature registercommand.az feature register \ --namespace "Microsoft.ContainerService" \ --name "KataVMIsolationV2Preview"Verify the registration status by using the
az feature showcommand.az feature show \ --namespace "Microsoft.ContainerService" \ --name "KataVMIsolationV2Preview"When the status shows Registered, refresh the registration of the Microsoft.ContainerService resource provider by using the
az provider registercommand.az provider register --namespace "Microsoft.ContainerService"
Create a cluster with Pod sandboxing
Perform the following steps to deploy an Azure Linux AKS cluster using the Azure CLI.
Create an AKS cluster by running the
az aks createcommand with the following parameters:Parameter Required value Notes --workload-runtimeKataVmIsolationEnables Pod sandboxing on the node pool. --os-skuAzureLinuxPod sandboxing supports only the Azure Linux OS SKU. --node-vm-sizeA generation 2 VM size that supports nested virtualization For example, use Standard_D4ads_v6.--node-count3Creates three nodes for this example. Adjust the value for your workload. Parameter Required value Notes --workload-runtimeKataVmIsolationEnables Pod sandboxing on the node pool. --os-skuAzureLinuxPod sandboxing supports only the Azure Linux OS SKU. --node-vm-sizeA generation 2 VM size that supports nested virtualization For example, use Standard_D4ads_v6.--node-count3Creates three nodes for this example. Adjust the value for your workload. The following example creates a cluster named myAKSCluster with three nodes in myResourceGroup:
az aks create \ --name myAKSCluster \ --resource-group myResourceGroup \ --os-sku AzureLinux \ --workload-runtime KataVmIsolation \ --node-vm-size Standard_D4ads_v6 \ --node-count 3 \ --generate-ssh-keysRun the following command to get access credentials for the Kubernetes cluster. Use the
az aks get-credentialscommand and replace the values for the cluster name and the resource group name.az aks get-credentials --resource-group myResourceGroup --name myAKSClusterList all Pods in all namespaces by running the
kubectl get podscommand.kubectl get pods --all-namespaces
Enable Pod sandboxing on an existing cluster
Add a node pool to your AKS cluster by using the
az aks nodepool addcommand with the following parameters:Parameter Required value Notes --resource-groupThe resource group that contains the existing AKS cluster For example, use myResourceGroup.--cluster-nameThe name of the existing AKS cluster For example, use myAKSCluster.--nameA unique node pool name For example, use nodepool2.--workload-runtimeKataVmIsolationEnables Pod sandboxing on the node pool. --os-skuAzureLinuxPod sandboxing supports only the Azure Linux OS SKU. --node-vm-sizeA generation 2 VM size that supports nested virtualization For example, use Standard_D4ads_v6.--node-count1Creates one node for this example. Adjust the value for your workload. The following example adds a node pool to myAKSCluster with one node in nodepool2 in the myResourceGroup:
az aks nodepool add --cluster-name myAKSCluster --resource-group myResourceGroup --name nodepool2 --os-sku AzureLinux --workload-runtime KataVmIsolation --node-vm-size Standard_D4ads_v6 --node-count 1Verify that the node pool was added successfully and that the Pod sandboxing node pool uses the expected workload runtime, OS SKU, and VM size.
az aks show \ --name myAKSCluster \ --resource-group myResourceGroup \ --query provisioningState \ --output tsvaz aks nodepool show \ --cluster-name myAKSCluster \ --resource-group myResourceGroup \ --name nodepool2 \ --query "{workloadRuntime:workloadRuntime, osSku:osSku, vmSize:vmSize}" \ --output tableThe cluster provisioning state should be
Succeeded. The node pool output should showKataVmIsolation,AzureLinux, and the VM size you selected.
Deploy applications using Pod sandboxing
By using Pod sandboxing, you can deploy a mix of standard pods that don't use the Kata runtime alongside Kata pods that use the runtime.
The runtime class selects the Kata runtime, but you should also define scheduling constraints when you need workloads to run on a specific Pod sandboxing node pool. For example, use the built-in AKS node pool label in a nodeSelector. Replace nodepool2 with the name of your Pod sandboxing node pool.
spec:
runtimeClassName: kata-vm-isolation
nodeSelector:
kubernetes.azure.com/agentpool: nodepool2
For stronger separation, you can taint the Pod sandboxing node pool and add a matching toleration to Kata workloads. A taint without a matching toleration prevents other workloads from scheduling on that node pool. For more information, see Use node taints in an AKS cluster.
Deploy an application with the Kata runtime
To deploy a pod with the Kata runtime on your AKS cluster, perform the following steps.
Create a file named kata-app.yaml to describe your Kata pod, and then paste the following manifest.
kind: Pod apiVersion: v1 metadata: name: isolated-pod spec: runtimeClassName: kata-vm-isolation containers: - name: kata image: mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11 command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]Deploy the Kubernetes pod by running the
kubectl applycommand and specify your kata-app.yaml file:kubectl apply -f kata-app.yamlThe output of the command resembles the following example:
pod/isolated-pod created
Deploy applications using Pod sandboxing
Use the following steps to run a new workload with runtimeClassName: kata-vm-isolation-v2. For an existing v1 workload, follow the migration instructions instead.
Deploy an application with the Kata runtime (kata-vm-isolation-v2)
Confirm your target Pod sandboxing node pool is eligible (Kubernetes 1.34.0 or later and Azure Linux node image version
202610.30.1or later).Verify the runtime class and node label by using the steps in Verify the runtime class and node label.
Create a file named
kata-v2-app.yamlwith the following manifest.apiVersion: v1 kind: Pod metadata: name: isolated-pod-v2 spec: runtimeClassName: kata-vm-isolation-v2 containers: - name: kata image: mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11 command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]Deploy the pod and verify that it uses the preview runtime class:
kubectl apply -f kata-v2-app.yaml kubectl wait --for=condition=Ready pod/isolated-pod-v2 --timeout=120s kubectl get pod isolated-pod-v2 -o jsonpath='{.spec.runtimeClassName}'The last command should return
kata-vm-isolation-v2.
Important
If you created a new cluster with a node pool that supports --workload-runtime KataVmIsolation , the runtimeclass should appear automatically. However, if you added a new node pool to an existing cluster, the reconciler might take a couple of minutes to update the cluster and add the kata-vm-isolation-v2 runtimeclass.
Create sandboxed pods from a paused, preinitialized template VM (kata-vm-isolation-v2 preview)
kata-vm-isolation-v2 creates new sandboxed pods by cloning from a paused, preinitialized template VM instead of cold-booting each pod VM.
Before you use this path, confirm the following limitations and requirements:
- The workload uses
runtimeClassName: kata-vm-isolation-v2. - The target node pool runs Azure Linux node image version
202610.30.1or later. Regional rollout can vary by region. For current availability, see the AKS release tracker. - You initialize the template VM on each eligible node.
- In the current preview, the template path can use a fixed 2-GiB memory and 2-vCPU guest VM size after template initialization. Keep pod requests and limits aligned, and validate scheduling behavior for your workload.
Important
The init container runs kata-ctl factory init as root on the node, review with your security team before deploying.
To initialize the template VM on each node, you can use a DaemonSet such as the following:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kata-v2-factory-init
namespace: kube-system
spec:
selector:
matchLabels:
app.kubernetes.io/name: kata-v2-factory-init
template:
metadata:
labels:
app.kubernetes.io/name: kata-v2-factory-init
spec:
automountServiceAccountToken: false
hostPID: true
nodeSelector:
kubernetes.io/os: linux
kubernetes.azure.com/kata-vm-isolation-v2: "true"
initContainers:
- name: factory-init
image: mcr.microsoft.com/azurelinux/base/core:3.0
command:
- /bin/bash
- -ec
- |
tdnf install -y util-linux
exec /usr/bin/nsenter "$@"
- kata-v2-factory-bootstrap
- --target=1
- --mount
- --uts
- --ipc
- --net
- --pid
- --root=/proc/1/root
- --wd=/
- --
- /bin/bash
- -ec
- |
config=$1
template=$2
lock=$3
test -r "$config"
command -v kata-ctl >/dev/null
exec 9>"$lock"
flock --exclusive 9
if test -f "$template/memory" &&
test -f "$template/state.json" &&
test -f "$template/config.json"; then
printf 'Factory template already exists at %s\n' "$template"
exit 0
fi
if mountpoint -q "$template" ||
{ test -d "$template" && test -n "$(ls -A "$template")"; }; then
printf 'Incomplete factory at %s; inspect and clean up manually before retrying.\n' "$template" >&2
exit 1
fi
kata-ctl --config "$config" factory init
- kata-v2-factory-init
- /usr/share/defaults/kata-containers/configuration-clh-azure-runtime-rs-v2.toml
- /run/vc/vm/template
- /run/kata-v2-factory-init.lock
securityContext:
privileged: true
runAsUser: 0
resources:
requests:
cpu: 100m
memory: 128Mi
containers:
- name: initialized
image: registry.k8s.io/pause:3.10
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 65534
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
resources:
requests:
cpu: 1m
memory: 4Mi
limits:
memory: 16Mi
Migrate workloads between runtime classes
Before you migrate, review the runtime-class migration considerations and v2 limitations. Use these steps to migrate an existing workload from kata-vm-isolation (v1) to kata-vm-isolation-v2 (preview):
- Complete the v2 preview prerequisites and feature registration. If your existing node pool isn't using a supported VM size and node image, and if you created it before feature registration, you might need to recreate it.
- Verify that the runtime class exists by running
kubectl get runtimeclass kata-vm-isolation-v2. - Change
runtimeClassName: kata-vm-isolationtoruntimeClassName: kata-vm-isolation-v2in your workload manifest. For a Deployment or another controller-managed workload, update the pod template. - Apply the updated controller manifest so the controller creates replacement pods. For a standalone Pod, delete the existing pod and recreate it from the updated manifest; you can't change
runtimeClassNameon an existing pod. Plan for any workload disruption before replacing pods. - Verify that the replacement pods are ready and that their
spec.runtimeClassNameiskata-vm-isolation-v2. Validate storage and mount behavior, startup latency, and resource usage in a preproduction environment before broad rollout. For memory-backedemptyDirvolumes, follow the v2 volume guidance. - If you find compatibility issues, roll back by restoring
runtimeClassName: kata-vm-isolationin the manifest and replacing the pods through the same process.
Troubleshooting
Verify the runtime class and node label
Verify the v2 runtime class exists:
kubectl get runtimeclass kata-vm-isolation-v2If the runtime class is missing, recheck feature registration, run
az aks update, wait for reconciliation to complete, and run the command again.Verify that nodes in the target Pod sandboxing pool carry the Kata runtime label. Replace
nodepool2with your node pool name:kubectl get nodes \ -l kubernetes.azure.com/agentpool=nodepool2 \ -o 'custom-columns=NAME:.metadata.name,KATA_V2:.metadata.labels.kubernetes.azure.com/kata-vm-isolation-v2'If the label is missing, ensure the node pool meets the v2 prerequisites (supported VM SKU, required Kubernetes version, and Azure Linux node image version
202610.30.1or later). Then upgrade the node image (az aks nodepool upgrade --node-image-only) and recheck. If the label is still missing, recreate the node pool with the required prerequisites.
Check which runtime class your workloads use (optional)
If you're unsure which runtime class is used in your pods, you can check by listing the runtime class selected by each pod across all namespaces:
kubectl get pods --all-namespaces \
-o 'custom-columns=NAMESPACE:.metadata.namespace,POD:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName'
In the RUNTIMECLASS column, kata-vm-isolation identifies v1 pods, and kata-vm-isolation-v2 identifies v2 preview pods. A value of <none> means the pod doesn't specify runtimeClassName and uses the node's default runtime. Listing available RuntimeClass resources alone doesn't show which runtime class your workloads select.
Reconcile cluster configuration after adding a Pod sandboxing node pool (optional)
In most cases, az aks nodepool add is sufficient and no extra reconcile step is required. If the runtime class or node labels don't appear as expected after adding a Pod sandboxing node pool, run az aks update and then recheck.
az aks update --name myAKSCluster --resource-group myResourceGroup
Verify kernel isolation (optional)
This section uses the isolated-pod Kata pod (runtimeClassName: kata-vm-isolation) that you created in Deploy an application with the Kata runtime. If you didn't deploy that pod, deploy it now.
To compare the kernels of Kata and non-Kata pods, create a file named normal-app.yaml. The following manifest defines a standard pod without the Kata runtime:
kind: Pod apiVersion: v1 metadata: name: normal-pod spec: containers: - name: non-kata image: mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11 command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"]Deploy the standard pod by using the
kubectl applycommand.kubectl apply -f normal-app.yamlTo access a container inside the AKS cluster, start a shell session by running the
kubectl execcommand. In this example, you're accessing the container inside isolated-pod.kubectl exec -it isolated-pod -- /bin/shKubectl connects to your cluster, runs
/bin/shinside the first container withinisolated-pod, and forwards your terminal's input and output streams to the container's process. You can also start a shell session to the container hosting the non-Kata pod to see the differences.After starting a shell session to the container from isolated-pod, run commands to verify that the kata container is running in a pod sandbox. It has a different kernel version compared to the non-Kata container outside the sandbox.
To see the kernel version run the following command:
uname -rThe following example resembles output from the pod sandbox kernel:
[user]/# uname -r 6.6.96.mshv1Start a shell session to the container from normal-pod to verify the kernel output:
kubectl exec -it normal-pod -- /bin/bashTo see the kernel version run the following command:
uname -rThe following example resembles output from the VM that's running normal-pod, which is a different kernel than the Kata pod running within the pod sandbox:
6.6.100.mshv1-1.azl3
Clean up resources
When you finish evaluating this feature, clean up resources you no longer need to avoid Azure charges. If you deployed a new cluster as part of your evaluation, run the following
az aks deletecommand to delete the AKS cluster.az aks delete --resource-group myResourceGroup --name myAKSClusterIf you deployed Pod sandboxing on an existing cluster, you can remove the pods using the
kubectl delete podcommand.kubectl get pods kubectl delete pod <kata-pod-name>To remove the Pod sandboxing compute resources, you can also delete the Kata node pool by using the
az aks nodepool deletecommand. Before deleting the node pool, ensure no workloads that you want to retain are running on it.az aks nodepool delete \ --cluster-name myAKSCluster \ --resource-group myResourceGroup \ --name nodepool2
Next steps
- Learn more about Azure Dedicated hosts for nodes with your AKS cluster to use hardware isolation and control over Azure platform maintenance events.
- To further explore Pod sandboxing isolation and explore workload scenarios, try out the Pod sandboxing labs.