Helm Configuration

Parameters

The helm chart provides a comprehensive set of configuration options. You may view the parameters and their descriptions by going to the GitHub source or by using helm:

Bash

# List available kata-deploy chart versions:
#   helm search repo kata-deploy-charts/kata-deploy --versions
#
# Then replace X.Y.Z below with the desired chart version:
helm show values --version X.Y.Z oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy

shims

Each shims entry selects a runtime configuration exposed as a Kubernetes RuntimeClass. See the kata-deploy RuntimeClass guide for the purpose and prerequisites of every built-in class before selecting which configurations to enable.

Kata ships a number of pre-built hypervisor artifacts, and the chart creates one RuntimeClass per enabled shim. Every shim values.yaml lists is enabled by default, and the file lists exactly those shims that run on an ordinary node — they read images through whatever snapshotter containerd already uses, and ask nothing of the host beyond hardware virtualization:

• qemu and qemu-runtime-rs — QEMU (amd64, arm64, s390x, ppc64le)
• clh and clh-runtime-rs — Cloud Hypervisor (amd64)
• clh-azure, clh-azure-runtime-rs and openvmm-azure-runtime-rs — Microsoft Hypervisor (mshv) hosts (amd64)
• dragonball — Dragonball (amd64)

So a default helm install gives you a set of RuntimeClasses that can all actually start a pod on a node of a supported architecture.

RuntimeClasses do not schedule by architecture

A RuntimeClass is cluster-wide, and the scheduling constraint the chart puts on it does not include the node's architecture. On a mixed-architecture cluster an amd64-only class such as kata-clh or kata-dragonball is still created, and a pod using it can be scheduled onto an arm64, s390x or ppc64le node, where that handler was never installed. On such a cluster, add the architecture to the class itself through the shim's runtimeClass.nodeSelector, for example --set-string shims.clh.runtimeClass.nodeSelector.kubernetes\.io/arch=amd64.

The hardware- and snapshotter-specific shims are not in values.yaml

Confidential computing (SNP, TDX, IBM SE), NVIDIA, Firecracker and peer-pod shims each come with a prerequisite the chart cannot satisfy on its own: a particular containerd snapshotter (nydus, erofs, devmapper), a specific CPU, a GPU, or a cloud provider running elsewhere. Enabled by default they would create RuntimeClasses that look usable and fail the first pod scheduled onto them, so each lives in a values file you opt into with -f, together with the snapshotter setup and CRI configuration it needs.

Pick the one that matches your nodes from Examples below — try-kata-tee.values.yaml, try-kata-nvidia-gpu.values.yaml, try-kata-nvidia-cpu.values.yaml, try-kata-fc.values.yaml or try-kata-remote.values.yaml.

You may selectively enable or disable specific shims. For example:

YAML

shims:
  disableAll: true
  qemu:
    enabled: true
  clh:
    enabled: false

Shims can also have configuration options specific to them:

YAML

  qemu-runtime-rs:
    enabled: ~
    supportedArches:
      - amd64
    dropIn: |
      [agent.kata]
      dial_timeout = 999
    allowedHypervisorAnnotations: []
    containerd:
      snapshotter: ""
    runtimeClass:
      nodeSelector:
        example.io/feature: "true"

The optional shims.<shim>.dropIn field lets you add a custom Kata drop-in for a default (non-custom) runtime. kata-deploy writes it as config.d/50-user-overrides.toml for that shim.

enabled: true alone cannot bring back a shim that is not in values.yaml

A shim is only considered for an architecture that its own supportedArches lists, and a shim block that does not set the field matches none — so --set shims.qemu-tdx.enabled=true on its own would install nothing, which the chart rejects rather than leaving you a RuntimeClass no pod can run on. Use the profile that carries the shim, or copy its whole block out of that profile, which is also where its snapshotter and guest-pull settings live.

Containerd pod annotations

kata-deploy builds each containerd runtime's pod_annotations allowlist from that runtime's effective Kata configuration. Only io.katacontainers.config.hypervisor.* annotations whose names occur in the runtime's enable_annotations list are forwarded. This includes entries enabled with shims.<shim>.allowedHypervisorAnnotations or a Kata configuration drop-in. Agent, runtime, and container-resource annotations are not forwarded unless listed below.

Additional containerd annotation patterns can be enabled globally or for one shim:

values.yaml

containerd:
  extraPodAnnotations:
    - sgx.intel.com/epc

shims:
  qemu:
    containerd:
      extraPodAnnotations:
        - example.com/kata-*

Patterns use containerd's matching syntax.

Warning

containerd.extraPodAnnotations: ["io.katacontainers.*"] restores the former, unrestricted Kata annotation forwarding. That is a security risk: any Kata annotation reaches the runtime.

With env.multiInstallSuffix, each installation derives the allowlists from its own installed Kata configurations and applies them only to its own suffixed runtime handlers.

It's best to reference the default values.yaml file above for more details.

NVIDIA guest settings

The NVIDIA GPU images boot NVRC as their init process, which brings the NVIDIA stack up before the Kata agent starts. The shims.<shim>.nvrc block configures it. Every key becomes an nvrc.* guest kernel parameter, so it applies to all sandboxes on that shim and takes effect when a sandbox boots. The GPU shims are not in values.yaml, so the block lives in the profile that carries them:

try-kata-nvidia-gpu.values.yaml

shims:
  qemu-nvidia-gpu:
    nvrc:
      enableDCGM: true

enableDCGM runs nv-hostengine and dcgm-exporter inside the guest. The exporter serves GPU metrics on port 9400 of the sandbox, which is the pod IP, so anything that can reach the pod can scrape /metrics without a sidecar:

Bash

curl "http://$(kubectl get pod my-gpu-pod -o jsonpath='{.status.podIP}'):9400/metrics"

It is off by default because it costs guest memory and an extra process in every sandbox on the shim. Enable it per shim, on the shims whose workloads you want to observe.

Note

The block is only meaningful on the qemu-nvidia-gpu* shims, and kata-deploy refuses to install if it finds it on any other shim. DCGM itself ships in the NVIDIA GPU guest images — with composable images it arrives in the GPU extension — so a shim using a guest image without it will not serve metrics.

Under the hood kata-deploy appends nvrc.dcgm=on to the shim's kernel command line, in the same config.d/30-kernel-params.toml drop-in that carries the proxy and debug settings.

defaultShim

defaultShim selects, per architecture, which shim the auto-created default RuntimeClass (runtimeClasses.createDefault) resolves to:

values.yaml

defaultShim:
  amd64: qemu-runtime-rs
  arm64: qemu-runtime-rs
  s390x: qemu-runtime-rs
  ppc64le: qemu-runtime-rs

Since the Kata Containers 4.0 release, the default is the Rust runtime (runtime-rs, qemu-runtime-rs) on every architecture that ships a runtime-rs build (x86_64, aarch64, s390x and ppc64le). The Go runtime remains selectable (e.g. via the kata-qemu RuntimeClass) but is deprecated. See the config migration guide for per-option differences when migrating.

Custom Runtimes

Kata allows you to create custom runtime configurations. This is done by overlaying one of the pre-existing runtime configs with user-provided configs. For example, we can use the qemu-nvidia-gpu as a base config and overlay our own parameters to it:

YAML

customRuntimes:
  enabled: false
  runtimes:
    my-gpu-runtime:
      baseConfig: "qemu-nvidia-gpu"  # Required: existing config to use as base
      dropIn: |                      # Optional: overrides via config.d mechanism
        [hypervisor.qemu]
        default_memory = 1024
        default_vcpus = 4
      runtimeClass: |
        kind: RuntimeClass
        apiVersion: node.k8s.io/v1
        metadata:
          name: kata-my-gpu-runtime
          labels:
            app.kubernetes.io/managed-by: kata-deploy
        handler: kata-my-gpu-runtime
        overhead:
          podFixed:
            memory: "640Mi"
            cpu: "500m"
        scheduling:
          nodeSelector:
            katacontainers.io/kata-runtime: "true"
      # Optional: CRI-specific configuration
      containerd:
        snapshotter: "nydus"  # Configure containerd snapshotter (nydus, erofs, etc.)
      crio:
        pullType: "guest-pull"  # Configure CRI-O runtime_pull_image = true

Again, view the default values.yaml file for more details.

Drop-In Runtime Configuration

The base runtime configuration shipped with Kata Containers can be modified using an overlay method. This can be done directly on the filesystem using the instructions found here. You can also use the customRuntimes.runtimes.[name].dropIn configuration in the helm chart to achieve the same results.

shims.<shim>.hypervisor.rootless

Runs a shim's VMM as a throwaway unprivileged user per sandbox, instead of root:

values.yaml

shims:
  qemu-runtime-rs:
    hypervisor:
      rootless: true

Only the runtime-rs QEMU shims and clh-runtime-rs run their VMM unprivileged, so only they accept it: qemu-runtime-rs and its variants, plus clh-runtime-rs. Asking for it on any other shim fails the install. Everything outside the VMM — the shim, virtiofsd, the guest — is unaffected either way.

Enabling it turns rootless = true on for that shim and provisions the host device access an unprivileged VMM needs, because neither half is any use without the other:

Device	Group	Needed by
/dev/kvm	kvm	every VMM
/dev/sev	kata-sev	AMD SEV-SNP, when an SNP shim is rootless
/dev/uv	kata-se	IBM Secure Execution, when qemu-se-runtime-rs is rootless

Each shim asks only for what it needs, so a node running an SNP shim privileged alongside a rootless qemu-runtime-rs never gets the kata-sev group.

Each device gets group read/write, recorded as a udev rule under /etc/udev/rules.d so it outlives a reboot or the device being re-created. Uninstall removes the rule, as does the next install once no shim asks for rootless. The TEE devices deliberately do not join kvm, so a plain rootless sandbox cannot reach the SEV or ultravisor platform interface just because it needs KVM, and the groups are created empty, so no existing user gains anything.

A device the node does not have is skipped, and one that already grants unprivileged access is left exactly as the node has it, recorded in the rule only when it already has the rule's group and mode.

Only devices the VMM opens by path are provisioned, which is why /dev/vhost-vsock and /dev/vhost-net are absent: the shim opens them as root and passes the descriptors to the VMM, so the VMM user needs no access to them.

Requires udev on the node

Reboot persistence relies on the node running a udev implementation that reads /etc/udev/rules.d (systemd-udevd or eudev) and keeping /etc across reboots — true of every supported k8sDistribution. A node using busybox mdev, or an image that rebuilds /etc at boot such as Bottlerocket, accepts the file and never acts on it. The devices are still reconciled, so sandboxes run until the next reboot, but the permissions do not stick. See How to run a rootless VMM for what to do on those hosts.

Requires deploymentMode: job

The DaemonSet does its host work from a single unprivileged container with no host root, so it cannot create the groups or reconcile the device nodes. The chart refuses a rootless shim in that mode rather than configuring sandboxes that could not start. job is the default mode.

Intel TDX needs nothing here when its quote generation service is reached over AF_VSOCK, which is the supported path. A deployment that points the shim at a QGS Unix socket has to grant the VMM user write access to it separately; see How to run a rootless VMM for that and for the full host access contract.

k8sDistribution

Different Kubernetes distributions keep containerd's configuration in different places, so the chart needs to be told which one this cluster is. The value picks the host directory that gets mounted into the install:

values.yaml

k8sDistribution: k8s   # k8s | k3s | rke2 | k0s | microk8s

Any other value means the default location, /etc/containerd/ — so kubeadm and vanilla are as good as k8s. Set containerd.configDir instead if your containerd matches none of the presets.

A wrong value stops the install before changing the node

This value picks the directory, while which file to write is worked out on the node itself, from the CRI runtime actually running there. Declaring k8s on a k3s cluster therefore describes a perfectly good k3s configuration inside a directory k3s does not read — a node that never runs a Kata workload, with nothing to show for it. The install compares the two and stops before writing anything, naming the value to set. An explicit containerd.configDir overrides the derivation this check is about, so it does not apply in that case.

The value carries one thing beyond that directory: where the kubelet keeps its root directory. A pod's ConfigMap, Secret, projected and downward-API volumes are written under it, and the Go runtime watches that path so later updates to them reach the running guest; the kubelet's Pod Resources API socket sits under it too, which both runtimes read to learn the GPUs a pod was allocated before cold-plugging them. k0s uses /var/lib/k0s/kubelet and microk8s /var/snap/microk8s/common/var/lib/kubelet; k3s, rke2 and vanilla Kubernetes leave the kubelet's own /var/lib/kubelet alone and need nothing.

The kubelet is not the CRI runtime

Unlike the containerd directory, this cannot be worked out on the node: a k0s node running CRI-O or a containerd of its own still keeps its volumes under /var/lib/k0s. So declare the flavour even when you pin containerd.configDir — that pin takes only the check above out of the way. The volume watch is the Go runtime's alone, runtime-rs recognising those volumes by the shape of their paths wherever the kubelet root is, but the socket is read by both. Getting either wrong is quiet: volume updates stop arriving, and GPU cold plug falls back to CDI annotations.

Only default installations are supported

The install finds the node's CRI runtime by probing the systemd units each distribution is known by, and restarts that unit, waits on it for readiness, and hangs the snapshotter's service dependency off it:

Distribution	Units probed
K3s	k3s, k3s-agent
RKE2	rke2-server, rke2-agent
k0s	k0scontroller, k0sworker
MicroK8s	snap.microk8s.daemon-containerd

A node running one of these under any other unit name, under another init system, or with its containerd configuration somewhere other than the location that distribution installs it in, is not supported. Nothing detects it, and k8sDistribution cannot make up the difference: it says which configuration layout to write, never which unit to act on.

K3s is the easiest to get into this state. Its installer accepts INSTALL_K3S_NAME, INSTALL_K3S_SYSTEMD_DIR and INSTALL_K3S_TYPE, and can install under openrc rather than systemd, so leave those unset.

erofs and nydus require a node that has not used them before

kata-deploy writes erofs and nydus settings into containerd's configuration. Those settings — dm-verity mode, the merged-layer layout, the nydus proxy root — have to match what existing layers were built to, or containerd can no longer mount them.

Changing them on a node that has already pulled images under a different erofs or nydus configuration breaks those layers. Deploy kata-deploy with erofs or nydus only on nodes where neither snapshotter has been used before, and leave snapshotter.setup empty on nodes that already have one of them configured.

nodeBinaries

Some of what Kata needs on a node is not part of Kata: containerd's EROFS snapshotter, for instance, needs a mkfs.erofs from erofs-utils 1.8.2 or newer, which most distributions still do not package. When updating the node's packages is not on offer, nodeBinaries takes the binaries out of container images instead:

values.yaml

nodeBinaries:
  erofs-utils:                                      # (1)!
    image: quay.io/kata-containers/erofs-utils:1.9.3
    binaries: [mkfs.erofs, dump.erofs, fsck.erofs]  # (2)!
    pullPolicy: IfNotPresent                        # (3)!

1. Each key names an entry and becomes the name of the container staging it, so it has to be a valid container name — lowercase letters, digits and dashes, beginning and ending with a letter or a digit — and cannot be one of the names kata-deploy's own containers use. The render fails, naming the key, when it is neither.
2. Only what is listed here is taken, so an image built on a distribution does not put the rest of its userland on the node. Each one names a single binary, not a path or a pattern, and the render fails on anything else.
3. Optional; defaults to the chart's imagePullPolicy.

Every entry gets a container of its own, which copies the listed binaries into a pod-local volume and reaches nothing of the node's. One further container then installs whatever was staged into /usr/local/bin, ahead of /usr/bin in containerd's PATH, and records the names it installed so that a later run can take exactly those out again. That container runs the kubectl image, kata-deploy's own being distroless and having no shell to do this with.

Each image needs a POSIX shell, cp, and every binary the entry lists, statically built, in one of /usr/local/bin, /usr/local/sbin, /usr/bin, /usr/sbin, /bin, /sbin or its root. Private images use the chart's imagePullSecrets.

Adding another binary is a values change and nothing else, so this is the way to cover anything else a node turns out to lack.

It will not replace a binary it did not install

A file already in /usr/local/bin under a name an entry claims fails the install, rather than being replaced, and the same goes for two entries claiming the same name. Nothing on the node records where a binary in /usr/local/bin came from, so kata-deploy only ever removes what its own marker file names. Remove the file, or drop it from nodeBinaries to keep using it.

An install it refuses this way changes nothing: every name is checked before any of them is written or removed, so the node keeps the set it already had.

An uninstall takes the binaries out again, and changing an entry replaces what it installed. Dropping every entry leaves them in place until the release is uninstalled.

Side-by-side installs own separate sets

Each release's marker file is named after its env.multiInstallSuffix, so installing or uninstalling one leaves the binaries another one installed alone. Two releases claiming the same name is still a conflict: whichever installs second finds a file it did not install and fails.

One image per architecture being deployed to

An image with no manifest for a node's architecture stalls that node's install on the pull, so cover every architecture in the cluster. The erofs-utils image above is published for amd64 and arm64 only.

This requires deploymentMode: job. The staged pipeline is what puts the binaries in place before the host check looks for them; the DaemonSet runs the whole install in one container and has no such ordering. Setting nodeBinaries in daemonset mode fails the render rather than deploying something that cannot work.

Deployment Modes (DaemonSet vs Job)

The chart can install Kata on nodes in one of two ways, selected with the top-level deploymentMode value.

Both modes install all enabled runtimes and their RuntimeClasses, including the TEE, NVIDIA, Firecracker and remote configurations supplied by profiles, subject to their supported architectures. Choosing a mode does not change the runtime selection; it changes how kata-deploy prepares and manages the nodes.

• daemonset: the long-running kata-deploy DaemonSet installs Kata on every matching node and reverts it when the pod is terminated (i.e. on uninstall). This is the historical behavior and is unchanged.
• job (default): there is no always-on component. A tiny k8s-job-dispatcher Job runs as a post-install/post-upgrade hook, enumerates the selected nodes live via the Kubernetes API, and creates one node-pinned install Job per node. Each per-node Job runs the staged install pipeline as ordered initContainers and then exits:

Text Only

host-check -> artifacts   (initContainers)  ->  cri (main)

With selinux.enabled, one more privileged one-shot container runs ahead of all of them to load the policy the confined stages ask for.

On helm uninstall, a pre-delete dispatcher fans out per-node Jobs that run the pipeline in reverse (revert-cri -> remove-artifacts). Unlike the DaemonSet, nothing keeps running on the node after installation completes, and the dispatcher itself only ever talks to the API server — it never touches the host (so it ships as a separate, minimal image, job.dispatcherImage).

The privilege split is explicit, and it is what job mode buys you over the DaemonSet: the privileged pods hold no API credentials at all. Everything that needs the API server is done by the dispatcher, which runs fully unprivileged (runAsNonRoot, all capabilities dropped, no privilege escalation, read-only root filesystem, RuntimeDefault seccomp), never touches the host, and can be confined to nodes you trust.

job is the default; ask for daemonset explicitly to keep the historical behavior:

Job (default)DaemonSet

values.yaml

deploymentMode: job

values.yaml

deploymentMode: daemonset

The main tradeoffs are:

	job	daemonset
After installation	Per-node Jobs exit, freeing their pod resources; privileged host work carries no API token.	A kata-deploy pod and its API token remain on every selected node for the life of the release.
New matching nodes	Require a Helm upgrade or scheduled reconciliation.	The DaemonSet automatically places an installer pod on each new matching node.
Additional node preparation	The staged pipeline can install host binaries through nodeBinaries and provision rootless VMM access.	Install required host binaries yourself before deploying; nodeBinaries and the chart's rootless option require Job mode.

job mode installs Kata on the nodes in the cluster when the release is applied

The job deployment installs Kata on the nodes that are part of the cluster when helm install or helm upgrade runs. A node that joins the cluster afterwards, whether it replaces an existing node or is added by scaling up, needs special handling to get Kata installed: nothing installs it until the release is applied again, or a scheduled reconcile picks it up. Until then it does not carry the katacontainers.io/kata-runtime label, so pods asking for a Kata RuntimeClass are not scheduled onto it. See Adding nodes in job mode.

daemonset mode has no such gap: the DaemonSet places an installer pod on every matching node as it joins. The price is in permissions: that pod is privileged, carries the kata-deploy ServiceAccount token, which can patch any node in the cluster, and stays on every node Kata runs on for the life of the release. Root on any of those nodes can read that token. In job mode the privileged pods carry no token and exit once the node is installed; see Where the credentials live.

The runtime prerequisites still apply in either mode: selecting a profile does not provide missing hardware, firmware, GPU passthrough setup or external services. See the RuntimeClass guide for the requirements of each profile.

Upgrading a release installed before job became the default

deploymentMode is immutable for the life of a release, so an existing daemonset release does not silently move to per-node Jobs: the upgrade is refused instead. Keep deploymentMode: daemonset in your values to stay where you are, or helm uninstall first to adopt job.

Where the credentials live

The pods that run on your nodes hold no API credentials at all: the per-node Jobs are rendered with automountServiceAccountToken: false and no ServiceAccount of their own. This matters because root on a node can read the ServiceAccount token of any pod running there.

	runs where	privileged on the host	API rights
dispatcher (install, uninstall, and each scheduled reconcile)	one pod, only where you let it schedule (Where the dispatcher runs)	no	nodes: list, get, patch; jobs: create, get, list, delete and pods: list in the release namespace; events: create in default, which is the only namespace an Event about a Node can live in; nodes/proxy: get only when guest pull or image conversion is configured; pods: get, cronjobs: get, list, watch, delete and watch added to jobs only with job.reconcile enabled, because the uninstall hook waits out the deletions it makes and a waited delete is served from an informer
per-node Jobs	every node Kata is installed on	yes	none — no token is mounted
post-delete hook (uninstall only)	one pod, wherever it schedules	no	delete on ClusterRoles, ClusterRoleBindings, Roles, RoleBindings and ServiceAccounts
verification Job (only if verification.pod is set)	one pod, wherever it schedules	no	pods and pod logs (create, delete, get, list, watch), nodes/events/daemonsets/jobs: get, list

The last two rows are the exceptions worth knowing about, and both exist in either mode. The post-delete hook deletes the RBAC the release keeps, so its own rights have to outlive everything else; the verification Job runs the pod you gave it and reports what happened. Both are short-lived — one runs during helm uninstall, the other after helm install/helm upgrade — but neither is pinned, so for those few seconds their tokens can be on any schedulable node. The verification Job is opt-in: leave verification.pod unset and it does not exist.

Three things need the Kubernetes API, so none of them happen inside a per-node Job:

• Labelling the node, and everything that gates it. The dispatcher claims a node with katacontainers.io/kata-runtime=false before its Job starts. It promotes the label to true only once three things hold: that Job succeeded, the node reports Ready again (job.waitNodeReadySeconds), and the node's own .status.runtimeHandlers confirms its CRI runtime is serving the handlers this release installed. Only then does it lift the configured startupTaints. A pod on the node could not offer any of that. The label cannot appear on a node whose pipeline failed half way, nor on one whose runtime quietly ignored the configuration that was written. It survives a kubelet that re-registers after the runtime restart and drops it, because the dispatcher re-applies it until it holds. On uninstall the value drops to false, which stops workloads being scheduled, before that node's Job starts taking the node apart — and the label itself goes only once that Job has succeeded.

  In job mode the claim is mandatory: the dispatcher refuses to start a host-mutating Job unless both labels were written atomically. Otherwise a half-failed install could modify a host that the default uninstall selector cannot discover. In daemonset mode the claim stays best-effort.

  The handler check has one limit worth knowing: .status.runtimeHandlers is populated by kubelet from Kubernetes 1.30 on. A node that does not report the field at all cannot answer, and is labelled on the strength of its Job succeeding. A node that reports the field without any of this release's handlers in it fails, and an empty list counts the same way: that node has answered, it just has nothing this release installed. One handler is enough to pass, because the release names the handlers for every architecture it can install and a node only serves its own; the ones it does not serve are logged. A release that registers no handler names — every shim disabled and no custom runtime — has nothing to ask about, and skips the check. - The node's container runtime version. It is the one fact the install cannot work out on the host itself, so the dispatcher reads it from the Node objects it has already listed and passes it down as an environment variable. Everything else the install needs to know about its node, the Kubernetes flavour included, it gets by asking the host. - The cluster-scoped objects: the RuntimeClasses and the NodeFeatureRule advertising TEE key counts are rendered by the chart (see TEE key advertisement), so nothing on a node needs write access to them. helm uninstall removes them for free.

How this compares to daemonset mode

Installing Kata means writing to the host and restarting the CRI runtime, so something privileged has to run on each node in either mode. Two things differ. The DaemonSet pod carries the kata-deploy ServiceAccount, which can patch any node in the cluster and read any node's kubelet configuration, and it stays on the node — token and all — for the life of the release. A per-node Job carries no token and exits when it is done.

So in job mode a compromised worker node yields no cluster credentials, unless the post-delete hook or the verification Job happens to be running there at that moment. The component that does hold the node-patching token can be kept on the control plane. In daemonset mode that token is, by construction, on every node Kata runs on, for the life of the release.

Switching an existing daemonset release to job mode

The privileged kata-deploy ServiceAccount, ClusterRole and ClusterRoleBinding are annotated helm.sh/resource-policy: keep, so that a DaemonSet pod being terminated still has the API access its own cleanup needs. keep also means Helm leaves them behind on an upgrade that no longer renders them — which is exactly what switching to job mode does. The release stops using them, but the credential that can patch any node in the cluster is still there.

They are cluster-scoped and un-suffixed, so a release in another namespace may share them; the chart cannot safely delete them for you. Once no daemonset release is left, remove them by hand:

Bash

kubectl delete clusterrolebinding kata-deploy-rb
kubectl delete clusterrole kata-deploy-role
kubectl delete serviceaccount kata-deploy-sa -n <release namespace>

With env.multiInstallSuffix set, the names carry that suffix.

Why a dispatcher instead of Helm-rendered per-node Jobs

Rendering one Job per node directly in the chart does not scale: Helm stores the whole rendered release in a single (~1 MiB) Secret and runs hook resources sequentially, so large fleets blow the size limit and/or take far too long. A single Indexed Job or a JobSet removes those limits but cannot guarantee one pod per node once parallelism < node-count: Kubernetes' topology-spread and affinity scheduling ignore completed pods, so as paced pods finish, later pods pile onto a subset of nodes and leave others uncovered.

The dispatcher sidesteps both problems: the Helm release stays O(1) (just the dispatcher + a constant-size ConfigMap holding the per-node Job templates), node membership is resolved at run time, and the dispatcher itself paces the rollout (at most job.parallelism per-node Jobs in flight) while guaranteeing one Job per node. Per-node Jobs are garbage-collected via an ownerReference to the dispatcher and job.ttlSecondsAfterFinished.

Where the dispatcher runs

The dispatcher holds the one token in job mode that can enumerate every node in the cluster and create the privileged per-node Jobs, and root on whatever node it lands on can read it. Confine it to nodes you trust — typically the control plane — with job.dispatcherNodeSelector and job.dispatcherTolerations:

values.yaml

job:
  dispatcherNodeSelector:
    node-role.kubernetes.io/control-plane: ""
  dispatcherTolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

The toleration is needed because control-plane nodes are normally tainted. These settings say nothing about where Kata is installed — that remains the top-level nodeSelector / affinity / tolerations, and the per-node Jobs do not inherit the dispatcher's placement. When job.dispatcherTolerations is empty it falls back to the top-level tolerations, so the dispatcher stays schedulable on a cluster whose every node is tainted.

What pinning does and does not cover

Pinning the dispatcher keeps the release's one cluster-wide, long-lived identity — the one that can enumerate and patch every node — off your workers. The privileged pods that do run on every node mount no token at all.

It does not pin the two short-lived hooks: the post-delete RBAC cleanup and, if you set verification.pod, the verification Job (see Where the credentials live). Both can land on any schedulable node for as long as they run.

That is still a boundary daemonset mode cannot draw, since the pod holding its node-patching token has to run on every node by definition.

Adding nodes in job mode

The dispatcher only runs on helm install / helm upgrade / helm uninstall. There is no dispatcher watching for new nodes, so when you add nodes later, re-run helm upgrade; the dispatcher re-enumerates the cluster and installs the new nodes:

Bash

helm upgrade kata-deploy "${CHART}" --version "${VERSION}" --reuse-values

Each per-node stage is idempotent (it skips when already applied), so the upgrade only does real work on the newly added nodes.

Let your cluster lifecycle tooling re-apply the release

Prefer Flux, Argo CD, or another cluster lifecycle layer to re-apply the release when nodes change, for example as part of the same change that grows a node pool or rolls nodes onto a new image. kata-deploy's installation steps are idempotent, so rerunning them is expected and safe: nodes that are already installed finish quickly with nothing to do, and only the new ones get real work.

Doing it on a schedule instead

On a fleet that grows on its own — an autoscaler, or nodes joining faster than anybody notices — the upgrade above needs somebody to run it, and until they do the new node adds no Kata capacity. Nothing lands on it wrongly (every RuntimeClass selects katacontainers.io/kata-runtime, which is exactly what the install has not written there yet), so a pod asking for a Kata runtime class stays Pending while a node that could have run it takes non-Kata work instead — and if it was the pending pod that grew the fleet in the first place, the next node changes nothing either. job.reconcile turns that upgrade into a CronJob running the same dispatcher, against the same selectors and the same per-node templates:

values.yaml

job:
  reconcile:
    enabled: true
    schedule: "*/15 * * * *"

A tick installs the nodes that have nothing to show yet and leaves the rest of the fleet untouched, so a tick over a settled fleet lists nodes, finds nothing to do, and exits without creating a single pod. It also stands aside when a release rollout is in flight, rather than deleting the per-node Jobs that rollout is waiting on. Failures are visible where you would look for them: the CronJob keeps its last three failed Jobs, and a tick that failed on a node fails as a Job rather than being retried immediately — the next tick is the retry.

It only ever adds nodes

A node that has dropped out of the selection is left alone, where helm upgrade would run the cleanup pipeline on it. That asymmetry is deliberate: a node falling out is usually a label gone wrong somewhere, and taking a host apart on a timer with nobody watching is the worse of the two outcomes. Removals stay with the upgrade you run yourself.

A joining node waits for the next tick

The schedule sets how long a new node goes without Kata. With */15 * * * *, a node can wait up to fifteen minutes, plus its own install, before it is labelled and Kata pods are scheduled onto it, and pods asking for a Kata RuntimeClass stay Pending in the meantime. Choose the interval based on how long your workloads can wait. A tick over a settled fleet only lists the nodes and exits, so a shorter interval costs little.

It covers new nodes, not changes to nodes it has already installed

A tick decides what to do from what the install recorded on each Node object, and it skips nodes that already show a finished install. A node replaced in place is one of those: rebuilt, but re-registering under the same Node object and keeping its labels, it looks installed to the tick even though the host has nothing on it. Changing the release's values likewise needs a helm upgrade. Re-applying the release from your lifecycle tooling covers both cases, because every per-node stage then checks the host itself. Treat the schedule as a safety net for nodes joining between those runs, not as a replacement for them.

Off by default

Enabling this stands up a recurring, privileged rollout, so it is opt-in. It also needs a job.dispatcherImage of 0.2.0 or newer; older dispatchers do not understand the flags a scheduled run needs and every tick fails at startup.

helm uninstall takes the schedule away before it starts reverting nodes (a pre-delete hook that runs ahead of the uninstall dispatcher and removes the CronJob together with anything it still has in flight), so a tick cannot reinstall a node the uninstall has just cleaned.

Recovering from a failed or deleted dispatcher

The dispatcher runs as a blocking post-install/post-upgrade hook Job with restartPolicy: Never and backoffLimit: 0, so if its pod is evicted, drained, or deleted mid-rollout the Job is marked failed and is not restarted automatically — helm install/helm upgrade surfaces the failure rather than leaving you silently half-installed.

What survives the dispatcher dying:

• Per-node Jobs already created keep running. They are independent, nodeName-pinned Jobs, not children of the dispatcher pod, so installs that were already dispatched run to completion. Only nodes still queued (never dispatched) are skipped, so at worst you get partial coverage — never a half-mutated host, because each stage is idempotent.
• Those nodes are not labelled, though. Labelling is the dispatcher's job (it is what lets the per-node Jobs run without a token), so a node whose install finished after the dispatcher died is left installed but not advertised, and no Kata workload is scheduled there. It still carries katacontainers.io/kata-runtime=false from being claimed, so helm uninstall reaches it, and the re-run below labels it.
• Those per-node Jobs carry a (non-controller) ownerReference to the dispatcher Job, so they survive pod deletion but are garbage-collected once the dispatcher Job itself is removed or its job.ttlSecondsAfterFinished elapses. Keep that TTL comfortably larger than a single node's install so in-flight Jobs are not reaped early.

Recovery is the same one-liner as adding nodes — re-run helm upgrade:

Bash

helm upgrade kata-deploy "${CHART}" --version "${VERSION}" --reuse-values

The before-hook-creation delete policy first removes the stale dispatcher Job (cascading away any leftover per-node Jobs); the fresh dispatcher then re-enumerates nodes live, recreates the per-node Jobs (adopting any that still exist rather than duplicating them), and because every stage is idempotent the already-installed nodes are fast no-ops. Coverage converges on the re-run.

Finding out why a node failed

A rollout that fails says which nodes failed and what stopped each of them in three places, as long as the dispatcher lived long enough to say so. Helm prints its summary directly, and the node Event and annotation remain available after the command exits.

From helmFrom the nodeAfterwards

The chart asks Helm to print the dispatcher's log when the hook fails, and the dispatcher ends that log with one line per failed node, so the command that failed is also the one that explains it:

$ helm install kata-deploy ...

level=INFO msg="[2026-09-03T18:22:11Z INFO ] fanning out 2 per-node Job(s) with parallelism 1\n...
Error: 2 node(s) failed:\n  worker-2: its job kata-deploy-install-worker-2 failed:
load-kernel-modules exited 1: this node has no usable virtualization backend: neither
/dev/kvm nor /dev/mshv is present [BackoffLimitExceeded: Job has reached the specified
backoff limit]\n  worker-5: its job kata-deploy-install-worker-5 failed: cri never
started: ImagePullBackOff: Back-off pulling image \"kata-deploy:bad-tag\"\n"
Error: INSTALLATION FAILED: failed post-install: resource Job/kube-system/kata-deploy-install-dispatcher not ready. status: Failed, message: Job Failed. failed: 1/1

Helm prints hook logs as a single escaped line

Helm writes hook output through its structured logger, so the whole log arrives as one level=INFO msg="..." record with newlines escaped to \n. Helm's own closing Error: line says only that the hook Job failed — the reason is inside that record. The dispatcher keeps its output short so the record stays readable, and sed gives it back its newlines:

Bash

helm install kata-deploy "${CHART}" 2>&1 | sed -e 's/\\n/\n/g' -e 's/\\"/"/g'

Nothing is lost either way: the same reason is on the node, in the two forms below, after the command has scrolled away.

The same reason is recorded as an Event against the Node, so it turns up in kubectl describe node and in whatever already collects events from the cluster:

$ kubectl describe node worker-2

Events:
  Type     Reason     Age   From                Message
  ----     ------     ----  ----                -------
  Warning  JobFailed  2m    kata-deploy-job-dispatcher  its job kata-deploy-install-worker-2
                            failed: load-kernel-modules exited 1: this node has no
                            usable virtualization backend

Listing them directly needs the default namespace rather than the release's: a Node has no namespace, and that is the only one the API server accepts an Event about a cluster-scoped object in — the same place the kubelet's own node events go.

Bash

kubectl get events -n default --field-selector involvedObject.kind=Node

Events expire, so the result is also written to the node itself and stays until the next rollout overwrites it:

$ kubectl get nodes -l kata-deploy-job-dispatcher/result=failed

NAME       STATUS   ROLES    AGE   VERSION
worker-2   Ready    <none>   9d    v1.34.1

$ kubectl get node worker-2 -o jsonpath='{.metadata.annotations}'

kata-deploy-job-dispatcher/error:       its job kata-deploy-install-worker-2 failed: ...
kata-deploy-job-dispatcher/finished-at: 2026-09-03T18:22:41Z

What makes any of that possible is that each stage writes why it is giving up to /dev/termination-log, which lands in the pod's status rather than only in its log. The dispatcher reads it the moment a Job is judged, and passes it on. The dispatcher's own pod keeps the same summary in its status as a fallback for the three places above: a dispatcher killed before it could report anything is explained by kubectl describe pod alone.

A node that is stuck, rather than failed

Nothing marks a Job whose pod cannot be scheduled or cannot pull its image: it stays running until job.activeDeadlineSeconds expires, which is an hour on the defaults. Two minutes in, and every five after that, the dispatcher says what the wait is for — as a log line and as a Warning/JobPending Event on the node.

Where the log still helps

The summary carries one line per node. The per-node Job's own log carries everything the stage printed on the way there, and job.ttlSecondsAfterFinished (10 minutes by default) is how long you have to read it. Raise it on a cluster where nobody is watching the rollout live.

Choosing which nodes get a Job

Node selection is not configured per deployment mode. Both modes read the same nodeSelector / affinity and tolerations, so one values file installs Kata on the same nodes whichever mode you deploy with.

In daemonset mode the scheduler applies those rules when it places the kata-deploy pods. In job mode there is no pod to place — the dispatcher pins each per-node Job to a node by name — so the dispatcher applies the same rules itself while enumerating nodes:

1. nodeSelector and affinity.nodeAffinity are compiled into node queries, which the dispatcher resolves live against the API server. Each nodeSelectorTerm becomes one query and their results are combined with OR, matching Kubernetes' own semantics; the nodeSelector labels are ANDed into every query, which is how Kubernetes combines the two.
2. Nodes carrying a taint your tolerations do not cover are then dropped, exactly as the scheduler would refuse to place a DaemonSet pod there.
3. job.nodes overrides all of the above with an explicit list of node names, used verbatim.

An empty nodeSelector therefore does not mean "every node": step 2 still applies, and that is what keeps Kata off control-plane nodes by default without any label filtering.

On upgrade, job mode also compares that desired set with the nodes carrying this installation's marker. Nodes in owned - desired run the cleanup pipeline before the desired nodes are installed, so narrowing or changing a selector does not leave the old nodes labelled and configured forever.

values.yaml

# Install on nodes carrying a specific label:
nodeSelector:
  kata-containers: "enabled"

# Richer selection - Linux workers that are either GPU- or SNP-capable
# (a node matching either term is selected):
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - { key: kubernetes.io/os, operator: In, values: ["linux"] }
            - { key: nvidia.com/gpu.present, operator: In, values: ["true"] }
        - matchExpressions:
            - { key: kubernetes.io/os, operator: In, values: ["linux"] }
            - { key: feature.node.kubernetes.io/cpu-cpuid.SEV_SNP, operator: In, values: ["true"] }

# Pin to explicit nodes (job mode only; tainted nodes included, since naming a
# node is unambiguous):
job:
  nodes: ["worker-1", "worker-2"]

The job.* node-selection keys have been removed

Earlier releases selected nodes with job.nodeSelector, job.nodeSelectorExpressions and job.nodeAffinity, separately from the DaemonSet's own knobs. Having two sources of truth meant a values file that pinned Kata to a few nodes in daemonset mode could silently install it everywhere in job mode. Rendering now fails with a migration hint if any of those keys is still set: move the rule to the top-level nodeSelector or affinity.nodeAffinity, and keep job.nodes for explicit node names.

Not every affinity rule can be expressed as a node query

Job mode resolves nodes with a label-selector LIST against the API server, so matchFields and the Gt/Lt operators cannot be represented and are rejected at render time — use job.nodes to target nodes by name. preferredDuringSchedulingIgnoredDuringExecution is accepted but ignored: a preference never restricts where a DaemonSet may land either, so honouring only the required terms keeps both modes in agreement.

Use job.parallelism to pace the rollout — it caps how many per-node Jobs run concurrently (e.g. to limit how many CRI runtimes restart at once on a big fleet). It is effectively capped at the number of targeted nodes.

Nodes that become eligible late

A DaemonSet is level-triggered: it installs a node whenever that node becomes eligible, however long that takes. The dispatcher instead resolves nodes once, at install time, and eligibility frequently arrives seconds later — the feature.node.kubernetes.io/* labels that affinity.nodeAffinity matches are written by node-feature-discovery, which the chart can install in the very same release, and a freshly joined node clears its start-up taints only once it settles.

So the install dispatcher keeps re-resolving for job.waitForNodesSeconds (default 120) while nothing is eligible yet, and fails if the wait expires with no eligible node. Failing is deliberate: in job mode an empty selection is permanent — nothing installs the node later — so a silent success would leave the fleet without Kata and nothing to point at.

Finding one eligible node is not the end of the wait either. Eligibility arrives per node, so the dispatcher waits until the set stays unchanged for job.nodeSettleSeconds (default 15) or the overall wait expires. A single unchanged poll is not enough: the next node may be labelled a second later.

values.yaml

job:
  # Give a slow-labelling cluster longer (see the timeout warning below)...
  waitForNodesSeconds: 300
  # Require a 30-second quiet period after the last newly eligible node.
  nodeSettleSeconds: 30
  # ...or resolve once and treat "no eligible node" as a no-op, the way a
  # DaemonSet with no matching node quietly installs nothing.
  # waitForNodesSeconds: 0

Raise helm --timeout alongside it

The wait is spent inside the dispatcher hook, and Helm bounds that hook by the release timeout (--timeout, 5 minutes by default) — the same budget that has to cover extracting the artifacts and restarting the CRI runtime on every node. Raising waitForNodesSeconds on its own only trades a dispatcher error naming the selectors it tried for Helm's opaque timed out waiting for the condition, which leaves the release failed and the dispatcher Job orphaned.

The uninstall dispatcher never waits: it has to run to completion on a release that may never have labeled a node.

Bounding a node that never finishes

The dispatcher waits for every node it dispatched to, so one host wedged on something it cannot finish — a CRI restart that never returns, a hung mount — would otherwise hold up the whole rollout until Helm's own timeout fired. Two knobs bound that:

values.yaml

job:
  # Kubernetes fails a per-node Job that runs longer than this, and the
  # dispatcher reports that node as failed and carries on with the others.
  activeDeadlineSeconds: 3600
  # How long a finished per-node Job is kept. The dispatcher reads each Job's
  # result by polling it, so a Job collected before its next poll leaves its node
  # with no result at all - which reads as a failure. The chart refuses anything
  # below 60.
  ttlSecondsAfterFinished: 600

Installing on control-plane nodes

Control-plane nodes are normally tainted, which is what keeps Kata off them in both modes. Reaching them takes a matching toleration — and, if you want only those nodes, a selector as well:

Control-plane nodes onlyControl-plane nodes and workers

values.yaml

nodeSelector:
  node-role.kubernetes.io/control-plane: ""
tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

values.yaml

# No selector: every node is eligible, and the toleration lets the tainted
# control-plane nodes through as well.
tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

Single-node clusters usually need nothing

Single-node distributions (k3s, k0s --single) do not taint their node, and kubeadm clusters used for local testing are typically untainted with kubectl taint nodes --all node-role.kubernetes.io/control-plane-. In both cases the node is selected by default. On older clusters the label and taint key may be node-role.kubernetes.io/master instead.

In job mode, a node skipped for a taint it does not tolerate is named in the dispatcher log along with the taint responsible, so an unexpectedly small rollout is easy to explain:

$ kubectl logs job/kata-deploy-install-dispatcher

skipping node cp-1: it carries the taint node-role.kubernetes.io/control-plane:NoSchedule,
which this install does not tolerate (a DaemonSet would not have been scheduled there either)

If every selected node is skipped this way the dispatcher fails rather than reporting success with nothing done, and its error quotes the toleration to add.

Choosing which nodes are cleaned up on uninstall

Because the cleanup dispatcher resolves nodes live when it runs at helm uninstall (the dispatcher does the lookup, not Helm at render time), the node set is not frozen into the stored release. This means the default cleanup selector can simply be "nodes carrying the katacontainers.io/kata-runtime label" — i.e. exactly the nodes the install touched, regardless of how the install selector has drifted since.

The selector matches the label's key, not the value "true", and that is deliberate. A node is claimed with katacontainers.io/kata-runtime=false before anything is written to it — by the dispatcher in job mode, before it even creates that node's Job — and only promoted to "true" once the node is kata-capable. Uninstall therefore also reaches nodes where the install failed part way through — the ones most likely to have artifacts or CRI configuration left on them. Nothing schedules onto a claimed node in the meantime, since the RuntimeClasses select the exact value "true". For the same reason an uninstall demotes the label to "false" rather than removing it, and removes it only once that node's cleanup Job has succeeded: a cleanup that fails is still a node the next helm uninstall can find. Where several installations share a node, neither the demotion nor the removal happens while another one is still holding it — see How installations keep out of each other's way on a node.

In job mode claiming is a precondition for dispatch: if the labels cannot be written, that node fails before any host mutation begins. This keeps the default cleanup selector complete even when the API server rejects or times out a patch.

Cleanup pods also tolerate every taint, unlike the install's, which carry the tolerations you configured. Where an install may run is your decision; where an uninstall must run is decided by what is already on the node. A NoExecute taint added since the install would otherwise evict the cleanup pod from a node that has Kata on it, and nothing would come back for it.

Override it under job.cleanup (cleanup.nodes, then cleanup.nodeSelector ANDed with cleanup.nodeAffinity, else all nodes):

values.yaml

# Only uninstall from specific nodes:
job:
  cleanup:
    nodes: ["worker-1"]

# Use an explicit selector instead of the kata-runtime label default:
job:
  cleanup:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - { key: node-role.kubernetes.io/control-plane, operator: DoesNotExist }

Uninstall is deliberately independent of install

Cleanup is not narrowed by the top-level nodeSelector/affinity, and tainted nodes are cleaned rather than skipped. It has to be able to revert every node the install ever touched, even one whose labels or taints have changed since — so it targets what was installed, not what is selected now.

See the default values.yaml for the remaining job.* options (e.g. dispatcherImage, parallelism, ttlSecondsAfterFinished, backoffLimit).

SELinux

The install stages run unprivileged, which on an SELinux-enforcing node means they run as container_t — a domain that is denied the host work they exist to do. Writing /opt/kata, writing the CRI drop-in and driving systemd are all refused, and the install fails with AVC denials in the node's audit log.

selinux.enabled fixes that without giving the privilege back:

values.yaml

selinux:
  enabled: true

This loads a Kata-owned policy module on each node and runs each stage in a least-privilege domain of its own. It works in both deployment modes.

domain	used by	may
kata_deploy_check_t	host-check	read only, plus ask systemd whether the CRI is running
kata_deploy_artifacts_t	artifacts, remove-artifacts	write /opt/kata, the snapshotter unit, /etc/modules-load.d
kata_deploy_cri_t	cri, revert-cri	write the CRI configuration, manage the snapshotter unit, restart the CRI
kata_deploy_node_binaries_t	node-binaries-install, node-binaries-remove	write /usr/local/bin
kata_deploy_t	the DaemonSet's single container	all of the above except /usr/local/bin

job mode runs each stage in its own container, so each gets the narrowest domain its own work needs: artifacts cannot reach the CRI configuration, and neither it nor cri can touch /usr/local/bin. cri does reach /opt/kata, because revert-cri takes the nydus binaries back out of it. daemonset mode runs the whole install in one container, so it needs the union — minus kata_deploy_node_binaries_t, since nodeBinaries requires job mode.

The containers that stage nodeBinaries get no domain of their own. They are the only ones running images Kata does not build, and they write nothing but a pod-local emptyDir, so plain container_t is both enough for them and the right blast radius.

Enabling it on a cluster that is not entirely SELinux

Nodes with SELinux disabled are unaffected: the policy stage exits without doing anything, and the runtime ignores the requested domains. So the value is safe to set once for a mixed cluster rather than per node pool.

What it needs, and what it changes on the node

Loading a policy module is itself privileged, so the chart adds one privileged one-shot container, selinux-policy, that runs the node's own semodule. It is first in the pipeline and exits before any confined stage starts, which is what buys every host-working stage after it staying unprivileged. It runs in the cleanup pipeline too, because those stages are confined as well and a node whose module went missing could otherwise never be uninstalled.

It is not the only privileged container in job mode: load-kernel-modules was already one, and stays one. In daemonset mode it is the only one.

The nodes need policycoreutils (for semodule) installed, unless their policy already defines the domains — the stage checks before it complains, so a node whose policy is managed elsewhere needs nothing from Kata. The module is shipped as source CIL and compiled by the node's own secilc at load time, so one artifact works across selinux-policy versions and distributions.

Nothing is relabelled. The module defines no filecon rules, so /opt/kata keeps the labels it has today (usr_t, inherited from /opt) and the Kata runtime sees exactly what it saw before. The scope is the installer only.

Moving the install directory or the CRI configuration

env.installationPrefix and containerd.configDir can put either directory anywhere on the node, where it carries a label the shipped module knows nothing about. So the stage reads the label each one actually has and, when it is not one the module already grants, loads a one-line module of its own — kata-deploy-install-<type>, kata-deploy-cri-config-<type> — adding that type to the rules. Still no relabelling: the node's labels decide, not Kata.

Several releases on one node

The policy store is the node's, not the release's, so every kata-deploy on a node shares one module. It is installed at a priority carrying the policy's own revision, and semodule keeps the highest, so an older image installing or uninstalling its own release cannot take a newer release's rules away.

That priority is also what removal has to name:

Bash

semodule --list-modules=full | grep kata-deploy
semodule -X <priority> -r kata-deploy

The module is left loaded on uninstall

This matches how host kernel modules are already handled: helm uninstall drops the modules-load.d file but never unloads a module. Remove the modules listed above by hand once no release on the node needs them.

Off by default, because it mutates the node's policy store — something a cluster that manages its own SELinux policy may prefer to do itself. The module is at tools/packaging/kata-deploy/selinux/kata-deploy.cil.

Examples

The shims that need a specific containerd snapshotter or specific hardware are deliberately absent from values.yaml (see shims). Each of them ships instead as a ready-made values file — a profile — that you pass to helm with -f/--values, and that brings along the snapshotter setup, CRI settings and defaultShim its shims need.

All profiles default to job mode and can install their runtimes using either deployment mode. To use a DaemonSet for a new release, pass --set deploymentMode=daemonset after the profile's -f argument. Prepare the required host binaries first (for EROFS, mkfs.erofs from erofs-utils 1.8.2 or newer), and leave nodeBinaries empty. The optional nodeBinaries block in the NVIDIA profiles requires Job mode. See nodeBinaries and Deployment Modes for the preparation and lifecycle differences.

The profiles are packaged with the chart, so you can get a copy to read and edit without cloning the repository:

Bash

helm pull oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION --untar
ls kata-deploy/try-kata-*.values.yaml

Pick the profile that matches your nodes — don't stack two

A profile is a complete values file, not a fragment: it starts from shims.disableAll: true, enables its own shims, and sets its own snapshotter.setup. Passing two of them lets the later -f win key by key, which leaves a mixture neither profile describes — quite possibly shims mapped to a snapshotter that was never set up. To serve genuinely different node pools from one cluster, install one release per pool, each with its own nodeSelector and env.multiInstallSuffix (see Multiple Kata installations on the Same Node).

Each is published as a release asset and -f takes a URL, so helm fetches the file itself. Replace VERSION in both the flag and the URL, or use /releases/latest/download/<file> for the newest release.

Releases up to and including 4.1.0

Those releases do not carry the presets as assets. Fetch the desired file from its tag instead, replacing <file> with the preset filename: https://raw.githubusercontent.com/kata-containers/kata-containers/refs/tags/VERSION/tools/packaging/kata-deploy/helm-chart/kata-deploy/<file>

try-kata-tee.values.yaml

Confidential computing: the TEE (Trusted Execution Environment) shims, with the nydus snapshotter and guest pull they need to keep the image off the host.

Bash

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f https://github.com/kata-containers/kata-containers/releases/download/VERSION/try-kata-tee.values.yaml

Includes:

• qemu-snp and qemu-snp-runtime-rs - AMD SEV-SNP (amd64)
• qemu-tdx and qemu-tdx-runtime-rs - Intel TDX (amd64)
• qemu-se and qemu-se-runtime-rs - IBM Secure Execution for Linux (SEL) (s390x)
• qemu-coco-dev - Confidential Containers development (amd64, s390x)
• qemu-coco-dev-runtime-rs - Confidential Containers development Rust runtime (amd64, arm64, s390x)

The TEE RuntimeClasses also take part in TEE key advertisement and get NFD-based node selectors when NFD is around, so confidential pods are only scheduled where the hardware is.

try-kata-nvidia-cpu.values.yaml

The NVIDIA CPU-only shims: the NVIDIA guest userspace without GPU passthrough. They run with shared_fs=none, so the profile sets up the erofs snapshotter with memory-backed rw layers and dm-verity, and installs using the job deployment mode (no always-on DaemonSet on the node).

Bash

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f https://github.com/kata-containers/kata-containers/releases/download/VERSION/try-kata-nvidia-cpu.values.yaml

Includes:

• qemu-nvidia-cpu - NVIDIA base image without GPU passthrough (amd64, arm64)
• qemu-nvidia-cpu-runtime-rs - NVIDIA base image without GPU passthrough, using the Rust runtime (amd64, arm64)

erofs needs containerd 2.2.0 or newer

The erofs snapshotter is containerd's own, and the profile configures it through the auto-imported conf.d drop-in mechanism. Both arrived in containerd 2.2.0, which is the minimum kata-deploy enforces for erofs.

try-kata-nvidia-gpu.values.yaml

The NVIDIA GPU shims, including their confidential variants. The profile defaults to job deployment mode; all six runtimes can also be installed with daemonset, with the node preparation described above. It sets up both snapshotters the shim set needs: erofs for the plain GPU handlers, nydus for the TEE ones.

Bash

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f https://github.com/kata-containers/kata-containers/releases/download/VERSION/try-kata-nvidia-gpu.values.yaml

Includes:

• qemu-nvidia-gpu and qemu-nvidia-gpu-runtime-rs - GPU passthrough (amd64)
• qemu-nvidia-gpu-snp and qemu-nvidia-gpu-snp-runtime-rs - GPU with AMD SEV-SNP (amd64)
• qemu-nvidia-gpu-tdx and qemu-nvidia-gpu-tdx-runtime-rs - GPU with Intel TDX (amd64)

The RuntimeClasses select on nvidia.com/cc.ready.state, the label the NVIDIA GPU Operator sets, so a node serves either the plain or the confidential handlers depending on the mode its GPUs are in. See NVIDIA GPU passthrough and Kata QEMU for the full setup.

try-kata-fc.values.yaml

The Firecracker shim. Firecracker does not implement filesystem sharing, so images have to reach the guest as block devices — containerd's devmapper snapshotter.

Bash

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f https://github.com/kata-containers/kata-containers/releases/download/VERSION/try-kata-fc.values.yaml

Includes:

• fc - Firecracker (amd64)

devmapper is the one snapshotter kata-deploy will not set up for you

nydus and erofs it can configure; devmapper needs a thin-pool on the host, which is a storage decision only you can make. Set it up on every node before installing this profile — ctr plugins ls | grep devmapper must report ok — otherwise the kata-fc RuntimeClass exists but every pod using it fails to start. See How to use Kata Containers with Firecracker.

try-kata-remote.values.yaml

The remote hypervisor shim, better known as peer pods. It starts no VM on the node: sandbox creation is handed to a provider running elsewhere — in practice Confidential Containers' cloud-api-adaptor, which creates the VM in your cloud. Images are pulled inside that pod VM, hence nydus and guest pull.

Bash

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f https://github.com/kata-containers/kata-containers/releases/download/VERSION/try-kata-remote.values.yaml

Includes:

• remote - remote hypervisor / peer pods (amd64, ppc64le, s390x)

The provider is not part of this chart

This profile installs the shim and its kata-remote RuntimeClass, nothing more. The provider, its cloud credentials and the pod VM image belong to the cloud-api-adaptor deployment and have to be in place first, or the RuntimeClass will exist with nothing behind it.

nodeSelector

We can deploy Kata only to specific nodes using nodeSelector

Bash

# First, label the nodes where you want kata-containers to be installed
$ kubectl label nodes worker-node-1 kata-containers=enabled
$ kubectl label nodes worker-node-2 kata-containers=enabled

# Then install the chart with `nodeSelector`
$ helm install kata-deploy \
  --set nodeSelector.kata-containers="enabled" \
  "${CHART}" --version  "${VERSION}"

You can also use a values file:

YAML

nodeSelector:
  kata-containers: "enabled"
  node-type: "worker"

Bash

$ helm install kata-deploy -f values.yaml "${CHART}" --version "${VERSION}"

Applies to both deployment modes

nodeSelector selects nodes in daemonset and job mode alike — see Choosing which nodes get a Job for how job mode applies it. Note that leaving it empty does not mean "every node": nodes carrying a taint you do not tolerate are still excluded, which is what keeps Kata off control-plane nodes.

Combining it with affinity.nodeAffinity

Both may be set at once and are ANDed, exactly as they would be on any pod spec: a node has to satisfy the nodeSelector labels and match one of the nodeSelectorTerms. Job mode reproduces this by folding the nodeSelector equalities into every term, so the two modes select the same nodes.

tolerations

Tolerations decide which tainted nodes Kata may be installed on. Selecting a node is not enough on its own: if the node carries a NoSchedule or NoExecute taint that your tolerations do not cover, it is excluded — in daemonset mode by the scheduler, and in job mode by the dispatcher applying the same rule.

That default is what keeps Kata off control-plane nodes, and off nodes your platform has reserved for something else, without you having to exclude them by label.

values.yaml

tolerations:
  - key: "reservedFor"
    operator: "Exists"
    effect: "NoSchedule"

To install on control-plane nodes, see Installing on control-plane nodes.

Cordoned and pressured nodes are still installed on

Kubernetes gives every DaemonSet pod an implicit set of tolerations so that node conditions — unschedulable (a cordoned node), plus disk, memory and PID pressure — never stop a node agent from running. Job mode applies the same implicit set, so cordoning a node does not make it silently miss the install in either mode.

podLabels

You can add extra labels to the kata-deploy DaemonSet pods. These are applied in addition to the name: kata-deploy label that the chart uses internally. The chart ignores a name key in podLabels and always sets the required selector label itself.

Bash

$ helm install kata-deploy \
  --set podLabels.team=platform \
  "${CHART}" --version "${VERSION}"

Or via a values file:

YAML

podLabels:
  team: platform

podAnnotations

You can add annotations to the kata-deploy DaemonSet pods for whatever your environment needs: Prometheus scrape hints, policy markers, revision metadata, and so on. The chart does not set any annotations on the kata-deploy DaemonSet pod template by default, so nothing is reserved or overwritten unless you set podAnnotations yourself.

Bash

$ helm install kata-deploy \
  --set-string podAnnotations.prometheus\.io/scrape=false \
  "${CHART}" --version "${VERSION}"

Or via a values file:

YAML

podAnnotations:
  prometheus.io/scrape: "false"
  example.com/owner: platform-team

affinity

Use affinity when you need more granular scheduling controls than nodeSelector alone. nodeSelector only matches exact key/value pairs on a node; affinity gives you matchExpressions (e.g. In, NotIn) and rules about other pods on the same node. For example, you might want kata-deploy on nodes reserved for your platform team but not on nodes that run the GPU operator.

nodeAffinity selects nodes in both modes; pod affinity only in daemonset mode

affinity.nodeAffinity is the match-expression form of nodeSelector and, like it, decides which nodes get Kata in both deployment modes. The two can be combined, and are ANDed. podAffinity/podAntiAffinity describe how to schedule a pod, so they only apply in daemonset mode; job mode pins each per-node Job to a node by name and has no scheduling decision to influence. See Choosing which nodes get a Job for which nodeAffinity constructs job mode can and cannot express.

Bash

# First, label the nodes where kata-deploy should run
$ kubectl label nodes worker-node-1 node.cloud/reserved=platform-team
$ kubectl label nodes worker-node-2 node.cloud/reserved=platform-team

# Then install the chart with affinity
$ helm install kata-deploy -f values.yaml "${CHART}" --version "${VERSION}"

YAML

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node.cloud/reserved
              operator: In
              values:
                - platform-team
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - gpu-operator
        topologyKey: kubernetes.io/hostname

When node-feature-discovery.enabled=true, the chart also merges in a nodeAffinity rule that requires hardware virtualization support.

How NFD and user nodeAffinity are combined

Within a single nodeSelectorTerm, matchExpressions and matchFields are AND-ed (all must match). Across nodeSelectorTerms, terms are OR-ed (any one term may match).

If you set affinity.nodeAffinity yourself, only your required nodeSelectorTerms participate in the merge. They are combined with the built-in virtualization terms as (NFD OR-group) AND (user OR-group): each built-in term is AND-ed with each of your required terms. Multiple user required terms remain OR among themselves. NFD virtualization requirements cannot be bypassed by user affinity.

If you set nodeAffinity without requiredDuringSchedulingIgnoredDuringExecution, the built-in NFD required terms are still applied. Other affinity fields (podAffinity, podAntiAffinity, and preferredDuringSchedulingIgnoredDuringExecution) are passed through unchanged.

Multiple Kata installations on the Same Node

For debugging, testing and other use-case it is possible to deploy multiple versions of Kata on the very same node. All the needed artifacts are getting the multiInstallSuffix appended to distinguish each installation. BEWARE that one needs at least containerd-2.0 since this version has drop-in conf support which is a prerequisite for the multiInstallSuffix to work properly.

Bash

$ helm install kata-deploy-cicd       \
  -n kata-deploy-cicd                 \
  --set env.multiInstallSuffix=cicd   \
  --set env.debug=true                \
  "${CHART}" --version  "${VERSION}"

Note: runtimeClasses are automatically created by Helm (via runtimeClasses.enabled=true, which is the default).

Now verify the installation by examining the runtimeClasses:

Bash

$ kubectl get runtimeClasses
NAME                                 HANDLER                              AGE
kata-clh-azure-cicd                  kata-clh-azure-cicd                  77s
kata-clh-azure-runtime-rs-cicd       kata-clh-azure-runtime-rs-cicd       77s
kata-clh-cicd                        kata-clh-cicd                        77s
kata-clh-runtime-rs-cicd             kata-clh-runtime-rs-cicd             77s
kata-dragonball-cicd                 kata-dragonball-cicd                 77s
kata-openvmm-azure-runtime-rs-cicd   kata-openvmm-azure-runtime-rs-cicd   77s
kata-qemu-cicd                       kata-qemu-cicd                       77s
kata-qemu-runtime-rs-cicd            kata-qemu-runtime-rs-cicd            77s

That is the default shim set (shims). Installing with one of the profiles instead gives you that profile's classes: adding -f try-kata-tee.values.yaml here would list the TEE set — kata-qemu-snp-cicd, kata-qemu-tdx-cicd and the rest — in place of the classes above.

How installations keep out of each other's way on a node

One thing is deliberately not suffixed: the node label katacontainers.io/kata-runtime, which every installation's RuntimeClasses select. It says "this node can run Kata", not "this installation is here", so it cannot be used to work out whether removing it is safe.

Each installation therefore also marks the nodes it holds with a label of its own, named after its suffix — kata-deploy.katacontainers.io/cicd for the example above, and kata-deploy.katacontainers.io/default for an installation that sets no suffix. The value follows the shared label's: false while the installation is being put in place, true once the node can run its workloads.

Bash

$ kubectl get node worker-1 -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep kata
"katacontainers.io/kata-runtime":"true"
"kata-deploy.katacontainers.io/default":"true"
"kata-deploy.katacontainers.io/cicd":"true"

Every installation's RuntimeClasses select both labels — the shared one and its own mark, including kata-deploy.katacontainers.io/default when no suffix was set. The default installation can still share a node with a suffixed release, so it needs the same gate. Removing one mark stops that installation's workloads being sent there immediately, even when the shared label stays for another one.

An uninstall removes its own mark from every node it reaches, and the shared label only from the nodes where no other mark is left. So uninstalling kata-deploy-cicd above leaves worker-1 running Kata for the other installation, while a node that only ever had cicd on it loses the label and stops being a Kata node. Cleanup still restarts the shared CRI runtime after removing this installation's configuration: otherwise the live runtime could keep advertising a handler whose binary has just been deleted. The other installation's configuration remains and is reloaded by that restart. Host-mutating stages from different releases take the same node-local lock, so concurrent installs or uninstalls cannot race their configuration edits and restarts.

A mark reading false does not count as another installation serving Kata: if the last true mark goes and only a half-finished installation is left, the shared label is set back to false rather than removed. Nothing is scheduled on the strength of it, and the installation that is still working on the node keeps a label its own uninstall can find.

Upgrade every installation before uninstalling any

Nodes installed by a version that did not write marks carry none, and an uninstall cannot attribute such a node to one release. Run helm upgrade on every installation without changing its node selection before uninstalling or narrowing one of them, so each release has first marked the nodes it owns.

env.multiInstallSuffix is immutable for the life of a Helm release. The chart stores the first value under a suffix-independent ConfigMap name and rejects an upgrade that changes it: the install directory, handlers, resource names, and node marker all derive from the suffix, so changing it in place would create a second installation while forgetting how to remove the first.

For containerd, a suffixed installation also requires drop-in configuration support. Older whole-file configuration has one shared backup; uninstalling one release would restore a file that predates every release and erase the surviving handlers. Installation therefore fails before modifying CRI configuration when that unsafe combination is detected.

Both containerd drop-ins carry the suffix, the generated one as kata-deploy-<suffix>.toml and the containerd.userDropIn content as zz-kata-deploy-user-<suffix>.toml, for the same reason: each release removes its own files on uninstall, and a name two releases shared would be taken from whichever of them is still deployed.

With whole-file configuration, uninstall restores the backup taken at install time. A node whose containerd had no configuration file at all gets one written for it, and uninstall deletes that file again. Any other whole-file configuration is left untouched, with a warning in the log: without a backup or a record of having written the file, kata-deploy cannot tell its own configuration from an administrator's.

On the first upgrade from a chart version that predates that state ConfigMap, the chart verifies the previous identity from the existing mode-specific resource whose name derives from the suffix. If it cannot find one, the upgrade stops and the error names the ConfigMap data that can be seeded explicitly; guessing would be the unsafe choice.

The same state makes deploymentMode immutable and rejects values other than daemonset and job. An in-place mode switch can strand nodes owned by the old controller, and recording a new mode before its hook succeeds can also make a rollback reject the previous mode. Uninstall the existing mode cleanly before installing the other one.

In job mode this means an uninstall may visit a node that only ever belonged to another installation — the default cleanup selection is the shared label, which is by definition not yours alone — and find nothing of its own to remove. That is deliberately the safe direction: it reaches every node it might have touched. If you would rather it visited only its own nodes, select on its mark:

values.yaml

job:
  cleanup:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kata-deploy.katacontainers.io/cicd
                operator: Exists

Both deployment modes keep the same books

The marks are the same labels in either mode — written by the dispatcher in job mode and by the pod on the node in daemonset mode — so an uninstall in one mode does see an installation running in the other. Mixing modes across installations is not a combination we test, though: give every installation the same deploymentMode.

RuntimeClass Node Selectors for TEE Shims

Manual configuration: Any nodeSelector you set under shims.<shim>.runtimeClass.nodeSelector is always applied to that shim's RuntimeClass, whether or not NFD is present. Use this when you want to pin TEE workloads to specific nodes (e.g. without NFD, or with custom labels).

Auto-inject when NFD is present: If you do not set a runtimeClass.nodeSelector for a TEE shim, the chart can automatically inject NFD-based labels when NFD is detected in the cluster (deployed by this chart with node-feature-discovery.enabled=true or found externally):

• AMD SEV-SNP shims: amd.feature.node.kubernetes.io/snp: "true"
• Intel TDX shims: intel.feature.node.kubernetes.io/tdx: "true"
• IBM Secure Execution for Linux (SEL) shims (s390x): feature.node.kubernetes.io/cpu-security.se.enabled: "true"

The chart detects NFD once and uses that result for both these selectors and TEE key advertisement. Detection succeeds when this chart enables NFD, an external NFD workload is found by label, or the NFD API is registered. Auto-injection only runs when NFD is detected and the shim has no manual runtimeClass.nodeSelector.

Detection needs a live cluster

helm template cannot detect external NFD. Set a manual runtimeClass.nodeSelector and nodeFeatureRules.create: true when rendering manifests without cluster access.

TEE key advertisement

A confidential VM consumes a hardware key slot — an encrypted-state ID on AMD SEV-SNP, a key ID on Intel TDX — and a node has a small, fixed number of them. The chart models that as an extended resource so the scheduler stops placing confidential pods on a node whose slots are all taken, instead of letting the VM fail to start:

• a NodeFeatureRule tells node-feature-discovery to advertise the per-node counts as sev-snp.amd.com/esids and tdx.intel.com/keys
• the matching RuntimeClasses request one of them in overhead.podFixed, so every pod that uses the class consumes a slot

Both halves are one switch, because requesting a resource nothing advertises would leave confidential pods Pending forever:

values.yaml

nodeFeatureRules:
  create: auto   # auto | true | false

auto renders them when NFD is in the picture, by the same detection the RuntimeClass node selectors use: installed by this chart (node-feature-discovery.enabled=true), already present in the cluster, or its CRD is registered. true and false decide outright.

false turns off both halves: no rule, and no confidential RuntimeClass asks for a TEE key. It is the escape hatch for a cluster that wants no part of this — not the way to keep the requests while managing the rule elsewhere. If you create an equivalent rule yourself, leave this at auto (or set it to true) so the requests are still rendered, and delete the chart's rule if it duplicates yours.

Naming, and a rule you may have to delete by hand

The chart's rule is named kata-tee-keys, suffixed with env.multiInstallSuffix when that is set. A rule called amd64-tee-keys belongs to a release that has not been upgraded yet, and can be deleted once every release has been.