Kata Containers 4.2.0

4.2.0 is the second monthly snapshot after 4.0.0, and follows 4.1.0. It has no single headline the way 4.1.0 had OpenVMM and the tarball split. Instead, three efforts that began earlier each move a long way forward.

The guest userspace moves to Ubuntu 26.04 LTS, across every image, every initrd and every architecture. kata-deploy keeps taking work off the node's owner: it now loads the kernel modules the enabled features need, can put a missing host binary on a node from a container image, can confine its own install stages under SELinux, and can keep installing on nodes that join after the release was applied. And rootless runtime-rs QEMU, which in 4.1.0 could boot only a plain sandbox, now handles GPU passthrough, block-backed rootfs images and Cloud Hypervisor.

Read the breaking changes before upgrading

Cloud Hypervisor on MSHV hosts no longer disables nested virtualization by itself, on either runtime; it is now configured rather than detected. The two NVIDIA Helm presets install with the DaemonSet again, so those nodes must carry the EROFS and device-mapper kernel modules themselves. disable_block_device_use is gone from runtime-rs. And job mode's dispatcher image moved to a new registry, which matters if you mirror it.

New features

Guest images move to Ubuntu 26.04 LTS

Every Ubuntu-based guest image and initrd moves from noble (24.04 LTS) to resolute (26.04 LTS) — all four architectures and every variant: plain, confidential, and the whole NVIDIA family including the GPU extension (#13584). The x86_64 cbl-mariner 3.0 image is unaffected.

The NVIDIA side moves with it, onto the ubuntu2604 CUDA and tools repositories, container toolkit 1.19.1-1, and the datacenter-gpu-manager-4-core DCGM package rather than the meta-package whose CUDA flavours are hundreds of megabytes the guest never uses.

Action required for anything built against the guest

Local rootfs builds need OS_VERSION=resolute rather than OS_VERSION=noble, and the guest's glibc and library versions change. An out-of-tree guest extension has to be rebuilt: the confidential guest extension ships no libc of its own, so it must match the rootfs ABI, which versions.yaml now records as .externals.coco-guest-components.variant: ubuntu26.04.

kata-deploy prepares the node

Two changes hand kata-deploy work that used to be left to whoever built the node.

Kernel modules are loaded in job mode. A node with no kvm, vhost or erofs module loaded cannot run Kata, and noticing that was nobody's job. Each per-node install Job now starts with one short-lived privileged init container, holding no Kubernetes credentials, that loads what the enabled runtimes and snapshotters need (#13712). It mounts the host root read-only, because only the host's own modprobe matches the running kernel, and it fails the node before anything on it is changed if a required module will not load. What loaded is recorded under the host's /etc/modules-load.d so it survives a reboot; uninstall drops that file but never unloads a module, since module state is host-global and may be in use by something else.

The module plan follows the architecture and the enabled features rather than being a fixed list:

Modules	When	Required?
kvm, kvm_intel/kvm_amd	x86_64	Attempted; the stage requires only that one of /dev/kvm or /dev/mshv ends up present
kvm, vhost	aarch64, riscv64	Required
vhost, vhost_net	always	Required
vhost_vsock	QEMU among the enabled shims	Required
erofs, loop	erofs in snapshotter.setup	Required
dm_mod, dm_verity	snapshotter.erofsDmverity	Required

x86_64 KVM is attempted rather than demanded on purpose: Kata's VMMs run on either KVM or MSHV and choose at run time, a Hyper-V root partition cannot load KVM and does not need to, and kvm loads happily on a machine with virtualization disabled in firmware while creating no device. What matters is that a usable device node exists at the end.

DaemonSet mode is deliberately untouched — it has no privileged container, and loading modules from a pod that lives forever is exactly the always-on privileged agent that job mode set out to avoid.

Missing host binaries can come from an image. A node packaging erofs-utils older than 1.8.2 cannot convert layers with the options kata-deploy gives containerd's EROFS differ, and updating that node's distribution packages is not always on offer. Nothing about the problem is specific to erofs-utils, so a generic nodeBinaries map lets anything a node lacks come from a container image (#13762):

values.yaml

deploymentMode: job
nodeBinaries:
  erofs-utils:
    image: quay.io/kata-containers/erofs-utils:1.9.3
    binaries: [mkfs.erofs, dump.erofs, fsck.erofs]
    pullPolicy: IfNotPresent  # optional, defaults to imagePullPolicy

Only the listed binaries are taken, so an image built on a distribution does not drop the rest of its userland on the node. They are installed into /usr/local/bin, ahead of /usr/bin in containerd's PATH, under the same per-node lock as every other host mutation, and removed again on uninstall. Marker files are named after env.multiInstallSuffix, so side-by-side installations own separate sets. A file already in place that kata-deploy did not install fails the install rather than being replaced, and two entries offering the same binary name fail it too — either way nothing on the node is touched, so it keeps the working set it had.

Requirements

nodeBinaries requires deploymentMode: job, because only the staged pipeline puts the binaries in place before the host check looks for them. The image needs a POSIX shell, cp, and each listed binary statically built in one of /usr/local/bin, /usr/local/sbin, /usr/bin, /usr/sbin, /bin, /sbin or its root. An image with no manifest for a node's architecture stalls that node's install on the pull, so cover every architecture you deploy to — the published erofs-utils image is amd64 and arm64 only.

The same work fixed an EROFS host check that rejected a perfectly new-enough erofs-utils on every arm64 node, and another that could validate a mkfs.erofs containerd would never run on a node carrying two versions.

SELinux confinement for the installer

4.1.0 dropped privileged: true from kata-deploy. On an SELinux-enforcing node that leaves the install stages running as container_t, denied every host operation they exist to perform — writing /opt/kata, writing the containerd drop-in, driving systemd (#13751).

Setting selinux.enabled loads a Kata-owned policy module that gives each stage a least-privilege domain instead (#13777):

Domain	Stages	Allowed to
kata_deploy_check_t	host-check	Read-only inspection, plus D-Bus unit status
kata_deploy_artifacts_t	artifacts, remove-artifacts	Write /opt/kata, the nydus unit, /etc/modules-load.d
kata_deploy_cri_t	cri, revert-cri	Write the CRI configuration and restart it
kata_deploy_node_binaries_t	node-binaries-install, -remove	Write /usr/local/bin
kata_deploy_t	DaemonSet mode's single container	The union of the above

The module is loaded using the node's own semodule, so one policy file works across selinux-policy versions, and at priority 400 + policy-revision, so an older image cannot overwrite a newer release's rules. It defines no file contexts and relabels nothing, so /opt/kata keeps its inherited usr_t; a moved env.installationPrefix or containerd.configDir is handled by reading the label those directories actually carry.

The policy modules are left loaded on uninstall

Enabling this mutates the node's policy store, and cleanup does not undo it. List what was loaded with semodule --list-modules=full | grep kata-deploy and remove each entry with semodule -X <priority> -r <name>.

Note

selinux.enabled defaults to false. Nodes with SELinux disabled are unaffected, so it is safe to set across a mixed cluster. Enforcing nodes need policycoreutils for semodule, unless their policy already defines the domains. Enabling it reintroduces one short-lived privileged container per pipeline, in exchange for every stage after it staying unprivileged.

Job mode: keeping up with a fleet that grows

In job mode the dispatcher runs as a release hook, so it reaches exactly the nodes that existed when the release was applied. A node joining later matches the same selectors and still gets nothing until somebody re-runs helm upgrade — and on a cluster with an autoscaler nobody reliably does, so a pod asking for a Kata RuntimeClass waits while a node that could have run it takes other work.

A CronJob now does that upgrade's work on a schedule, running the same dispatcher against the same selectors and per-node templates (#13718):

values.yaml

job:
  reconcile:
    enabled: false
    schedule: "*/15 * * * *"

A tick stands aside while a release rollout is in flight and leaves out nodes that already show a finished install, so a tick over a settled fleet is a couple of API calls and no pods at all. Uninstall deletes the schedule before it starts reverting nodes, because Helm deletes the CronJob only once the pre-delete hooks are done, and a tick firing in between would reinstall what cleanup had just removed.

Additive only, and off by default

A recurring privileged rollout is something to ask for explicitly. It also only ever adds nodes: one that fell out of the selection is usually a label gone wrong somewhere, so it is left for the next upgrade rather than dismantled on a timer with nobody watching.

Job mode says which node failed, and why

A failed rollout used to say only that a node had failed. The actual reason sat in that node's Job pod, which job.ttlSecondsAfterFinished deletes minutes later — usually before anyone goes looking. It now reaches you three ways (#13796): helm install prints the dispatcher's per-node summary rather than only saying that a hook Job failed; each stage writes why it is giving up to /dev/termination-log, so kubectl describe pod shows it; and each node's result is recorded as an Event against the Node and as a label on the Node itself, which survives the pods:

Bash

kubectl get nodes -l kata-deploy-job-dispatcher/result=failed

Two RBAC grants come with it

The chart now always requests pods: ["list"] in the release namespace and events: ["create"] in the default namespace for job-mode installs, rather than gating them behind a value — a rollout whose reporting RBAC was refused is one nobody can debug. A Node is cluster-scoped, and default is the only namespace the API server accepts an Event about it in, so clusters with restrictive policy there may need to allow that Role and RoleBinding.

kata-monitor over TLS

kata-monitor served /metrics, /sandboxes and its debug endpoints over plain HTTP, so anything that could reach the pod could read them and operators had no way to enforce encryption or a site cryptographic policy (#13257).

Setting monitor.tls.secretName mounts that Secret — expecting keys cert.pem and key.pem — read-only at /etc/kata-monitor/certs, serves HTTPS, and flips the pod annotation prometheus.io/scheme from http to https while adding prometheus.io/port, so Prometheus discovery follows:

values.yaml

monitor:
  tls:
    secretName: kata-monitor-certs
    minVersion: VersionTLS13

The binary's own flags are --tls-cert-file, --tls-key-file, --tls-min-version (VersionTLS12 or VersionTLS13) and --tls-cipher-suites. Cipher suites Go considers insecure are rejected outright, and --tls-cipher-suites together with --tls-min-version=VersionTLS13 is refused, because crypto/tls does not allow configuring TLS 1.3 suites.

Note

TLS is opt-in; HTTP remains the default both in the chart and in the reference DaemonSet at docs/how-to/data/kata-monitor-daemonset.yml. The Secret has to exist before the DaemonSet rolls out. When TLS is enabled and minVersion is unset, the effective minimum is VersionTLS12.

Rootless QEMU reaches GPU passthrough and block-backed images

4.1.0 made rootless QEMU work with shared_fs = "none". What it could not do was reach a device or a disk, because an unprivileged QEMU cannot traverse root-owned parent directories even when the privileged shim is authorized for the leaf. The shim now opens those files itself, while it still has privileges, and hands QEMU the descriptors — for VFIO devices through the iommufd and vfio-pci fd properties (#13548), and for block sources through -add-fd and QMP add-fd (#13546). That covers GPU passthrough, CoCo init-data disks, disk-backed emptyDir volumes, raw block devices and multi-layer EROFS rootfs images.

Because the descriptor path is used for privileged QEMU too, existing privileged deployments get it transparently. Two side effects are worth knowing: host device permissions, ownership and ACLs no longer need relaxing for the VMM user, and containerd snapshotter directories stay restrictive, since QEMU gets read-only access to the selected layers rather than traversal rights over the snapshot tree.

One thing did need a policy decision. After dropping privileges QEMU loses the exemption that keeps VFIO DMA mappings out of RLIMIT_MEMLOCK, and containerd commonly supplies an 8 MiB hard limit, so IOMMUFD setup failed with ENOMEM. A QEMU child carrying VFIO devices now has both its soft and hard RLIMIT_MEMLOCK raised to the configured guest memory plus ten percent before privileges are dropped, and the launch fails if that cannot be done.

Cloud Hypervisor joins in as well (#13628). Its backend made the credential-dropping calls but discarded their errors, so it could reach exec() with inherited supplementary groups, the wrong primary GID, or even the root shim's UID; it now shares QEMU's fail-closed helper. And both runtimes recorded a rootless storage root without creating it — QEMU got away with it because VM templating created the directory early, while Cloud Hypervisor waits until StartVM(), by which point filesystem sharing had created the shared parent as root and the unprivileged VMM could not traverse it to bind its API socket.

Still out of rootless scope

Legacy VFIO group nodes and device hotplug after launch keep the direct host-path route; the descriptor transport covers modern IOMMUFD and VFIO character devices known before QEMU starts. The Go runtime has neither transport. Remaining rootless work is tracked in #13424.

Dragonball: an entropy source, and a much faster restore

virtio-rng. entropy_source and its annotation already existed in kata-types, but the Dragonball backend never consumed it, so Dragonball guests had no virtio RNG device — and a short-lived microVM can stall in early boot on getrandom, systemd or TLS waiting for entropy. Dragonball now implements the virtio-rng device and inserts it during cold start when entropy_source is set, so the guest gets a /dev/hwrng (#13380):

configuration-dragonball.toml

entropy_source = "/dev/urandom"
valid_entropy_sources = ["/dev/urandom", "/dev/random", ""]

The device is cold-plug only; insertion after boot is rejected. Two safety behaviours are worth knowing: host-provided entropy sits outside the confidential guest trust boundary, so the device is skipped for confidential guests; and the configured source is probed before insertion, so a path that resolves but cannot be opened by the shim logs a warning instead of stopping every sandbox on the node from starting.

Restore got 2.3x faster. Snapshot state already carries guest memory, the finalized kernel command line and every vCPU register, yet Dragonball's restore path ran a full cold-boot initialization first and then overwrote it. A restore-specific path now rebuilds only the host-side devices and KVM vCPU objects (#13758). In the author's A/B measurements — one vCPU, 2 GiB, image-backed template, no shared filesystem — restore-to-agent-ready fell from 26.472 ms to 11.449 ms, a 56.8% reduction. A template-restored VM also no longer needs a configured or readable kernel file. Incomplete snapshot state is now rejected before vCPUs launch, and snapshot memory regions are validated one-to-one against the rebuilt guest address space, so an incomplete snapshot cannot leave expected guest RAM silently zero-filled.

NVIDIA: GPU metrics from inside the guest

Asking a GPU guest for metrics used to mean hand-editing a config.d drop-in on every node. A new per-shim block turns it on instead (#13719):

values.yaml

shims:
  qemu-nvidia-gpu:
    nvrc:
      enableDCGM: true

kata-deploy turns that into nvrc.dcgm=on on that shim's guest kernel command line, in the same config.d/30-kernel-params.toml drop-in that already carries the proxy and debug settings — so turning it off removes the drop-in again. The block is named after NVRC, the init process of the NVIDIA guest images, because every key in it becomes an nvrc.* guest parameter. The exporter then serves on port 9400 of the sandbox, which is the pod IP:

Bash

curl "http://$(kubectl get pod my-gpu-pod -o jsonpath='{.status.podIP}'):9400/metrics"

The same change fixes a guest-image regression: dcgm-exporter had been dropped from the images when DCGM started coming from the CUDA repository, and because dcgm is in the default stack, NVRC panicked as PID 1 the moment it tried to start it.

Note

enableDCGM defaults to false, because it costs guest memory and an extra process in every sandbox on that shim. Values files written before this setting keep rendering — a shim with no nvrc block reads as off. Naming a non-GPU shim is refused at install time rather than silently doing nothing.

QEMU seccomp sandbox on the NVIDIA runtime classes

QEMU's seccomp_sandbox parameter existed but no runtime actually used it: the NVIDIA runtime-rs configurations inherited the generic template variable, which defaults to empty. The four NVIDIA QEMU classes — CPU, GPU, GPU+SNP, GPU+TDX — now ship on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny (#13673). Every other QEMU configuration is unchanged.

This may cost performance

Enabling QEMU's seccomp sandbox has a runtime cost; enabling /proc/sys/net/core/bpf_jit_enable on the host reduces it. If that is a concern, comment out seccomp_sandbox in the configuration file used by the relevant runtime class, or override DEFSECCOMPSANDBOXPARAM_NV at build time. A workload that trips one of the denied syscall classes would newly fail.

Confidential Containers

Generated policies pin the guest-pull storage count. rules.rego counted image_guest_pull storages only in order to subtract them from the expected total, so it accepted a CreateContainer request carrying none of them, or several. A new rule requires exactly one when the cluster is in guest-pull mode and none otherwise, so a missing, duplicated or unexpected guest-pull storage is denied (#13689). One storage per container is exactly what a correctly behaving runtime sends, so no action is needed beyond regenerating your policies.

Initdata works on Cloud Hypervisor. The Cloud Hypervisor driver assumed block devices always arrive after boot, so a pre-boot request was dropped with ignoring early add device request. That assumption does not hold for the initdata image, which the sandbox prepares alongside its other resources, so the cc_init_data annotation was silently inert on clh-runtime-rs (#12720). Pre-boot block devices are now queued and attached after the VM rootfs (#13668).

Trustee and the guest components also move to their current revisions (#13830).

Helm values presets are published as release assets

The documented try-kata-* examples passed their values files to Helm by bare filename, which only works with the chart or the repository already unpacked locally. try-kata-tee.values.yaml, try-kata-nvidia-cpu.values.yaml and try-kata-nvidia-gpu.values.yaml are now attached to the GitHub release under their plain names, so -f can take a URL and Helm fetches the file itself (#13780):

Bash

helm install kata-deploy "${CHART}" --version "${VERSION}" \
  -f "https://github.com/kata-containers/kata-containers/releases/download/${VERSION}/try-kata-tee.values.yaml"

Both NVIDIA presets now also enable the bundled node-feature-discovery (#13781). They always depended on it — the GPU preset's RuntimeClass selectors match amd.feature.node.kubernetes.io/snp and intel.feature.node.kubernetes.io/tdx, which NFD advertises, and both rely on it to keep Kata off nodes without virtualization support — but neither switched it on. If you already run your own NFD, set node-feature-discovery.enabled=false.

Breaking changes

Nested virtualization is configured, not detected

Cloud Hypervisor enables nested virtualization unconditionally, and both runtimes worked around that by probing for /dev/mshv and forcing nested vCPUs off whenever they found it. 4.1.0 added disable_nested_virtualization to the runtime-rs Cloud Hypervisor configurations (#13577), but the runtime override stayed — so on an MSHV host the configuration option was a lie.

The override is now gone from both runtimes, and the configuration is the single source of truth. runtime-rs drops its /dev/mshv probe (#13815), and the Go runtime drops its own and gains the option, persisted across sandbox saves so it survives a shim restart (#13805).

Configuration	disable_nested_virtualization
configuration-clh.toml	false
configuration-clh-azure.toml	true

Action required on MSHV hosts

Nested virtualization is no longer disabled for you. If you run Cloud Hypervisor on MSHV, either select the Azure profile — which already sets it — or set disable_nested_virtualization = true in your own configuration. This applies to anyone on a stock non-Azure configuration-clh.toml, and to anyone carrying a custom or pre-4.1.0 configuration forward.

disable_block_device_use is removed from runtime-rs

As documented, disable_block_device_use gates only the rootfs block device. The Go runtime defaults it to true and honours that definition — it is how GHSA-5fc8-gg7w-3g5c was remediated there. In runtime-rs it defaulted to false, so that advisory never applied, and setting it to true was simply broken: it disabled all block devices rather than just the rootfs, and cleared the block device driver without falling back to 9p or virtio-fs.

Rather than fix a knob nobody could use, it is removed from runtime-rs entirely, along with its annotation (#13682). The Go runtime is untouched and keeps the option with its true default.

What actually needs action

A leftover disable_block_device_use = ... in a hand-maintained runtime-rs configuration is silently ignored, so no configuration migration is strictly required. The annotation is the sharper edge: if you had added disable_block_device_use to a runtime-rs enable_annotations list, a pod carrying that annotation is now rejected with Invalid annotation type io.katacontainers.config.hypervisor.disable_block_device_use. It was not in any shipped enable_annotations list, so a default configuration already ignored it with a warning.

The NVIDIA Helm presets install with the DaemonSet again

Job mode became the chart default earlier in this cycle, and both NVIDIA presets asked for it explicitly. NVIDIA QA then found bugs in job mode, so rather than ship the newer install model with known issues the default reverts to deploymentMode: daemonset and the switch is postponed to 4.3.0 (#13814). Job mode stays fully supported and is one deploymentMode: job away; nothing about either mode's behaviour changed.

Relative to 4.1.0 the chart-wide default is unchanged — it was daemonset there too. The presets are what changed: try-kata-nvidia-cpu.values.yaml and try-kata-nvidia-gpu.values.yaml went from job to daemonset, pinned by name so each profile still records the model it was validated against.

Action required if you use either NVIDIA preset

Only the per-node Jobs load host kernel modules, and both presets run EROFS with dm-verity. Those nodes must now carry erofs, loop, dm_mod and dm_verity themselves, persistently across reboots. The install's host check fails the node naming the feature it could not find, rather than configuring a snapshotter it cannot run. The commented-out nodeBinaries erofs-utils block in both presets now also requires setting deploymentMode: job back; leaving it on daemonset fails the render rather than deploying something that cannot work.

Note

If you never set deploymentMode yourself, nothing changes for you between 4.1.0 and 4.2.0.

The job-mode dispatcher moved to a new registry

Job mode's dispatcher — the component that enumerates selected nodes live and fans out one node-pinned install Job per node — is now the independently released k8s-job-dispatcher project rather than an in-tree crate (#13685, #13720). Because it releases on its own cadence, its version is pinned in the chart instead of following the chart appVersion.

	4.1.0	4.2.0
job.dispatcherImage.reference	quay.io/kata-containers/kata-deploy-job-dispatcher	ghcr.io/kata-containers/k8s-job-dispatcher
job.dispatcherImage.tag	"" (chart appVersion)	0.3.0

Action required if you mirror the dispatcher image

quay.io/kata-containers/kata-deploy-job-dispatcher is no longer built or published. Air-gapped and mirrored registries need ghcr.io/kata-containers/k8s-job-dispatcher instead. If you pinned job.dispatcherImage.reference to the old image, repoint it.

Note

Node and Job tracking labels are deliberately unchanged — the dispatcher is still passed --tracking-label-prefix=kata-deploy-job-dispatcher, because that prefix names the labels stamped on Jobs and nodes rather than the binary, and renaming it would orphan the labels on already-installed clusters. Existing installs are not disturbed.

Newer toolchains are needed to build from source

Rust moves to 1.96 (#13687) and Go to 1.26.7, the latter because the 1.25 series is now out of support (#13688). Nothing changes for consumers of the release artefacts.

Notable fixes

Security

No security advisory was fixed in this release.

SetIPTables ran the wrong binary for IPv6. Two path constants in the agent named binaries that were not the intended ones: USR_IPTABLES_RESTORE was /usr/sbin/iptables-store, a typo for a file that does not exist, and USR_IP6TABLES_RESTORE was /usr/sbin/ip6tables-save. Because the agent selects its binary by testing whether the /usr/sbin variant exists, the IPv6 test succeeded on any guest shipping ip6tables-save and the restore path ran the save binary; the IPv4 misspelling never existed, so selection fell back to /sbin and the fault stayed latent. Both are fixed (#13789).

Three supply-chain gaps were closed too. The virtiofsd release archive is now checksum-verified before extraction, and a mismatch fails the build rather than falling back to a source build (#13657). pcilibs-rs is back on kata-containers/pcilibs-rs, having been pointed at an individual contributor's fork by mistake in the 4.1.0 cycle, and no longer fetches the PCI IDs database over the network at build time (#13723, #13794). And the libseccomp source tarball published as a release asset is a real tarball again (#13684) — see Known issues for what that means for 4.0.0 and 4.1.0.

Dependency updates carrying security fixes are listed under Dependencies.

Runtime

• A VM that fails to boot now leaves a console log behind. Both runtimes started their guest console watcher only after hypervisor startup and QMP initialization had succeeded — so exactly when the guest kernel log matters most, nothing was captured. In runtime-rs the same code had two further faults: it connected to the console socket exactly once, losing a race QEMU can easily win, and it propagated that failure with ?, aborting an otherwise healthy launch because console logging could not be set up. The watcher now starts before hypervisor startup and retries until the endpoint appears, failure is non-fatal, and on a startup failure the VM is stopped before the watcher is, so the final output is still consumed (#13437 for runtime-rs, #13819 for the Go runtime).
• Dragonball snapshot restore no longer wedges the guest's timer. A microVM restored from a snapshot could resume with a local APIC timer that never fired again; a few percent of restores wedged this way, usually stalling the agent inside CreateSandbox until the shim gave up. The guest arms its timer in TSC-deadline mode, which spans two ioctls, and KVM silently discards the deadline write while the LVT timer is still in another mode — then zeroes the stored deadline as KVM_SET_LAPIC switches mode. Restoring MSRs first therefore left nothing armed, and only an unrelated interrupt forcing Linux to reprogram the clock event device could rescue the guest, which is why idle vCPUs stayed wedged and busy ones did not. The ordering is fixed, and a deadline saved as zero is rearmed so an already-expired timer fires immediately rather than never (#13798).
• Three runtime-rs bugs under CRI-O. Restoring CRI-O CI found all of them (#13559). kubectl exec into a container declaring tty: true lost its output: CRI-O derives an exec's process from the container's, so the runtime asked the guest for a terminal while choosing which streams to copy from the container's flag rather than the request's, and the agent then served stdout and stderr readers from a single pty that raced for the same bytes until both ended in EIO. An OOM kill was reported to the kubelet as Reason=Error rather than OOMKilled, because a container's memory limit is enforced by the guest kernel so the host cgroup stays quiet, and CRI-O does not read the shim's TaskOOM event for VM runtimes. And under static_sandbox_resource_mgmt a CRI-O sandbox fell back to one vCPU and default_memory, because only containerd's io.kubernetes.cri.sandbox-* annotations were read; CRI-O's own is now read too. That last one matters most for QEMU, whose runtime-rs configuration enables static_sandbox_resource_mgmt on every architecture.
• csi-kata-directvolume no longer panics on a failed CreateVolume. The non-SPDK error branch logged its failure by dereferencing the volume it was not given, so a request over the driver's --max-volume-size, or a failed MkdirAll, took the driver down with a nil-pointer dereference: csi-provisioner never received a clean error, the kubelet killed the container, and the driver entered CrashLoopBackOff, blocking all kata-direct provisioning on the node. The real gRPC error is now returned (#13444).

Agent

• Containers using multi-layer EROFS can be removed again. Container destruction removes the bundle before storage cleanup runs, and the multi-layer EROFS handler placed its backing mounts — the mounted ext4 upper and the read-only EROFS lower filesystems — inside the bundle, so bundle removal tried to delete through still-mounted filesystems. Every remove_container RPC for an EROFS container failed, with errors such as Read-only file system (os error 30), and on a writable layer it could delete data. A failing container destroy then blocked cleanup of the sandbox's other resources. Backing mounts now live under /run/kata-containers/erofs-multi-layer/<cid>, outside the bundle (#13756, fixing #13757).

Deployment

• The kubelet root directory follows the Kubernetes flavour. The drop-in that relocates it was keyed off the node's CRI runtime being k0s' own containerd, so a k0s node running CRI-O got nothing and MicroK8s never got one at all. It is now keyed off k8sDistribution — k0s to /var/lib/k0s/kubelet, microk8s to /var/snap/microk8s/common/var/lib/kubelet, with k3s, RKE2 and vanilla needing nothing. The drop-in also relocates pod_resource_api_sock, whose build-time default does not exist on those flavours, which had been silently leaving GPU cold plug to fall back to CDI annotations (#13559).

Set k8sDistribution correctly even if you override containerd.configDir

The chart used to withhold K8S_DISTRIBUTION when containerd.configDir was set, and now always passes it. Getting the flavour wrong on k0s or MicroK8s stays quiet: ConfigMap and Secret volume updates stop reaching running guests, and GPU cold plug silently falls back to CDI annotations.

NVIDIA

• Multi-GPU B200 and B300 nodes work. NVRC moves to v0.1.7 for a multi-GPU fix on those systems (#13835). Before it, a confidential pod requesting three GPUs started fine while four hung in ContainerCreating until timeout, the logs repeating IOMMU_IOAS_MAP failed: Bad address, PCI BAR? (#13270).
• DCGM is pinned to a known version. It was installed unpinned from the NVIDIA repositories, which only ever offer the newest, so a package landing upstream silently re-based the guest — which duly happened and broke the arm64 image build. DCGM and its exporter are now pinned (#13832).
• Guest images can no longer disagree with the kernel that signed them. The NVIDIA kernel is built with CONFIG_MODULE_SIG_FORCE=y against a key generated fresh on every build, but the image cache key did not track it — so a published image could be paired with a superseded kernel. The guest booted, NVRC's first modprobe was rejected with "Key was rejected by service", and because NVRC is PID 1 the guest panicked, leaving the pod in ContainerCreating with nothing indicating which half was stale (#13728).

Dependencies

Component	Old	New	PR
Guest rootfs and initrd (Ubuntu)	noble (24.04 LTS)	resolute (26.04 LTS)	#13584
NVRC	v0.1.5	v0.1.7	#13719, #13835
NVIDIA container toolkit	1.18.1-1	1.19.1-1	#13584
DCGM / dcgm-exporter	unpinned	1:4.7.0-1 / 4.8.3.002-1	#13832
Go	1.25.13	1.26.7	#13688
Rust	1.95	1.96	#13687
containerd/containerd (client)	1.7.33	1.7.35	#13813
containerd/cgroups	v1.1.0	/v3 v3.0.5	#13655
google.golang.org/grpc	1.82.1	1.83.2	#13774, #13807
rtnetlink	0.14.0	0.23.0	#13785
klauspost/compress	1.18.0	1.18.7	#13750
OpenTelemetry	1.43.0	1.44.0	#13750

The containerd client bump covers CVE-2026-53495, and the gRPC bumps bring an HTTP/2 frame-flood mitigation and a set of xDS authorization fixes. The Go module bumps in #13750 remediate GO-2026-5841, GO-2026-5158, GO-2026-5942 and GO-2026-5970. The netlink bump drops Kata's last dependency on the unmaintained paste crate (RUSTSEC-2024-0436).

Note

The containerd bump updates the client library the shim links, not the containerd daemon on your hosts. Patch that separately. It also moves containerd/cgroups to /v3, which changes import paths for anyone vendoring src/runtime.

Known issues

NUMA and vCPU pinning remain disabled by default on the NVIDIA configurations. DEFENABLENUMA_NV and DEFENABLEVCPUPINNING_NV for the Go runtime, and DEFENABLEVCPUSPINNING_NV for runtime-rs, still default to false because of #13539, which remains open. Re-enable them explicitly if you need them for GPU workload placement, and track the issue.

Intel TDX stays on edk2-stable202511. Unchanged from 4.1.0: an update to edk2-stable202605 made TDX guests consistently time out connecting to the agent over vsock and was reverted before that release. The other OVMF variants are on edk2-stable202508.

The libseccomp source assets published with 4.0.0 and 4.1.0 are broken. They contain a GitHub 404 page rather than the tarball, because the release script built the download URL wrongly and did not fail on the error. That is fixed from 4.2.0 onwards, but already-published assets are immutable. These assets exist to satisfy the LGPL-2.1 written offer of source for the libseccomp that kata-agent is statically linked against; if you need the 4.0.0 or 4.1.0 source, fetch it from the libseccomp project directly (#13684).

Testing and support coverage

Some CI changes in this release move what the project gates on. They do not remove code, but they do change what upstream verifies for you.

Change	Effect	PR
CRI-O tests restored	A k0s + QEMU amd64 workflow is back, after the old CRI-O infrastructure was retired. Three bats suites are skipped under CRI-O, for its long CreateContainer retry loop and its inability to report stats for VM-based sandboxes	#13559
SEV-SNP TEE tests are required again	qemu-snp had stopped gating merges while AMD's Key Distribution Service was unreliable; with the VCEK now served from an offline store on the runner, both qemu-snp and qemu-snp-runtime-rs gate merges	#13806, #13729
OpenVMM Kubernetes tests are required	Two AKS legs and the arm64 OpenVMM build now gate merges, after ten consecutive passing runs	#13675
NVIDIA GPU tests are not required	Serious NVIDIA runner infrastructure problems; the amd64 GPU suite still runs but no longer blocks a merge	#13737
ppc64le build checks moved to nightly	They no longer gate pull requests, though the agent and runtime components do still run their KVM- and vsock-dependent tests on a self-hosted runner	#13771, #13792
genpolicy and kata-deploy tests are not required	Both stopped gating merges while stabilising — genpolicy for registry pull flakes, kata-deploy pending the newly added kubeadm flavour becoming the required one. Both still run	#13740, #13701
NVIDIA inference tests moved to open models	vLLM serving Qwen2.5-0.5B-Instruct and all-MiniLM-L6-v2 replaces NVIDIA NIM on pull requests, so GPU inference coverage no longer needs an NGC API key or gated multi-gigabyte pulls. NIM moves to nightly	#13726

New coverage arrived too: two kata-deploy SELinux suites on an enforcing runner, one per deployment mode (#13804); the rootless suite generalized from QEMU to any VMM, now covering Cloud Hypervisor, EROFS-backed handlers, a disk-backed emptyDir and runtime-rs CoCo-dev (#13628); the kata-monitor suites run against HTTPS as well as HTTP (#13257); and an in-guest DCGM test (#13719).

Documentation

Helm configuration gained sections for the NVIDIA guest settings, nodeBinaries, scheduled reconcile, finding out why a node failed, and SELinux. The EROFS snapshotter guide now opens by saying that kata-deploy does all of its setup for you, and what the node still has to bring in each deployment mode. Installation points at systemd-detect-virt for telling bare metal from a nested guest (#13765). And the storage design document now states that disable_block_device_use is a Go-runtime setting, direct block device mounting being the runtime-rs default and the flag unsupported there.

For contributors, the contributing guide moved from the kata-containers/community repository into this one as CONTRIBUTING.md, so developer guidance lives with the code, and it now carries the project's AI policy and an initial set of best practices (#13722, #13727).

Upgrading

1. If you run Cloud Hypervisor on an MSHV host, set disable_nested_virtualization = true or switch to the Azure profile before upgrading. Neither runtime disables nesting for you any more.
2. If you use try-kata-nvidia-cpu.values.yaml or try-kata-nvidia-gpu.values.yaml, those presets now install with the DaemonSet, so make sure erofs, loop, dm_mod and dm_verity are loaded on every selected node and persist across reboots — or set deploymentMode: job explicitly to keep the previous behaviour.
3. If you mirror container images, add ghcr.io/kata-containers/k8s-job-dispatcher and drop quay.io/kata-containers/kata-deploy-job-dispatcher.
4. On k0s or MicroK8s, set k8sDistribution correctly even if you override containerd.configDir — it now also selects the kubelet root directory, and getting it wrong fails quietly.
5. If you build a custom guest rootfs, move to OS_VERSION=resolute. Anything built against the 24.04 guest ABI needs rebuilding.
6. If you allow disable_block_device_use in a runtime-rs enable_annotations list, remove it: pods carrying that annotation are now rejected.
7. If you run NVIDIA GPU workloads and are sensitive to VMM overhead, measure them — QEMU now runs inside a seccomp sandbox on those runtime classes.
8. If you generate agent policies, regenerate them so they pick up the guest-pull storage cardinality rule.
9. Review the full breaking changes and known issues sections above.

For general guidance, see Upgrading and the installation instructions.