Kata Containers 4.1.0

4.1.0 is the first monthly snapshot after 4.0.0, the major release that made the Rust runtime (runtime-rs) the default. Where 4.0.0 was about switching runtimes, 4.1.0 is about building on that switch: a new hypervisor backend, a new network model, VM templates on Cloud Hypervisor and Dragonball, and a debug story for guests that have become too small to debug the old way.

Alongside that, two long-running efforts land visible pieces in this release. The release payload is now split by runtime, so you download the shim and guest images you actually use, and kata-deploy gives up a large share of the host privileges it used to demand — including, in job mode, the Kubernetes credentials it used to hand to every privileged per-node pod.

runtime-rs also reaches the last architecture that was still Go-only, so it is now the default shim everywhere.

Read the breaking changes before upgrading

Two changes affect nearly everyone: the release tarball has been split in two, and kata-deploy's job mode no longer accepts its own node-selection keys. kata-deploy also pins each installation's identity, so an upgrade that changes env.multiInstallSuffix or deploymentMode is now refused. All three fail loudly rather than silently, but all three need action.

Confidential Containers: this release fixes a high-severity policy issue

GHSA-fmg6-v47x-52wr let a malicious operator mount attacker-chosen guest paths over specific locations in a policy-protected guest. It affects 4.0.0 and earlier. See Security.

New features

OpenVMM

runtime-rs gained a backend for OpenVMM, the cross-platform VMM written in Rust, talking to the OpenVMM process over ttrpc with protocol buffer definitions vendored from the OpenVMM repository (#13346).

It ships as the openvmm-azure-runtime-rs shim with its own configuration-openvmm-azure-runtime-rs.toml, is tested on Azure Linux with MSHV, and advertises discard support so block-plain emptyDir volumes work (#13545).

The shim is deployable on x86_64 only: kata-deploy lists amd64 as its single supported architecture. CI also builds OpenVMM for arm64 as of #13616, which is groundwork rather than a supported target for now.

L3 forwarding network model

A new internetworking_model = "l3forwarding" for runtime-rs forwards traffic between the plugin-provided interface and the guest tap using L3 routing and proxy ARP instead of tc filters (#13281). It exists to give Istio Ambient-style service meshes with node proxies a clean integration point, though it does not implement that integration itself.

Experimental

IPv4 only, and it supports a single network interface per network namespace.

Cloud Hypervisor: VM templates and a nested-virtualization switch

VM template factory support arrived for Cloud Hypervisor under runtime-rs, reaching parity with the Go runtime: pause, resume, save and restore through the Cloud Hypervisor API, memory zones and backing files, and restore in the VM startup path (#13387). The same change fixed runtime-rs leaving the source VM behind after creating a template, and aligned the Cloud Hypervisor API models with v51.1, which also fixes VM creation and resize at the supported 256-vCPU limit.

Note

VM templating with EROFS currently handles single-layer container images only; multi-layer images are not yet supported.

Cloud Hypervisor enables nested virtualization unconditionally, which some hosts cannot accommodate. runtime-rs now exposes disable_nested_virtualization for the Cloud Hypervisor configurations, generated to preserve today's behaviour and settable at build or run time (#13577). The existing MSHV override, which always disables nested vCPUs, is unchanged.

Dragonball: VM templates on x86_64

Dragonball gained checkpoint and restore support, wired into the runtime-rs Dragonball backend, so a sandbox VM can be dumped once as a template and later pods restored from it instead of cold booting (#13366).

The template is captured at a clean quiesce point: after the sandbox VM has booted and its agent has answered, but before any guest sandbox is created. The source VM keeps running and the restored VM receives its first create_sandbox from the pod that consumes it. In the author's measurements a template start took 0.25s against 1.16s for a cold start.

Configure it through the Dragonball configuration, the same way the other VMMs express templating:

configuration-dbs.toml

boot_to_be_template = false
boot_from_template = true
memory_path = "/run/vc/vm/template/memory"
device_state_path = "/run/vc/vm/template/state"

runtime-rs on ppc64le

ppc64le was the last architecture still served only by the Go runtime. It now builds and ships runtime-rs: the shim is in the ppc64le static tarball, CI runs the qemu-runtime-rs Kubernetes tests there, and the Helm chart lists ppc64le among the supported architectures (#12552). qemu-runtime-rs is now the default shim on ppc64le, as it already was everywhere else (#13581).

ppc64le boots an initrd rather than a rootfs image for the Rust runtime, and k8s-cpu-ns is skipped there because its 128 MB memory limit is below the 512 MB QEMU needs on Power.

Rootless QEMU without filesystem sharing

Rootless QEMU now works with shared_fs = "none", and the combined rootless and seccomp-sandbox test coverage extends to the configurations that qualify — including kata-qemu-coco-dev and kata-qemu-coco-dev-runtime-rs, which use nydus guest pull without enabling a TEE (#13429).

Two problems stood in the way. runtime-rs bound the QMP socket below the per-sandbox jailer root before anything created that directory, which virtio-fs had been creating as a side effect of its own pre-start setup; the jailer root is now created before the QEMU command line is built. And both runtimes were sending CopyFile destinations — the path /etc/resolv.conf travels when there is no shared filesystem — with the host VMM user's XDG_RUNTIME_DIR prefix, which is not a valid path inside the guest.

That second problem was a symptom of a broader confusion between guest-local and host-dependent paths, which is now resolved: sandbox shm, memory-backed volumes, hugepage mounts and runtime-rs guest-pull rootfs paths stay under /run/kata-containers in the guest instead of inheriting the host's /run/user/<uid> convention (#13518). Host paths and guest mount points whose layout is genuinely coupled to host-side filesystem sharing keep the rootless prefix.

Note

The EROFS snapshotter, the confidential RuntimeClasses and device passthrough remain outside rootless scope; that work is tracked in #13424.

NVSwitch passthrough on HGX and DGX

NVSwitches report PCI class 0x0680, which both runtimes were rejecting as a host bridge. That blocked running Fabric Manager in a service VM with access to every NVSwitch on the node.

The Go runtime now filters on an explicit list of permitted class IDs (#13540), and runtime-rs separates the IOMMUFD path — where each /dev/vfio/devices/vfioX character device is exactly one device and there is no group to enumerate — from the legacy IOMMU group path, whose class filter used a subset bit mask that swept in the entire 0x06 base class (#13524). Fabric Manager also now finds its NVSwitch topology files through the composable GPU extension mount (#13471).

Underneath, the PCI enumeration, VFIO and IOMMUFD helpers moved out of kata-sys-util into a standalone pcilibs-rs crate so tools can reuse them without pulling in the hypervisor crate (#13411, #13526).

Debugging small guests: the devkit extension

Guest images keep getting smaller and more locked down, and some — notably the NVIDIA base with its NVRC init — ship no shell at all, so the agent debug console has nothing to exec.

The new devkit guest extension is a self-contained Ubuntu toolbox (strace, ltrace, tcpdump, lsof, iproute2, pciutils, …) packaged as a measured EROFS + dm-verity image and cold-plugged into a sandbox on demand, without rebuilding or modifying the real guest rootfs (#13497). Guest helpers overlay a writable tmpfs on the read-only extension and chroot into the merged tree, so apt install works for anything the image does not already carry, and /real_root exposes the actual guest for inspection. Tools shipped by other mounted extensions are now on PATH inside that chroot, so chroot /real_root nvidia-smi resolves (#13585).

Enable it with the Helm devkit flag, which is only effective together with debug and gives each enabled shim a kata-<shim>-devkit RuntimeClass.

Not for production or confidential deployments

The devkit is a debugging aid, gated behind debug, and deliberately not wired into production or CoCo/TEE deployments.

kata-ctl exec and kata-ctl cp

kata-ctl exec can now run a single command instead of only attaching a terminal, returning the command's output on stdout and exiting with its exit code, which makes it usable from scripts (#13570):

Bash

kata-ctl exec "${sandbox_id}" -- cat /proc/cmdline

A new kata-ctl cp moves files in and out of a guest over the same debug console, with docker cp addressing:

Bash

kata-ctl cp ./tool "${sandbox_id}":/tmp/tool
kata-ctl cp "${sandbox_id}":/var/log ./guest-logs

This matters most with shared_fs = "none" and a read-only measured rootfs, where there is no host directory to drop a file into. Because the transfer needs a shell, tar and base64 in the guest, cp requires the devkit extension and says so up front.

The console, not cp, is the security boundary

The debug console is a root shell, so anyone who can open it can already move files without kata-ctl. What controls access is whether the console exists at all: the agent only serves one when the guest booted with agent.debug_console, and a confidential guest measures that kernel command line.

kata-monitor sees both runtimes

kata-monitor discovered sandboxes only under /run/vc/sbs, the Go runtime's path, which left runtime-rs deployments without metrics. It now watches /run/kata as well, so either runtime — or both on one node — is collected with no runtime-type flag (#13611).

Confidential Containers

IBM Secure Execution moved to the composable base + extension model under runtime-rs, matching SNP and TDX. qemu-se-runtime-rs gets a new boot-image-se-runtime-rs target built with --composable, producing kata-containers-se-runtime-rs.img so it can coexist with the monolithic image; qemu-se on the Go runtime is unchanged (#13529). The s390x vanilla initrd stopped forcing AGENT_INIT=yes, since the extension mount units need the systemd boot flow (#13508).

The CoCo guest components are no longer compiled during a Kata build. Both the monolithic confidential rootfs and the composable extension now consume the artefacts that the guest-components project publishes, pinned under .externals.coco-guest-components in versions.yaml (#13487). Provenance is verified with gh attestation verify and fails the build by default; set VERIFY_COCO_EXTENSION_PROVENANCE=no to bypass it. Verification is skipped on s390x, where gh has no binary.

Slimmer, more targeted QEMU builds

QEMU is now configured with all default features and devices off, enabling only what Kata needs, replacing the long list of --disable-* flags (#10708).

On top of that, a new qemu-no-shared-fs build serves the NVIDIA CPU and GPU runtime-rs classes, which boot a block rootfs with shared_fs = "none", never run confidential and leave virtio-mem off. It drops VIRTIO_MEM, VIRTIO_BALLOON, VIRTIO_9P, VHOST_USER_FS, NVDIMM, the IOMMU and TEE backends, and every UEFI image, legacy BIOS blob and microvm firmware. The two confidential-guest builds get the same trim minus the TDX and SEV-SNP backends (#13520). Device assignment is untouched everywhere: vfio-pci, IOMMUFD and the whole root-port/switch-port/bridge-port topology stay in, and a post-build check fails the build if any device the runtimes can emit is missing.

All flavours install alongside each other and come from the same sources and the same versions.yaml entries. s390x is deliberately out of scope and keeps the generic build.

kata-deploy hardening

kata-deploy used to run as a fully privileged, hostPID pod that climbed into the host's namespaces with nsenter to poke systemd and copy files. It now talks to systemd over a socket, mounts only the host directories it needs, and keeps its own filesystem read-only — no nsenter, no hostPID, no privileged: true, no full host root mount (#13530).

Advertising per-node TEE key counts — SEV-SNP encrypted-state IDs, TDX key slots — used to require cluster-wide write access to NodeFeatureRules, RuntimeClasses and custom resource definitions from a container that was root on the host. Those objects are now rendered from the chart, so the privileges disappear rather than move, and nodeFeatureRules.create: auto reproduces the question the binary used to ask at run time (#13553).

Note

The rendered rule is deliberately not named after the one the binary used to apply, because Helm refuses to adopt an object it did not create and reusing the name would break helm upgrade on clusters that already have it.

In job mode, the dispatcher is the only component holding Kubernetes credentials, and its token is worth root on any node. It can now be confined to nodes you trust via job.dispatcherNodeSelector and job.dispatcherTolerations (#13554):

values.yaml

job:
  dispatcherNodeSelector:
    node-role.kubernetes.io/control-plane: ""
  dispatcherTolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

Node-facing work — claiming nodes, checking readiness and runtime handlers, publishing labels, removing startup taints — moved from the privileged per-node Jobs into the dispatcher (#13604), and with the dispatcher doing all of it, those Jobs no longer carry Kubernetes credentials at all: they run without a ServiceAccount and with token mounting explicitly disabled (#13606).

That closes the most valuable target in a job-mode install. A privileged pod with host mounts on every selected node was also being handed a token that turns one compromised worker into cluster-wide Node access, for API work the host-changing stages never needed to do themselves. Cleanup Jobs now tolerate every taint, so uninstall can still reach a node whose scheduling state changed since install. DaemonSet mode keeps its credentials, because its long-running pod still performs its own node lifecycle operations.

Host mutations are also serialized per node now, which matters once more than one installation exists: separate installations could otherwise edit the same runtime configuration and restart the same CRI service concurrently. Each per-node Job is bound to the selected host's machine identity — so a Node deleted and recreated under the same name while a Job is in flight is detected — and configuration changes and runtime restarts happen as one ordered lifecycle under a node-local lock, including on containerd setups without drop-in support (#13609).

Other kata-deploy and Helm improvements

Change	PR
snapshotter.erofsSnapshotterMode (disk/memory) now drives the generated containerd default_size, and unsupported modes are rejected	#13476
Changes to mounted ConfigMaps (shims.*.dropIn, containerd.userDropIn, custom runtime drop-ins) are hashed into pod-template annotations, so the DaemonSet actually rolls out	#13418
CHART_VALUES_OVERLAY deep-merges a values file into values.yaml at publish time, so a bare helm install of a published chart does the right thing	#13478
The NFD Helm chart is vendored, so runners no longer pull the remote chart repo and hit rate limits	#13613
podLabels reach the per-node Jobs in job mode, matching the DaemonSet	#13549
virtiofsd is no longer shipped for shims that use shared_fs = "none"	#13451

Breaking changes

The release tarball is split by runtime

kata-static shipped both shims and both flavours of guest image, so everyone downloaded the Go runtime and the monolithic images whether or not they used them. The merged archive is now two archives per architecture (#13488):

Tarball	Shim	Guest images	Also carries
kata-static-${VERSION}-${ARCH}.tar.zst	runtime-rs	Composable: rootfs-image plus the CoCo and NVIDIA extensions	Dragonball kernel
kata-go-static-${VERSION}-${ARCH}.tar.zst	Go	Monolithic: rootfs-image-confidential, rootfs-image-nvidia-gpu, …	Firecracker

Both are assembled from the same per-component build artifacts, so this costs no extra build time, and both are now produced on every supported architecture, ppc64le included.

Action required

If you download release assets directly, pick the archive that matches your runtime. Neither archive ships a host kata-agent binary any more — the agent only exists inside the guest images — so anything that consumed it from the tarball needs the standalone agent artifact instead.

kata-deploy job mode node-selection keys are removed

Job mode had its own node-selection knobs, entirely separate from the nodeSelector and affinity the DaemonSet reads. That produced a genuinely dangerous outcome: a values file pinning Kata to a handful of nodes in daemonset mode installed it fleet-wide in job mode, the NFD virtualization requirements were not applied at all, and taints were ignored entirely because the dispatcher pins Jobs with spec.nodeName and bypasses the scheduler.

Node selection is now compiled from the same top-level nodeSelector and affinity.nodeAffinity in both modes, and the dispatcher replicates the scheduler's taint admission (#13555).

Removed	Replacement
job.nodeSelector	top-level nodeSelector
job.nodeAffinity	top-level affinity.nodeAffinity
job.nodeSelectorExpressions	top-level affinity.nodeAffinity
job.cleanup.nodeSelectorExpressions	job.cleanup.nodeAffinity (standard Kubernetes shape)

These keys now fail rendering with a migration hint rather than being silently ignored. job.nodes remains the one node-selection knob specific to job mode, and it is what you want for naming nodes explicitly.

Job-mode node selection compiles to a label-selector query against the API server, so nodeAffinity forms that a label selector cannot express are rejected rather than quietly ignored:

• matchFields, which cannot match on fields — use job.nodes to target nodes by name.
• The numeric Gt and Lt operators.
• An empty required term list, or an empty nodeSelectorTerm. Kubernetes defines both as matching no nodes, which the dispatcher's queries cannot reproduce, and guessing wrong here means installing on every node (#13609). Drop requiredDuringSchedulingIgnoredDuringExecution entirely to select every otherwise eligible node.

Soft affinity is accepted and ignored on purpose — a preference never stops a DaemonSet from landing on a node either.

Note

An empty nodeSelector still does not mean "every node" in either mode: a node carrying a taint the install does not tolerate is still excluded.

kata-deploy installation identity is immutable

The install directory, runtime handlers, resource names and the per-node ownership marker all derive from env.multiInstallSuffix. Changing it in place therefore created a second installation while the release forgot how to remove the first, orphaning the nodes the original owned. The same applies to switching deploymentMode under an existing release.

Both values are now persisted in a <release>-kata-deploy-state ConfigMap and an upgrade that changes either one fails with a message naming the old and new values (#13607). To change either, uninstall the existing installation cleanly and install the new one as a separate release.

Releases installed before this state object existed are recognised from their mode-specific resource names, so an ordinary upgrade that keeps its values needs no action. An upgrade whose previous identity cannot be established is refused, and the error explains how to seed the ConfigMap if you need to assert it yourself.

kata-deploy RuntimeClasses select an ownership label

katacontainers.io/kata-runtime says that a node can run Kata, but not which installation made it so. With more than one installation on a node, uninstalling one release withdrew scheduling from another that was still serving it.

Every installation now also marks its nodes with kata-deploy.katacontainers.io/<suffix> — or kata-deploy.katacontainers.io/default when no suffix is set — and its RuntimeClasses select both that marker and the shared readiness label. Cleanup gives up the shared label only once no installation still claims the node. This applies to built-in and custom RuntimeClasses alike.

Note

Anything that selected nodes by reading a RuntimeClass nodeSelector sees two labels there now instead of one.

Guest debug moved to its own RuntimeClasses

Enabling deployment logging used to change the guest kernel command line, and therefore its measurements, on the normal RuntimeClass. kata-deploy now creates a separate kata-<shim>-debug RuntimeClass per shim and applies the guest debug settings — enable_debug, the agent debug console, agent.log=debug, initcall_debug — only there (#13480).

Use runtimeClassName: kata-<shim>-debug for pods that need guest debugging. kata-<shim> keeps a stable measurement profile even with debug: true.

NVIDIA runtime classes no longer accept annotations

The NVIDIA runtime-rs classes ship with an empty enable_annotations list, so pod annotations cannot alter their configuration by default (#13537). If you were relying on annotations there, allow the specific ones you need explicitly.

genpolicy derives the pause identity from the image

genpolicy treated every image's Config.User as a workload image user and patched the result with hard-coded pause UID and GID values, which does not match how containerd builds a pause identity. It now models the Kubernetes and containerd identity paths separately, retains the complete Config.User group for pause containers, and applies pod fsGroup and supplementalGroups to the pause process when nothing else selects a sandbox user (#13516).

pause_container_image is now the sole source of pause image identity. pause_container_id_policy and the hard-coded pause UID/GID defaults are gone from the shipped settings and the AKS drop-ins.

Deploy the binary and the settings together

This intentionally drops compatibility with custom settings that supply Process.User. A fresh genpolicy binary and a fresh settings file must be deployed as a pair.

The Go host binaries are static by default

The STATIC profile is now the default, so kata-runtime, containerd-shim-kata-v2 and kata-monitor build without cgo and without PIE. The payload consequently runs on musl-only hosts such as Alpine with no downstream rebuild (#13499).

Opt out with STATIC=no in the runtime Makefile or STATIC_RUNTIME=no at the packaging layer, which restores the old cgo + PIE build. s390x and ppc64le keep the dynamic build as their default. On s390x, STATIC=yes forces linkmode=external so the PT_S390_PGSTE program header that KVM_CREATE_VM requires is not dropped.

Dragonball network device managers merged

virtio-net, vhost-net and vhost-user-net were three device managers with three config types and three insert paths behind one InsertNetworkDevice request. They are now a single NetworkDeviceMgr dispatching on a backend enum (#13423), with three deliberate behaviour changes:

• A device with an empty iface_id is refused (MissingIfaceId). The id is the interface name inside the guest, so callers that inserted vhost-user-net devices without one must now supply it.
• Hot-plugging a device whose id is already in use is refused (DeviceIDAlreadyExist) instead of leaving an attached device with no handle to remove it. Before boot, a repeated id still updates the device configuration.
• Tearing a sandbox down no longer stops at the first device that fails to be destroyed; the failure is logged and the remaining devices are still removed.

The vhost-user socket path also became a conflict-checked resource alongside the tap name and guest MAC, so two devices claiming one socket are refused (DuplicatedUdsPath).

kata-deploy validates the Kubernetes distribution

k8sDistribution decides which host directory the chart mounts as /etc/containerd, while kata-deploy independently detects the runtime and picks a configuration file. When the two disagreed, Kata wrote a valid configuration into a directory the runtime never reads, and the install appeared to succeed. The chart's choice is now compared with the detected runtime during the host-check stage, before the node is changed, and a mismatch is an error naming the conflicting value (#13605).

An explicit containerd.configDir remains an escape hatch, since it replaces the chart's directory derivation.

Build system: cross-compilation paths removed

The build system carried a fair amount of cross-compilation machinery that nothing on the Kata side exercised, since every release and CI build runs on a native-arch runner. Those paths are gone and the build system supports native builds only (#13419).

pcilibs left kata-sys-util

PCI enumeration, VFIO helpers and InfiniBand diagnostics moved out of kata-sys-util into the external pcilibs-rs crate (#13411). Out-of-tree consumers of those kata-sys-util APIs need to depend on pcilibs-rs instead.

Notable fixes

Security

Generated policy allowed attacker-chosen guest paths to be mounted

GHSA-fmg6-v47x-52wr, rated high, is fixed in this release. It affects Kata 4.0.0 and earlier.

Applies to Confidential Containers only

This concerns configurations that use genpolicy to protect the guest from the host. Kata sandboxing, where the host is trusted, is not affected.

Two allow_mount rules in rules.rego were more permissive than intended. One ran check_mount against policy mounts whose source was empty, even though that check expects a regular expression there; the other matched any input storage rather than only block-based emptyDir. allow_storage also accepted an image_guest_pull storage without checking where it was mounted.

Together, a malicious operator could craft CreateContainer requests mounting arbitrary guest rootfs paths over /etc/hostname, /etc/hosts, /etc/resolv.conf, /var/run/secrets/azure/tokens, /var/run/secrets/kubernetes.io/serviceaccount, and any volumeMount in the container spec — and provision arbitrary guest or host content into /dev/shm and /dev/termination-log.

Those paths already carry host-provided data and are outside genpolicy's integrity checks, so a careful application should not have trusted them. The risk is an application that treated them as non-sensitive and shareable while the attacker had placed a sensitive file there, or that expected /dev/shm to be secret or exclusively guest-provided.

The policy now requires a non-empty source for the regular-expression rule, restricts the storage-backed rule to blk and scsi drivers with an empty policy source, and pins an image_guest_pull storage to the expected rootfs path. Reported by @AdamKorcz of Ada Logics and fixed by @burgerdev.

Regenerate your policies

The fix lives in rules.rego, so it takes effect when policies are generated with the 4.1.0 genpolicy. If you cannot upgrade yet, the advisory documents using the 4.1.0 rules.rego on its own as a workaround.

The agent's capability and namespace handling is hardened

The agent no longer trusts workload-visible procfs for capability discovery. It read /proc/sys/kernel/cap_last_cap after entering a workload-controlled mount namespace, so a workload that shadowed that file could inflate the capability set an exec received. Reproducing it with guest seccomp disabled expanded CapBnd, CapEff and CapPrm from 00000000a80425fb to 000001ffffffffff, handing the exec CAP_SYS_ADMIN and CAP_SYS_PTRACE.

The agent now enumerates capabilities through the kernel's PR_CAPBSET_READ interface, fails if discovery returns nothing, and verifies that the bounding set after dropping contains only OCI-allowed capabilities (#13615).

The same change fixes the second half of that escalation. Namespaces created for the container init process are pinned with agent-owned file descriptors, and the OCI spec records those agent proc-fd paths, so a later exec joins the pinned namespaces instead of resolving /proc/<init-pid>/ns/* again — which an exited init process or a reused PID could otherwise redirect.

Related, and covered under Agent below: the agent now rejects a working directory that escapes the container root (#13587).

Runtime

• Resource accounting leaked on failed container starts. A container that kept failing — a CrashLoopBackOff, for instance — accumulated its CPU and memory limits in the ResourceManager without ever releasing them, until the sandbox asked the hypervisor for an impossible memory size and either locked up or drove the host out of memory. The tracked resources are now subtracted on create and start failures (#12748).
• Routes derived from links ignored the netmask, exposing a /32 instead of the real prefix (#13633).
• The runtime is placed in the sandbox cgroup synchronously, closing a window where it was not yet accounted for (#13338).
• Shared volume paths are created below the rw path. The host read-only shared directory is a read-only bind mount of the rw one, so creating a mount point directly below it failed for read-only volumes (#13466).
• listxattr is allowed in the Dragonball seccomp filter. The inline virtio-fs backend calls it when handling FUSE LISTXATTR, so the virtiofs thread took a SIGSYS and the shim exited (#13510).
• The remote hypervisor (peer pods) implementation now passes the peer-pods-relevant tests in the openshift-tests-private suite, after fixes spanning CRI-O incompatibility, configuration and unsupported-functionality handling (#13612).
• Rootless QEMU sandbox setup rolls back on failure instead of leaking the temporary VMM user and its /run/user/<uid> directory when sandbox creation fails after rootless setup succeeded — for the Go runtime in #13506 and for runtime-rs, along with a read-only=on fix that let QEMU open the guest image at all as an unprivileged user, in #13428.
• Dragonball on TDX sets VIRTIO_F_ACCESS_PLATFORM on virtio devices (#13094) and handles the KVM_HC_MAP_GPA_RANGE vCPU exit so a guest can change the encryption status of a memory region (#13095).
• open() errors name the file, which makes a missing kernel module much easier to identify (#13477).
• The mem-agent no longer panics at startup. With mem_agent_enable = true, synchronous mem-agent paths called Tokio's blocking lock methods from inside a Tokio runtime, which panics by definition, leaving the workload stuck in container creation. The shared state now uses std::sync::RwLock, and the few asynchronous accessors take their locks through spawn_blocking (#13578).
• hybrid_vsock always attempts at least one connection. It derived its attempt count as reconnect_timeout_ms / dial_timeout_ms, which is zero when the dial timeout is the larger of the two, so a configuration with dial_timeout_ms > reconnect_timeout_ms never dialled at all. The formula now accounts for the initial attempt, which has no preceding sleep (#13648).
• Stopping an already-exited container process is treated as success. The runtime has always checked for ttRPC NOT_FOUND here, on the reasoning that a process it was asked to stop is indeed stopped, but the agent collapsed every error into INTERNAL, so that check never fired — and the runtime's own "process not found" handling was buggy underneath it. Both halves are fixed (#13435).

Agent

• The init process gets its own session instead of inheriting the agent's, which was making CRIU checkpoints fail with "a session leader is outside of its pid namespace" (#13463).
• A working directory outside the container root is rejected. After resolving the OCI-requested cwd, the agent calls getcwd() and refuses to start the process on ENOENT, which Linux reports when the cwd is unreachable from the process root. Without it, one exec helper could be referenced through /proc/<pid>/root by another and retain the guest root as its cwd after dropping capabilities (#13587).
• Dangling symlinks copy correctly, preserving ownership on the link itself via fchownat(AT_SYMLINK_NOFOLLOW) (#13625).
• A double close on pipes in rustjail is fixed by transferring file descriptor ownership explicitly (#13505).
• Watchable mounts work in rootless setups. The watchable tmpfs root is now derived from the supplied storage mount points rather than probed at fixed paths, so the /run/user/<uid> prefix the runtime adds no longer fails container creation (#13515).
• A stats read can no longer wedge the whole agent. stats_container held the global sandbox mutex across a synchronous, blocking cgroup read. Because signal_process and remove_container take the same lock, a stats read that blocked on an exited container's cgroup during teardown serialized every RPC and starved the kill path, leaving pods stuck Terminating while the host reported "ttrpc Receive packet timeout". The cgroup manager is now cloned under the lock and the blocking read runs outside it (#13453).

Rootless and agent policy

Agent policy enforcement does not support rootless mode. Policy evaluation remains fail-closed and happens before watcher setup, so policy-enabled rootless workloads continue to be rejected.

Deployment

• The nydus-snapshotter unit binds to the node's actual CRI unit instead of a hardcoded WantedBy=containerd.service. On k3s, RKE2, k0s and MicroK8s, where containerd runs inside another unit, the snapshotter stayed dead after every reboot and guest-pull pods hung at sandbox creation (#13482).
• Nydus cleanup is idempotent. A partial or retried install could leave artifacts without the systemd service, and the unconditional systemctl disable then turned that into an error that blocked removing the rest (#13603).
• The job-mode CRI stage converges across restart-induced retries. Restarting the CRI runtime tears down the init container doing the restarting on some platforms, notably AKS, until the Job exceeds its backoffLimit. The stage is now idempotent: if the effective config is byte-for-byte unchanged, a previous attempt already applied it and the self-terminating restart is skipped (#13551).
• The job dispatcher image carries its glibc dependencies on ppc64le and s390x, where the shared Rust builder falls back to the glibc target and distroless/static has no dynamic loader, so the dispatcher could not exec at all and job-mode installs never started (#13550).

Guest images and dependencies

• virtiofsd is now v1.14.0, which fixes u64 directory cookies above i64::MAX on NFS v4-backed shared directories, such as those on Amazon's Elastic File System. Those cookies went negative on the way to lseek64(), so ls, find and chown -R failed with "Invalid argument" and returned partial listings (#13479). This covers the external daemon used by QEMU and Cloud Hypervisor; the Dragonball in-process inline-virtio-fs path still needs its own bump.
• The NVIDIA rootfs is around 208 MB smaller in usr/lib. The build selected libraries with a broad libnv* glob that swept in the entire NVIDIA graphics and ray-tracing stack, none of which a headless compute VM ever exercises. It now reads sandboxutils-filelist.json, the driver's own capability-to-file map, which is the same ground truth nvidia-container-toolkit uses and stays correct across driver releases (#13427).
• The GPU extension builds without the monolithic image as a dependency (#13473), and the nvgpu bundle can be built against a local CUDA repository installer (#13455).
• google.golang.org/grpc moved to 1.82.1 in the runtime and the csi-kata-directvolume tool, picking up the HTTP/2 frame-flood mitigation and a set of authorization fixes (#13434, #13521).
• Go moved to 1.25.13, along with a set of core module bumps, remediating eleven advisories (#13656).
• cpu.max works in arm64 confidential guests. The arm64 confidential guest kernel fragment did not set CONFIG_CFS_BANDWIDTH, so cgroup v2 CPU bandwidth control was unavailable and pods broke under the kubelet's cgroup v2 default (#13438).

Known issues

NUMA and vCPU pinning are disabled by default on the NVIDIA configurations. DEFENABLENUMA_NV and DEFENABLEVCPUPINNING_NV for the Go runtime, and DEFENABLEVCPUSPINNING_NV for runtime-rs, now default to false because of #13539, which will not be fixed in this release (#13671). If you were relying on those defaults for GPU workload placement, re-enable them explicitly and track the issue.

Intel TDX stays on edk2-stable202511. An update to edk2-stable202605 (#13631) was reverted before the release (#13678): it made TDX guests consistently time out connecting to the agent over vsock, which surfaced in Nightly CI rather than in the PR checks, because the affected tests are not part of the baremetal-small-tee suite. The other OVMF variants are on edk2-stable202508.

Testing and support coverage

Some CI changes in this release narrow what the project gates on. They do not remove code, but they do change what upstream verifies for you.

Change	Effect	PR
Confidential Containers tests on RKE2, k3s, MicroK8s and k0s were removed	Those distributions are no longer covered by CoCo CI, since the project neither maintains them nor has the reviewers for related PRs. The nydus-snapshotter unit fix in #13482 did land for them.	#13498
Kubernetes tests on arm64 were reduced to qemu-coco-dev-runtime-rs	Only one arm64 runner is available; the NVIDIA CPU Kubernetes tests provide partial coverage	#13467
NVIDIA NIM Operator tests were removed	The NIMService BATS suite and its manifests are gone; the direct NIM pod tests and their KBS and sealed-secret helpers are intentionally retained	#13503
runtime-rs Kubernetes e2e tests are required on s390x	Matches runtime-rs being the default since 4.0	#13582

New coverage arrived too: a baseline for QEMU's rootless and seccomp sandbox controls, verifying a non-root effective UID and GID, active seccomp filtering and NoNewPrivs, and the exact -sandbox argument (#13426); and a test that two containers in a pod can talk over a Unix domain socket on a memory-backed emptyDir (#13619).

Kubernetes tests now run on the plain RuntimeClass for the verdict and only re-run failures on kata-<shim>-debug for journal triage, so guest measurements stay on the non-debug class (#13514).

Documentation

The virtio-fs guide regained its /dev/shm section, considerably expanded: why virtio-fs needs shared guest memory and what backs it per hypervisor, what the default 50% /dev/shm size actually limits, the two failure modes that are invisible to the kubelet (VM start failure and SIGBUS on a running VM), and how to size /dev/shm against Node Allocatable alongside systemReserved, kubeReserved and evictionHard (#12927).

See virtio-fs with Kata for the result.

Upgrading

1. Pick the right release archive. Most users want kata-static-${VERSION}-${ARCH}.tar.zst, which carries runtime-rs and the composable guest images. Take kata-go-static-* only if you still run the Go shim.
2. If you use kata-deploy in job mode, migrate the removed job.node* keys to the top-level nodeSelector and affinity.nodeAffinity before upgrading, and check that your tolerations cover the nodes you expect to install on — taints are now honoured in job mode.
3. Upgrade each kata-deploy release with the env.multiInstallSuffix and deploymentMode it was installed with. To change either, uninstall cleanly first and install the new identity as a separate release.
4. If you enable debug, move pods that need guest debugging to kata-<shim>-debug.
5. If you generate agent policies, deploy the new genpolicy binary and its settings file together, and regenerate your policies so they pick up the rules.rego fix for GHSA-fmg6-v47x-52wr.
6. On ppc64le, note that the default shim is now qemu-runtime-rs. Pin kata-qemu explicitly if you need to stay on the Go runtime.
7. Review the full breaking changes and known issues sections above.

For general guidance, see Upgrading and the installation instructions.