How to Use EROFS Snapshotter with Kata Containers (runtime-rs)

Project Overview

The EROFS snapshotter is a native containerd snapshotter that converts OCI container image layers into EROFS-formatted blobs. When used with Kata Containers runtime-rs, the EROFS snapshotter enables block-level image pass-through to the guest VM, bypassing virtio-fs / 9p entirely. This delivers lower overhead, better performance, and smaller memory footprints compared to traditional shared-filesystem approaches.

Deploying with kata-deploy? It does all of this for you: a kata-deploy deployment in job mode sets all of this up on every node it selects: containerd's EROFS snapshotter and differ configuration, the shims you enable pointed at the snapshotter, and the erofs and dm-verity modules loaded and recorded so they come back after a reboot.

Those settings need erofs-utils 1.8.2 or newer. Where a node packages something older, or nothing at all, nodeBinaries takes it from a container image instead:

YAML

deploymentMode: job
snapshotter:
  setup: ["erofs"]
nodeBinaries:
  erofs-utils:
    image: quay.io/kata-containers/erofs-utils:1.9.3
    binaries: [mkfs.erofs, dump.erofs, fsck.erofs]

try-kata-nvidia-cpu.values.yaml sets the EROFS side of this up and carries that block commented out, to uncomment on nodes needing it. Read on for what any of it means, or to set a host up without kata-deploy.

Quick Start Guide

This section provides a quick overview of the steps to get started with EROFS snapshotter and Kata Containers. For detailed instructions, see the Installation Guide section.

Quick Steps

1. Install erofs-utils: Install erofs-utils (version >= 1.8.2) on your host system
2. Configure containerd: Enable EROFS snapshotter and differ in containerd configuration
3. Configure Kata Containers: Set up runtime-rs with appropriate hypervisor settings
4. Run a container: Use ctr or Kubernetes to run containers with EROFS snapshotter

Prerequisites

Component	Version Requirement
Linux kernel	>= 5.4 (with erofs module, higher recommended)
erofs-utils	>= 1.8.2
containerd	>= 2.2 (with EROFS snapshotter and differ support, higher recommended)
Kata Containers	Latest main branch with runtime-rs
QEMU	>= 5.0 (VMDK flat-extent support and >= 9.0 higher recommended)

Note: When enabling the EROFS snapshotter on a node that is already in use, existing images in the containerd image store may not yet have EROFS snapshots. Since those images are already present, kubelet may skip pulling them again, so containerd can end up converting their layers into EROFS snapshots during CreateContainer rather than during the image pull phase. Configure kubelet's runtimeRequestTimeout to cover the slowest expected EROFS preparation path for the target node and workload. Large images can otherwise exceed kubelet's CRI request deadline before the sandbox starts.

Installation Guide

This section provides detailed step-by-step instructions for installing and configuring EROFS snapshotter with Kata Containers.

Step 1: Install erofs-utils

The configuration below uses fsmerge mkfs_options that require erofs-utils

= 1.8.2.

Bash

# Debian/Ubuntu
$ sudo apt install erofs-utils

# Fedora
$ sudo dnf install erofs-utils

Verify the version:

Bash

$ mkfs.erofs --version
# Should show 1.8.2 or higher

Load the kernel module:

Bash

$ sudo modprobe erofs

When using EROFS dm-verity mode, also load the dm-verity device-mapper target:

Bash

$ sudo modprobe dm-mod
$ sudo modprobe dm-verity

Make these modules load persistently before containerd starts, for example via your distribution's modules-load mechanism. If the required kernel support is missing when containerd starts, enabling the EROFS snapshotter can fail early instead of producing a useful workload-level error.

Step 2: Configure containerd

Enable the EROFS snapshotter and differ

Note: The following settings target containerd v2.3.0. The configured EROFS fsmerge mkfs_options require erofs-utils >= 1.8.2. Always cross-reference with the official documentation for your current versions.

Edit your containerd configuration (typically /etc/containerd/config.toml):

TOML

version = 3
...
  [plugins.'io.containerd.cri.v1.runtime']
    ...
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      ...
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
          runtime_type = 'io.containerd.kata.v2'
          pod_annotations = ["*"]
          container_annotations = ["*"]
          privileged_without_host_devices = false
          sandboxer = 'podsandbox'
  ...

  [plugins.'io.containerd.differ.v1.erofs']
    mkfs_options = ["-T0", "--mkfs-time", "--sort=none"]
    enable_tar_index = false

  [plugins.'io.containerd.service.v1.diff-service']
    default = ['erofs', 'walking']

  [plugins.'io.containerd.snapshotter.v1.erofs']
    default_size = '<SIZE>' # SIZE=6G or 10G or other size
    max_unmerged_layers = 0
    enable_fsverity = true

enable_fsverity = true enables host-side fs-verity handling for EROFS layer blobs when the kernel and backing filesystem support it. This is separate from the dm-verity mode described below: fs-verity protects layer blob files on the host, while dm-verity protects the block devices mounted by the Kata guest.

Containerd treats fs-verity support as best-effort and may skip it if the host kernel or backing filesystem does not support it. If fs-verity protection is required, make sure the filesystem backing containerd's EROFS snapshotter state supports fs-verity. For example, on ext4 this requires the verity filesystem feature on the relevant device:

Bash

$ sudo tune2fs -O verity <device-backing-containerd-state>

For dm-verity, enable metadata generation in the differ and force strict dm-verity use in the snapshotter:

TOML

  [plugins.'io.containerd.differ.v1.erofs']
    enable_dmverity = true

  [plugins.'io.containerd.snapshotter.v1.erofs']
    dmverity_mode = 'on'

Avoid switching a persistent node back and forth between dm-verity and non-dm-verity EROFS modes without cleaning or rebuilding the EROFS snapshotter state. Keep dm-verity consistently enabled or consistently disabled for a given snapshotter state, because existing layers may have been prepared without dm-verity metadata.

When dm-verity is enabled, avoid dmverity_mode = 'auto'. Auto mode does not solve stale layer state, because layers prepared without dm-verity metadata can still be used without dm-verity protection. Strict dmverity_mode = 'on' makes such stale layers fail instead of silently falling back to an unprotected path.

Verify the EROFS plugins are loaded

Check if EROFS module is loaded

Bash

$ lsmod | grep erofs
erofs                 188416  0
netfs                 614400  1 erofs

If not loaded:

Bash

$ sudo modprobe erofs

Restart containerd and check:

Bash

$ sudo systemctl restart containerd
$ sudo ctr plugins ls | grep erofs
io.containerd.snapshotter.v1    erofs    linux/amd64    ok
io.containerd.differ.v1         erofs    linux/amd64    ok

Check containerd snapshotter status

Bash

$ sudo ctr plugins ls | grep erofs
io.containerd.mount-handler.v1            erofs                    linux/amd64    ok
io.containerd.snapshotter.v1              erofs                    linux/amd64    ok
io.containerd.differ.v1                   erofs                    linux/amd64    ok

Both snapshotter and differ should show ok.

Step 3: Configure Kata Containers (runtime-rs)

Edit the Kata configuration file (e.g., configuration-qemu-runtime-rs.toml):

TOML

[hypervisor.qemu]
# shared_fs can be set to "none" since EROFS layers are passed via
# block devices, not via virtio-fs. If you still need virtio-fs for
# other purposes (e.g., file sharing), keep "virtio-fs".
# For pure block-device EROFS mode:
shared_fs = "none"

Note: The shared_fs = "none" setting is for the case where all container images use the EROFS snapshotter. If you have a mixed environment, keep shared_fs = "virtio-fs" so that non-EROFS containers can still use virtio-fs.

Quick Test

Once the installation is complete, you can quickly test with:

Using ctr for example.

Bash

# Pull the image
$ sudo ctr image pull docker.io/library/wordpress:latest

# Run with EROFS snapshotter and Kata runtime-rs
$ sudo ctr run --runtime io.containerd.kata.v2 --snapshotter=erofs --rm -t library/wordpress:latest test001 date
Wed Apr  1 07:10:53 UTC 2026

$ sudo ctr run --runtime io.containerd.kata.v2 --snapshotter=erofs --rm -t wordpress:latest test001 lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
vda    254:0    0   256M  0 disk
`-vda1 254:1    0   253M  0 part
vdb    254:16   0     6G  0 disk
vdc    254:32   0 759.7M  0 disk

Note: Ensure that the containerd CRI configuration maps the kata handler to the Kata runtime with snapshotter = "erofs" as shown in Step 2.

Architecture

The following diagram illustrates the data flow:

Text Only

  Host                                        Guest VM
  ====                                        ========

  containerd                                  kata-agent
      |                                           |
      v                                           v
  EROFS snapshotter                           1. mount ext4 /dev/vdX
      |                                          (writable upper)
      |-- Mount[0]: ext4 rw layer                 |
      |     (block device on host)            2. mount erofs /dev/vdY
      |                                          (read-only lower)
      |-- Mount[1]: erofs layers                  |
      |     source: layer.erofs               3. overlay mount
      |     device=extra1.erofs                  lowerdir=<erofs_mount>
      |     device=extra2.erofs                  upperdir=<ext4_mount>/upper
      |                                          workdir=<ext4_mount>/work
      v                                           |
  runtime-rs                                      v
      |                                       container rootfs
      |-- single erofs: attach as Raw            ready
      |-- multi erofs:  generate VMDK
      |     descriptor + attach as Vmdk
      |
      v
  QEMU (virtio-blk)
      |-- /dev/vdX: ext4 rw layer
      |-- /dev/vdY: erofs layer(s)

VMDK flat-extent descriptor (multi-layer case)

VMDK Descriptor Format (twoGbMaxExtentFlat) The descriptor follows the VMware Virtual Disk Format specification:

• Header: # Disk DescriptorFile marker and version info
• Extent descriptions: RW <sectors> FLAT "<filename>" <offset>
• sectors: number of 512-byte sectors for this extent
• filename: absolute path to the backing file
• offset: starting sector offset within the file (0-based)
• DDB (Disk Data Base): virtual hardware and geometry metadata

Files larger than 2GB are automatically split into multiple extents (MAX_2GB_EXTENT_SECTORS per extent) as required by the twoGbMaxExtentFlat format.

When multiple EROFS layers are merged, runtime-rs generates a VMDK descriptor file (twoGbMaxExtentFlat format):

Text Only

# Disk DescriptorFile
version=1
CID=fffffffe
parentCID=ffffffff
createType="twoGbMaxExtentFlat"

# Extent description
RW 2048 FLAT "/path/to/fsmeta.erofs" 0
RW 4096 FLAT "/path/to/layer1.erofs" 0
RW 8192 FLAT "/path/to/layer2.erofs" 0

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "15"
ddb.geometry.heads = "16"
ddb.geometry.sectors = "63"
ddb.adapterType = "ide"

QEMU's VMDK driver reads this descriptor and presents all extents as a single contiguous block device to the guest. The guest kernel's EROFS driver then mounts this combined device with multi-device support.

How it works

The containerd EROFS snapshotter prepares a multi-layer rootfs layout:

Text Only

Mount[0]: ext4  rw layer       --> virtio-blk device (writable upper layer)
Mount[1]: erofs layers          --> virtio-blk device (read-only, via VMDK for multi-extent)
Mount[2]: overlay               --> guest agent mounts overlay combining upper + lower

For the EROFS read-only layers:

• Single layer: The single .erofs blob is attached directly as a raw virtio-blk device.
• Multiple layers: Multiple .erofs blobs (the base layer + device= extra layers) are merged into a single virtual block device using a VMDK flat-extent descriptor (twoGbMaxExtentFlat format). QEMU's VMDK driver parses the descriptor and concatenates all extents transparently.

Inside the guest VM, the kata-agent:

1. Mounts the ext4 block device as the writable upper layer.
2. Mounts the erofs block device as the read-only lower layer.
3. Creates an overlay filesystem combining the two.

Verify QEMU VMDK support

The multi-layer EROFS rootfs relies on QEMU's VMDK block driver to present a VMDK flat-extent descriptor as a single virtual disk. QEMU must be compiled with VMDK format support enabled (this is typically on by default, but some minimal or custom builds may disable it).

Run the following command to check:

Bash

$ qemu-system-x86_64 -drive format=help 2>&1 | grep vmdk

You should see vmdk in the Supported formats list, for example:

Text Only

Supported formats: blkdebug blklogwrites blkreplay blkverify bochs cloop
compress copy-before-write copy-on-read dmg file ftp ftps host_cdrom
host_device http https luks nbd null-aio null-co nvme parallels preallocate
qcow qcow2 qed quorum raw replication snapshot-access ssh throttle vdi vhdx
vmdk vpc vvfat

If vmdk does not appear, you need to rebuild QEMU with VMDK support enabled.

Limitations

Hypervisor support: The fsmerged EROFS rootfs feature currently only supports QEMU as the hypervisor, because it depends on the VMDK flat-extent descriptor format for merging multiple EROFS layers into a single block device. The following hypervisors do not support VMDK format block devices at this time, and therefore cannot be used with fsmerged EROFS rootfs:

• Cloud Hypervisor (CLH) — no VMDK block device support (WIP)
• Firecracker — no VMDK block device support (WIP)
• Dragonball — no VMDK block device support (TODO)

For single-layer EROFS (only one .erofs blob, no device= extra layers), the blob is attached as a raw block device without a VMDK descriptor. This mode may work with other hypervisors that support raw virtio-blk devices, but has not been fully tested.

References

• EROFS documentation
• Containerd EROFS snapshotter
• Kata Containers architecture