Kata Containers with virtio-fs-nydus¶

Introduction¶

Nydus is a container image acceleration service that provides fast container startup and on-demand data loading. Kata Containers integrates with Nydus through virtio-fs, supporting two operational modes:

Operational Modes¶

Standalone Mode (virtio-fs-nydus)
Used with QEMU and Cloud-Hypervisor
Nydusd runs as an independent process
Mounts RAFS through nydusd and lets the guest kernel assemble overlayfs
Inline Mode (inline-virtio-fs) / Builtin Nydus
Used with Dragonball VMM
Nydusd is built into the VMM (builtin nydus)
Lower resource overhead and better performance
Suitable for lightweight VM scenarios

Refer to kata-nydus-design for detailed design documentation.

Architecture Overview¶

Standalone Mode Architecture¶

Text Only

┌─────────────────────────────────────────────────────────┐
│                      Host System                         │
│  ┌──────────────┐         ┌──────────────────────┐      │
│  │   nydusd     │◄────────┤  nydus-snapshotter   │      │
│  │  (standalone)│         │                      │      │
│  └──────┬───────┘         └──────────────────────┘      │
│         │ virtiofs                                       │
│         ▼                                                │
│  ┌──────────────┐                                       │
│  │     QEMU     │                                       │
│  │  / Cloud-Hypervisor                                  │
│  └──────────────┘                                       │
└─────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────┐
│                      Guest VM                            │
│  ┌──────────────────────────────────────────────┐       │
│  │  /run/kata-containers/shared/                 │       │
│  │    ├── containers/<cid>/rootfs  (overlay)     │       │
│  │    │     ├── upperdir                         │       │
│  │    │     ├── workdir                          │       │
│  │    │     └── lowerdir (from rafs)             │       │
│  │    └── rafs/<cid>/lowerdir (nydus image)      │       │
│  └──────────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────┘

Inline Mode (Builtin Nydus) Architecture¶

Text Only

┌─────────────────────────────────────────────────────────┐
│                      Host System                         │
│  ┌──────────────────────────────────────────────┐       │
│  │          nydus-snapshotter                    │       │
│  └──────────────────────────────────────────────┘       │
│                                                          │
│  ┌──────────────────────────────────────────────┐       │
│  │              Dragonball VMM                   │       │
│  │  ┌────────────────────────────────────────┐  │       │
│  │  │     Builtin Nydusd (virtiofs server)    │  │       │
│  │  │  ┌──────────────────────────────────┐   │  │       │
│  │  │  │  Vfs (Virtual File System)       │   │  │       │
│  │  │  │  ├── Rafs (nydus image backend)  │   │  │       │
│  │  │  │  └── PassthroughFs               │   │  │       │
│  │  │  └──────────────────────────────────┘   │  │       │
│  │  └────────────────────────────────────────┘  │       │
│  └──────────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────┐
│                      Guest VM                            │
│  ┌──────────────────────────────────────────────┐       │
│  │  /run/kata-containers/shared/containers/      │       │
│  │    ├── <cid>/rootfs          (overlay mount)  │       │
│  │    ├── <cid>/rootfs_lower/   (Rafs mount)     │       │
│  │    └── <cid>/snapshotdir/                     │       │
│  │          ├── fs/             (upperdir)       │       │
│  │          └── work/           (workdir)        │       │
│  │                                               │       │
│  │  Guest Kernel Overlay Assembly:               │       │
│  │    overlay lowerdir=rootfs_lower/             │       │
│  │           upperdir=snapshotdir/fs/            │       │
│  │           workdir=snapshotdir/work/           │       │
│  └──────────────────────────────────────────────┘       │
└─────────────────────────────────────────────────────────┘

Key Components¶

nydusd: The nydus daemon that provides:
Virtiofs server for file sharing
Rafs (Registry Accelerated File System) image mounting
Passthrough filesystem for host-guest file sharing
nydus-snapshotter: Containerd snapshotter that:
Manages nydus image layers
Provides RAFS bootstrap/config/snapshot directory information to Kata
Prepares container rootfs
Communicates with Kata runtime
Builtin Nydus (Inline Mode): Integrated nydus in Dragonball VMM:
Runs inside the VMM process, no separate daemon needed
Uses fuse_backend_rs Vfs to manage multiple filesystem backends
Supports Rafs for nydus image mounting
Supports PassthroughFs for host directory sharing
Lower memory footprint and faster startup
Overlay filesystem assembled by guest kernel instead of nydusd

Prerequisites¶

1. Install Nydus¶

Use the nydus latest branch and build nydusd:

Bash

git clone https://github.com/dragonflyoss/image-service.git
cd image-service
make nydusd

2. Deploy Nydus Environment¶

Deploy nydus environment as described in Nydus Setup for Containerd Environment.

3. Start Nydus Snapshotter¶

Start nydus-snapshotter with enable_nydus_overlayfs enabled:

Bash

./nydus-snapshotter --enable-nydus-overlayfs

4. Build Kata Containers¶

Use kata-containers latest branch to compile and build kata-containers.img.

Configuration¶

Shared Filesystem Types¶

Kata Containers supports the following shared filesystem types:

Type	Description	Hypervisor
`virtio-fs`	Standard virtio-fs with virtiofsd	QEMU, Cloud-Hypervisor
`virtio-fs-nydus`	Virtio-fs with standalone nydusd	QEMU, Cloud-Hypervisor
`inline-virtio-fs`	Inline virtio-fs with builtin nydus	Dragonball
`none`	Disable shared filesystem	All

Configuration for QEMU¶

Update configuration-qemu.toml (for the go runtime) or configuration-qemu-runtime-rs.toml (for runtime-rs):

TOML

# Enable virtio-fs-nydus for standalone mode
shared_fs = "virtio-fs-nydus"

# Path to nydusd binary (required for virtio-fs-nydus)
virtio_fs_daemon = "/usr/local/bin/nydusd"

# Optional: Extra arguments for nydusd
# Example: virtio_fs_extra_args = ["--log-level", "debug", "--threads", "4"]
virtio_fs_extra_args = []

# Cache mode for virtio-fs (never, auto, always)
virtio_fs_cache = "never"

# Optional: Enable DAX for better performance
virtio_fs_is_dax = false

Configuration for Cloud-Hypervisor¶

Update configuration-clh.toml (for the go runtime) or configuration-clh-runtime-rs.toml (for runtime-rs):

TOML

shared_fs = "virtio-fs-nydus"
virtio_fs_daemon = "/usr/local/bin/nydusd"
virtio_fs_extra_args = []
virtio_fs_cache = "never"

Configuration for Dragonball (Inline Mode / Builtin Nydus)¶

For Dragonball VMM, use inline mode (builtin nydus), update configuration-dragonball.toml:

TOML

shared_fs = "inline-virtio-fs"
# Note: virtio_fs_daemon is not needed for inline mode
# nydusd is built into Dragonball VMM
virtio_fs_cache = "never"

How Nydusd Works in Kata¶

Nydusd Startup Process¶

Standalone Mode¶

When using virtio-fs-nydus, Kata runtime starts nydusd with the following parameters:

Bash

nydusd virtiofs \
  --hybrid-mode \
  --log-level info \
  --apisock /path/to/nydusd-api.sock \
  --sock /path/to/virtiofs.sock

Key features:

Hybrid Mode: Enables both Rafs and passthrough filesystem support
API Socket: Provides HTTP API for runtime to mount Rafs images
Virtiofs Socket: Used by hypervisor for virtio-fs communication

After nydusd starts, Kata runtime automatically:

Waits for the nydusd API server to be ready
Mounts passthrough_fs at /containers within the nydusd virtiofs namespace
This maps to /run/kata-containers/shared/containers/ in the guest
The passthrough_fs provides the writable layer for container overlay

Inline Mode (Builtin Nydus)¶

When using inline-virtio-fs with Dragonball:

Nydusd is built into the Dragonball VMM binary
No separate process or daemon startup required
Virtiofs server initializes automatically during VMM boot
Filesystem backends (Rafs, PassthroughFs) are mounted via VMM API
Lower resource usage and faster initialization

Filesystem Layout in Guest¶

Standalone Mode (virtio-fs-nydus)¶

Text Only

/run/kata-containers/shared/                    # virtiofs mount point
├── containers/<container-id>/                  # passthrough_fs from host
│   ├── rootfs/                                 # container rootfs mount point
│   └── snapshotdir/                            # snapshot directory
│       ├── fs/                                 # upperdir (writable layer)
│       └── work/                               # workdir (overlay work directory)
└── rafs/<container-id>/lowerdir/               # Rafs mount (nydus image)

Inline Mode (inline-virtio-fs / Builtin Nydus)¶

Text Only

/run/kata-containers/shared/containers/
├── <container-id>/
│   ├── rootfs/                                 # container rootfs mount point
│   ├── rootfs_lower/                           # Rafs mount (lowerdir)
│   └── snapshotdir/
│       ├── fs/                                 # upperdir
│       └── work/                               # workdir
└── passthrough/                                # passthrough filesystem

Overlay Filesystem Assembly¶

Standalone Mode (Guest Kernel Overlay)¶

The nydus snapshotter provides the RAFS bootstrap path, nydus config, and snapshot directory through the container rootfs mount options. Runtime-rs passes the RAFS source/config to nydusd, then asks kata-agent to assemble the writable overlay in the guest kernel with:

Lowerdir: Rafs mount point (nydus image)
Upperdir: Writable layer from snapshot directory
Workdir: Overlay work directory

The nydusd mount request only mounts RAFS:

JSON

{
  "fs_type": "rafs",
  "source": "/path/to/bootstrap",
  "config": "{...nydus config...}"
}

Runtime-rs then sends an overlay Storage to kata-agent whose lowerdir points at /run/kata-containers/shared/rafs/<cid>/lowerdir and whose upper/work dirs come from /run/kata-containers/shared/containers/<cid>/snapshotdir/.

Inline Mode (Guest Kernel Overlay)¶

The guest kernel assembles overlay filesystem:

Kata agent receives overlay mount information
Lowerdir points to Rafs mount (rootfs_lower/)
Upperdir and workdir from snapshot directory
No native overlay support in builtin nydusd

Usage Examples¶

Running Containers with nerdctl¶

Bash

$sudo nerdctl run --snapshotter nydus --runtime io.containerd.kata.v2 --net=none --rm -it ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
pmem0     259:0    0  254M  1 disk
`-pmem0p1 259:1    0  253M  1 part

Using with Kubernetes¶

Create a Pod specification:

YAML

apiVersion: v1
kind: Pod
metadata:
  name: nydus-test
  annotations:
    io.containerd.osfeature: "nydus.remoteimage.v1"
spec:
  runtimeClassName: kata
  containers:
  - name: test
    image: ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5
    command: ["/bin/sleep", "600"]

Advanced Configuration¶

Nydusd Extra Arguments¶

You can pass additional arguments to nydusd through configuration:

TOML

virtio_fs_extra_args = ["--log-level", "debug", "--threads", "4"]

Prefetch Configuration¶

Enable prefetch for faster container startup:

TOML

# Path to prefetch file list
prefetch_list_path = "/path/to/prefetch_file.list"

Performance Tuning¶

Cache Mode¶

Choose appropriate cache mode based on your use case:

never: No caching, always fetch from host (default, safest)
auto: Cache with timeout, good for read-heavy workloads
always: Aggressive caching, best performance but may see stale data

TOML

virtio_fs_cache = "auto"

DAX (Direct Access)¶

Enable DAX for memory-mapped I/O:

TOML

virtio_fs_is_dax = true
virtio_fs_cache_size = 1024  # Size in MiB

Debugging and Path Mappings¶

Debug Logging¶

Enable debug logging for nydusd:

TOML

virtio_fs_extra_args = ["--log-level", "debug"]

Or via annotation:

YAML

annotations:
  io.katacontainers.config.hypervisor.virtio_fs_extra_args: "--log-level=debug"

Path Mappings¶

Standalone Mode (virtio-fs-nydus)¶

Component	Host Path	Guest Path	Notes
Virtiofs mount	N/A	`/run/kata-containers/shared/`	Root virtiofs mount point
Passthrough FS	`/run/kata-containers/shared/sandboxes/<sid>/rw/`	`/run/kata-containers/shared/containers/`	Mounted at `/containers` in nydusd namespace
Rafs mount	Bootstrap path from snapshotter	`/run/kata-containers/shared/rafs/<cid>/lowerdir`	Mounted via nydusd API
Container rootfs	`/run/kata-containers/shared/sandboxes/<sid>/rw/<cid>/rootfs`	`/run/kata-containers/shared/containers/<cid>/rootfs`	Overlay mount point
Snapshot dir	From snapshotter	`/run/kata-containers/shared/containers/<cid>/snapshotdir/`	Contains upperdir and workdir

Inline Mode (inline-virtio-fs)¶

Component	Host Path	Guest Path	Notes
Virtiofs mount	N/A	`/run/kata-containers/shared/containers/`	Root virtiofs mount point
Passthrough FS	`/run/kata-containers/shared/sandboxes/<sid>/rw/passthrough/`	`/run/kata-containers/shared/containers/passthrough/`	Uses PASSTHROUGH_FS_DIR
Rafs mount	Bootstrap path from snapshotter	`/run/kata-containers/shared/containers/<cid>/rootfs_lower/`	Mounted via DeviceManager
Container rootfs	`/run/kata-containers/shared/sandboxes/<sid>/rw/passthrough/<cid>/rootfs`	`/run/kata-containers/shared/containers/<cid>/rootfs`	Overlay mount point

Comparison: Standalone vs Inline Mode¶

Feature	Standalone (virtio-fs-nydus)	Inline (inline-virtio-fs)
Hypervisor	QEMU, Cloud-Hypervisor	Dragonball
Nydusd Process	Independent process	Built into VMM
Overlay Support	Guest kernel	Guest kernel
Performance	Good	Good
Resource Usage	Higher (separate process)	Lower (integrated)
Flexibility	More configurable	Less configurable
Use Case	General purpose	Lightweight VMs
Startup Time	Slower (daemon startup)	Faster (no daemon)
Memory Overhead	Higher	Lower

Kata Containers with virtio-fs-nydus¶

Introduction¶

Operational Modes¶

Architecture Overview¶

Standalone Mode Architecture¶

Inline Mode (Builtin Nydus) Architecture¶

Key Components¶

Prerequisites¶

1. Install Nydus¶

2. Deploy Nydus Environment¶

3. Start Nydus Snapshotter¶

4. Build Kata Containers¶

Configuration¶

Shared Filesystem Types¶

Configuration for QEMU¶

Configuration for Cloud-Hypervisor¶

Configuration for Dragonball (Inline Mode / Builtin Nydus)¶

How Nydusd Works in Kata¶

Nydusd Startup Process¶

Standalone Mode¶

Inline Mode (Builtin Nydus)¶

Filesystem Layout in Guest¶

Standalone Mode (virtio-fs-nydus)¶

Inline Mode (inline-virtio-fs / Builtin Nydus)¶

Overlay Filesystem Assembly¶

Standalone Mode (Guest Kernel Overlay)¶

Inline Mode (Guest Kernel Overlay)¶

Usage Examples¶

Running Containers with nerdctl¶

Using with Kubernetes¶

Advanced Configuration¶

Nydusd Extra Arguments¶

Prefetch Configuration¶

Performance Tuning¶

Cache Mode¶

DAX (Direct Access)¶

Debugging and Path Mappings¶

Debug Logging¶

Path Mappings¶

Standalone Mode (virtio-fs-nydus)¶

Inline Mode (inline-virtio-fs)¶

Comparison: Standalone vs Inline Mode¶

References¶