How to use Kata Containers and Containerd¶

This document covers the installation and configuration of containerd and Kata Containers. The containerd provides not only the ctr command line tool, but also the CRI interface for Kubernetes and other CRI clients.

This document is primarily written for Kata Containers v3.28 or above, and containerd v1.7.0 or above. Previous versions are addressed here, but we suggest users upgrade to the newer versions for better support.

Concepts¶

Kubernetes `RuntimeClass`¶

RuntimeClass is a Kubernetes feature first introduced in Kubernetes 1.12 as alpha. It is the feature for selecting the container runtime configuration to use to run a pod's containers. This feature is supported in containerd since v1.2.0.

Before the RuntimeClass was introduced, Kubernetes was not aware of the difference of runtimes on the node. kubelet creates Pod sandboxes and containers through CRI implementations, and treats all the Pods equally. However, there are requirements to run trusted Pods (i.e. Kubernetes plugin) in a native container like runc, and to run untrusted workloads with isolated sandboxes (i.e. Kata Containers).

As a result, the CRI implementations extended their semantics for the requirements:

At the beginning, Frakti checks the network configuration of a Pod, and treat Pod with host network as trusted, while others are treated as untrusted.
The containerd introduced an annotation for untrusted Pods since v1.0:
YAML
```
annotations:
   io.kubernetes.cri.untrusted-workload: "true"
```
Similarly, CRI-O introduced the annotation io.kubernetes.cri-o.TrustedSandbox for untrusted Pods.

To eliminate the complexity of user configuration introduced by the non-standardized annotations and provide extensibility, RuntimeClass was introduced. This gives users the ability to affect the runtime behavior through RuntimeClass without the knowledge of the CRI daemons. We suggest that users with multiple runtimes use RuntimeClass instead of the deprecated annotations.

Containerd Runtime V2 API: Shim V2 API¶

The containerd-shim-kata-v2 (short as shimv2 in this documentation) implements the Containerd Runtime V2 (Shim API) for Kata. With shimv2, Kubernetes can launch Pod and OCI-compatible containers with one shim per Pod. Prior to shimv2, 2N+1 shims (i.e. a containerd-shim and a kata-shim for each container and the Pod sandbox itself) and no standalone kata-proxy process were used, even with VSOCK not available.

Kubernetes integration with shimv2

The shim v2 is introduced in containerd v1.2.0 and Kata shimv2 is implemented in Kata Containers v1.5.0.

Install¶

Install Kata Containers¶

Follow the instructions to install Kata Containers.

Install containerd with CRI plugin¶

Note: cri is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default. You do not need to install cri if you have containerd 1.1 or above. Just remove the cri plugin from the list of disabled_plugins in the containerd configuration file (/etc/containerd/config.toml).

Follow the instructions from the CRI installation guide.

Then, check if containerd is now available:

Bash

$ command -v containerd

Install CNI plugins¶

If you have installed Kubernetes with kubeadm, you might have already installed the CNI plugins.

You can manually install CNI plugins as follows:

Bash

$ git clone https://github.com/containernetworking/plugins.git
$ pushd plugins
$ ./build_linux.sh
$ sudo mkdir /opt/cni
$ sudo cp -r bin /opt/cni/
$ popd

Install `cri-tools`¶

Note: cri-tools is a set of tools for CRI used for development and testing. Users who only want to use containerd with Kubernetes can skip the cri-tools.

You can install the cri-tools from source code:

Bash

$ git clone https://github.com/kubernetes-sigs/cri-tools.git
$ pushd cri-tools
$ make
$ sudo -E make install
$ popd

Configuration¶

Configure containerd to use Kata Containers¶

By default, the configuration of containerd is located at /etc/containerd/config.toml, and the cri plugins are placed in the following section:

TOML

[plugins]
  [plugins.cri]
    [plugins.cri.containerd]
      [plugins.cri.containerd.default_runtime]
        #runtime_type = "io.containerd.runtime.v1.linux"

    [plugins.cri.cni]
      # conf_dir is the directory in which the admin places a CNI conf.
      conf_dir = "/etc/cni/net.d"

The following sections outline how to add Kata Containers to the configurations.

Kata Containers as a `RuntimeClass`¶

For Kubernetes users, we suggest using RuntimeClass to select Kata Containers as the runtime for untrusted workloads. The configuration is as follows:

Kata Containers v3.28.0 or above
Containerd v1.7.0 or above
Kubernetes v1.33 or above

The RuntimeClass is suggested.

The following example registers custom runtimes into containerd:

You can check the detailed information about the configuration of containerd in the Containerd config documentation.

In containerd 2.x

TOML

version = 3
[plugins."io.containerd.cri.v1.runtime".containerd]
  [plugins."io.containerd.cri.v1.runtime".containerd.runtimes]
    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"
      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata.options]
        ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml"

In containerd 1.7.x

TOML

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
        ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml"

The following configuration includes two runtime classes:

plugins.<X>.containerd.runtimes.runc: the runc, and it is the default runtime.
plugins.<X>.containerd.runtimes.kata: The function in containerd (reference the document here) where the dot-connected string io.containerd.kata.v2 is translated to containerd-shim-kata-v2 (i.e. the binary name of the Kata implementation of Containerd Runtime V2 (Shim API)). By default, the containerd-shim-kata-v2 (short of shimv2) binary will be installed under the path of /usr/local/bin/.

And <X> is io.containerd.cri.v1.runtime for containerd v2.x and io.containerd.grpc.v1.cri for containerd v1.7.x.

In containerd 1.7.x

TOML

    [plugins.cri.containerd]
      no_pivot = false
    [plugins.cri.containerd.runtimes]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
         privileged_without_host_devices = false
         runtime_type = "io.containerd.runc.v2"
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
         privileged_without_host_devices = true
         pod_annotations = ["io.katacontainers.*"]
         container_annotations = ["io.katacontainers.*"]
         [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
            ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml"

In containerd 2.x

TOML

    [plugins."io.containerd.cri.v1.runtime".containerd]
      no_pivot = false
    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes]
      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
         privileged_without_host_devices = false
         runtime_type = "io.containerd.runc.v2"
        [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
         privileged_without_host_devices = true
         pod_annotations = ["io.katacontainers.*"]
         container_annotations = ["io.katacontainers.*"]
         [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata.options]
            ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml"

privileged_without_host_devices tells containerd that a privileged Kata container should not have direct access to all host devices. If unset, containerd will pass all host devices to Kata container, which may cause security issues.

pod_annotations is the list of pod annotations passed to both the pod sandbox as well as container through the OCI config.

container_annotations is the list of container annotations passed through to the OCI config of the containers.

This ConfigPath option is optional. If you want to use a different configuration file, you can specify the path of the configuration file with ConfigPath in the containerd configuration file. We use containerd 2.x configuration as an example here, and the configuration for containerd 1.7.x is similar, just replace io.containerd.cri.v1.runtime with io.containerd.grpc.v1.cri.

TOML

         [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata.options]
            ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"

Note: In this example, the specified ConfigPath is valid in Kubernetes/Containerd workflow with containerd v1.7+ but doesn't work with ctr and nerdctl.

If you do not specify it, shimv2 first tries to get the configuration file from the environment variable KATA_CONF_FILE. If you want to adopt this way, you should first create a shell script as containerd-shim-kata-v2 which is placed under the path of /usr/local/bin/. The following is an example of the shell script containerd-shim-kata-qemu-v2 which specifies the configuration file with KATA_CONF_FILE

Note: Just use containerd 2.x configuration as an example, the configuration for containerd 1.7.x is similar, just replace io.containerd.cri.v1.runtime with io.containerd.grpc.v1.cri

Bash

~$ cat /usr/local/bin/containerd-shim-kata-qemu-v2
#!/bin/bash
KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-qemu.toml /opt/kata/bin/containerd-shim-kata-v2 "$@"

And then just reference it in the configuration of containerd:

TOML

      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.kata-qemu]
         runtime_type = "io.containerd.kata-qemu.v2"

Finally you can run a Kata container with the runtime io.containerd.kata-qemu.v2:

Bash

$ sudo ctr run --cni --runtime io.containerd.kata-qemu.v2 -t --rm docker.io/library/busybox:latest hello sh

Note: The KATA_CONF_FILE environment variable is valid in both Kubernetes/Containerd workflow with containerd and containerd tools(ctr, nerdctl, etc.) scenarios.

If neither are set, shimv2 will use the default Kata configuration file paths (/etc/kata-containers/configuration.toml and /usr/share/defaults/kata-containers/configuration.toml and /opt/kata/share/defaults/kata-containers/configuration.toml).

Kata Containers as the runtime for untrusted workload¶

For cases without RuntimeClass support, we can use the legacy annotation method to support using Kata Containers for an untrusted workload. With the following configuration, you can run trusted workloads with a runtime such as runc and then, run an untrusted workload with Kata Containers:

TOML

    [plugins."io.containerd.grpc.v1.cri".containerd]
    # "plugins."io.containerd.grpc.v1.cri".containerd.default_runtime" is the runtime to use in containerd.
    [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "io.containerd.runtime.v1.linux"

    # "plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
    [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "io.containerd.kata.v2"

Note: The untrusted_workload_runtime is deprecated since containerd v1.7.0, and it is recommended to use RuntimeClass instead.

You can find more information on the Containerd config documentation

Kata Containers as the default runtime¶

If you want to set Kata Containers as the only runtime in the deployment, you can simply configure as follows:

TOML

    [plugins."io.containerd.grpc.v1.cri".containerd]
    [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
      runtime_type = "io.containerd.kata.v2"

Configuration for `cri-tools`¶

Note: If you skipped the Install cri-tools section, you can skip this section too.

First, add the CNI configuration in the containerd configuration.

The following is the configuration if you installed CNI as the Install CNI plugins section outlined.

Put the CNI configuration as /etc/cni/net.d/10-mynet.conf:

JSON

{
    "cniVersion": "0.2.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "172.19.0.0/24",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ]
    }
}

Next, reference the configuration directory through containerd config.toml:

TOML

[plugins.cri.cni]
    # conf_dir is the directory in which the admin places a CNI conf.
    conf_dir = "/etc/cni/net.d"

The configuration file of crictl command line tool in cri-tools locates at /etc/crictl.yaml:

YAML

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true

Run¶

Launch containers with `ctr` command line¶

Note: With containerd command tool ctr, the ConfigPath is not supported, and the configuration file should be explicitly specified with the option --runtime-config-path, otherwise, it'll use the default configurations.

To run a container with Kata Containers through the containerd command line, you can run the following:

Bash

$ sudo ctr image pull docker.io/library/busybox:latest
$ CONFIG_PATH="/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
$ sudo ctr run --cni --runtime io.containerd.kata.v2 --runtime-config-path $CONFIG_PATH -t --rm docker.io/library/busybox:latest hello sh

This launches a BusyBox container named hello, and it will be removed by --rm after it quits. The --cni flag enables CNI networking for the container. Without this flag, a container with just a loopback interface is created.

Launch containers using `ctr` command line with rootfs bundle¶

Get rootfs¶

Use the script to create rootfs

Bash

ctr i pull quay.io/prometheus/busybox:latest
ctr i export rootfs.tar quay.io/prometheus/busybox:latest

rootfs_tar=rootfs.tar
bundle_dir="./bundle"
mkdir -p "${bundle_dir}"

# extract busybox rootfs
rootfs_dir="${bundle_dir}/rootfs"
mkdir -p "${rootfs_dir}"
layers_dir="$(mktemp -d)"
tar -C "${layers_dir}" -pxf "${rootfs_tar}"
for ((i=0;i<$(cat ${layers_dir}/manifest.json | jq -r ".[].Layers | length");i++)); do
  tar -C ${rootfs_dir} -xf ${layers_dir}/$(cat ${layers_dir}/manifest.json | jq -r ".[].Layers[${i}]")
done

Get `config.json`¶

Use runc spec to generate config.json

Bash

cd ./bundle/rootfs
runc spec
mv config.json ../

Change the root path in config.json to the absolute path of rootfs

JSON

"root":{
    "path":"/root/test/bundle/rootfs",
    "readonly": false
},

Run container¶

Bash

CONFIG_PATH="/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
sudo ctr run -d --runtime io.containerd.kata.v2 --runtime-config-path $CONFIG_PATH --config bundle/config.json hello
sudo ctr t exec --exec-id ${ID} -t hello sh

Launch Pods with `crictl` command line¶

With the crictl command line of cri-tools, you can specify runtime class with -r or --runtime flag. Use the following to launch Pod with kata runtime class with the pod in the example of cri-tools:

Bash

$ sudo crictl runp -r kata podsandbox-config.yaml
36e23521e8f89fabd9044924c9aeb34890c60e85e1748e8daca7e2e673f8653e

You can add container to the launched Pod with the following:

Bash

$ sudo crictl create 36e23521e8f89 container-config.yaml podsandbox-config.yaml
1aab7585530e62c446734f12f6899f095ce53422dafcf5a80055ba11b95f2da7

Now, start it with the following:

Bash

$ sudo crictl start 1aab7585530e6
1aab7585530e6

In Kubernetes, you need to create a RuntimeClass resource and add the RuntimeClass field in the Pod Spec (see this document for more information).

If RuntimeClass is not supported, you can use the following annotation in a Kubernetes pod to identify as an untrusted workload:

YAML

annotations:
   io.kubernetes.cri.untrusted-workload: "true"

How to use Kata Containers and Containerd¶

Concepts¶

Kubernetes RuntimeClass¶

Containerd Runtime V2 API: Shim V2 API¶

Install¶

Install Kata Containers¶

Install containerd with CRI plugin¶

Install CNI plugins¶

Install cri-tools¶

Configuration¶

Configure containerd to use Kata Containers¶

Kata Containers as a RuntimeClass¶

Kata Containers as the runtime for untrusted workload¶

Kata Containers as the default runtime¶

Configuration for cri-tools¶

Run¶

Launch containers with ctr command line¶

Launch containers using ctr command line with rootfs bundle¶

Get rootfs¶

Get config.json¶

Run container¶

Launch Pods with crictl command line¶

Kubernetes `RuntimeClass`¶

Install `cri-tools`¶

Kata Containers as a `RuntimeClass`¶

Configuration for `cri-tools`¶

Launch containers with `ctr` command line¶

Launch containers using `ctr` command line with rootfs bundle¶

Get `config.json`¶

Launch Pods with `crictl` command line¶