For the last few weeks I’ve been working on Velarx, a container engine written in Rust that is built around composefs from the start. I used AI heavily to prototype it, otherwise in a few weeks I wouldn’t have gotten to something that runs real images and that I could compare with Podman. Like composefs, it stores every file by its fs-verity digest. The name comes from the Latin velox, swift, and arx, citadel, and those are the two goals: it has to be fast, and what it runs must be verified from the image signature down to every page the kernel reads.

The store#

Every regular file of every layer is stored once under objects/, named by its fs-verity digest, as in a composefs repository. A layer is never unpacked into a directory. It is a table of contents and an EROFS image with only the metadata, where each regular file is a stub with trusted.overlay.redirect and trusted.overlay.metacopy pointing to the object and carrying its digest. It is the layout from the composefs sealing post, with one EROFS image per layer instead of one per image, so a layer shared by many images is built only once.

There isn’t a single fs-verity digest for a file: it depends on the hash algorithm, the Merkle tree block size and the salt.

Velarx uses the same parameters as composefs, SHA-256 (or SHA-512) over 4096-byte blocks and no salt, and the object names say it, as in fsverity-sha256-12:<hex> where 12 is the log2 of the block size.

All the per-file work, hashing, enabling fs-verity and writing the EROFS images, is done when the pull ends. Starting a container then costs the same whatever the size of the image, there is nothing to unpack or chown, just a few mounts.

Everything else is in a single SQLite database: containers, images, layers, the tables of contents, every file and the layers referencing it, networks and IP allocations, ports, volumes, secrets, the build cache and the namespaces of the rootless user. Podman spreads the same information across the containers/storage JSON files and their lock files, the libpod database, the netavark configuration and its IPAM database and a secrets file. With one database, a change that touches several of these is one transaction, and garbage collection is just a query for the files that no layer references anymore.

Other differences from Podman#

There is no conmon. Each container has a velarx-monitor process that holds its stdio, writes the log, records the exit status in the database and releases the network itself when the container exits.

There is no pause process either. Rootless, all the processes of a user share one user namespace and one mount namespace for the boot, and nothing keeps them alive except the processes living in them. The database records the namespaces by the file handles of their nsfs files, which have a unique ID unlike inode numbers, so a command never joins a namespace that was recycled.

A daemon is not needed, each command can run the engine in its own process. When there is one, the REST API that the Podman and Docker clients use and the CRI server are separate processes that only translate their protocol and have no access to the store. The registry client and the decompression run in their own processes too, under Landlock and seccomp, so the code parsing what comes from the network never touches the store.

How fast is it#

I compared Velarx with Podman 5.8.9 on a Fedora 44 VM with a 7.4-rc2 kernel, 16 vCPUs and 4 GB of memory, both as root and with crun 1.30, pulling from a registry on localhost. Podman doesn’t use fs-verity, so to compare the same work both stores were on the XFS root file system, where Velarx runs without it. The small image is alpine, one layer of 3.9 MB (8.7 MB uncompressed), the big one python:3.12, 7 layers and 412 MB (1.13 GB uncompressed), both with gzip layers as they are on Docker Hub:

operation Velarx Podman Velarx/Podman
pull, small image, empty store 169.9 ms ± 21.8 ms 213.0 ms ± 8.5 ms 0.80
pull, big image, empty store 2.850 s ± 159.2 ms 3.852 s ± 177.1 ms 0.74
run --rm small image 52.1 ms ± 4.6 ms 246.7 ms ± 6.9 ms 0.21
run --rm --network none small image 33.4 ms ± 2.9 ms 148.1 ms ± 3.6 ms 0.23
run --rm big image 56.1 ms ± 7.1 ms 252.2 ms ± 14.9 ms 0.22
create, start and rm 144.7 ms ± 29.0 ms 319.0 ms ± 8.1 ms 0.45
rm 31.9 ms ± 1.7 ms 62.9 ms ± 1.9 ms 0.51
rmi 25.0 ms ± 1.5 ms 38.9 ms ± 1.9 ms 0.64
exec true 18.0 ms ± 0.7 ms 123.9 ms ± 3.1 ms 0.15

Running python:3.12 costs the same as running alpine, and exec is about 7 times faster.

The chain of trust#

The signature in policy.json covers the manifest. The manifest has the digests of the config and of the layers, and for each layer an io.containers.composefs.digest annotation with the fs-verity digest of its EROFS image. The EROFS image has the fs-verity digest of each file, and the kernel checks each file against it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
policy.json
  |  signature over the manifest
  v
manifest
  |  io.containers.composefs.digest of each layer
  v
EROFS image of each layer
  |  fs-verity digest of each file in the stubs
  v
objects/<fs-verity digest>
  |  overlayfs verity=on or verity=require
  v
fs-verity checks every page on read

For this to work the images are pushed in a modified zstd:chunked format. The table of contents of zstd:chunked already lists every file and its chunks. Velarx adds an fsverity field with the fs-verity digest of each regular file, written as sha256-12:<hex> so that the algorithm and the block size are part of the value, and the io.containers.composefs.digest annotation to each layer in the manifest. With the digests in the table of contents, a puller knows the name each file will have in the store before downloading it, so it can skip what it already has, ask other nodes for it, and verify each file as it arrives. With the EROFS digest, the image the puller builds from the table of contents can be compared with what the publisher signed. Images pushed by other tools have neither, so they are pulled eagerly and the digests are computed locally. They still run, but the chain stops at the manifest.

At every start the engine measures each layer image with FS_IOC_MEASURE_VERITY, which takes constant time since the kernel already has the Merkle tree, and compares it with the manifest. The same open file is then passed to EROFS with FSCONFIG_SET_FD, so nothing can be swapped between the check and the mount. At runtime overlayfs refuses with EIO any file whose digest doesn’t match the stub, and with verity=require also any file without one.

This doesn’t protect you from whoever owns the store, root or the rootless user can still edit the database and replace files. What they cannot do is change what a container runs without the start check or the kernel noticing, as long as the manifest is signed and the digests are required.

The cost of computing the Merkle trees is paid once, at pull time, when every file pulled is sealed with fs-verity. On an ext4 with fs-verity and verity=require, the cold pull of python:3.12 takes 5.2 s instead of the 4.1 s it takes on the same ext4 without it, while starting a container costs the same.

Peer-to-peer#

Another thing I wanted to experiment with is pulling images from peers.

Since every file is identified by its digest, it doesn’t matter who sends it. velarx-p2p is an optional service that lets nodes pull from each other before going to the registry:

1
2
3
4
5
6
7
[[sources.external]]
name = "peers"
socket = "/run/velarx/services/p2p-fetcher.sock"
peer = true

[pull]
sources = ["peers", "registry"]

What a peer sends goes through the same checks as what a registry sends, so a bad peer can make a pull slower but not wrong. Nodes are admitted with a pre-shared key or with certificates.

Each node tells the others which layers it has with a Bloom filter, so a pull knows whom to ask without asking everyone first. The interesting case is a rollout, where all the nodes pull the same image at the same time. A node shares each file as soon as it has verified it, without waiting for the rest of the layer, and each node fetches the files in a different order, so they trade files with each other instead of waiting for the first one to finish. With 20 nodes on 1 Gbit/s links, pulling 300 MB went from 47 to 25 seconds.

Rootless and FUSE#

As root, Velarx mounts the EROFS image of each layer, stacks them as overlayfs lower layers and adds objects/ as a data-only layer with datadir+. Rootless it can’t: EROFS cannot be mounted in a user namespace, and with userxattr overlayfs reads user.overlay.* instead of the trusted.overlay.* xattrs of the stubs. So rootless Velarx serves the EROFS images through FUSE, one directory per layer, translating the owners to the ids of the container and the trusted.overlay.* xattrs to user.overlay.*:

1
2
3
mount -t overlay overlay \
    -o userxattr,lowerdir+=layer3,lowerdir+=layer2,lowerdir+=layer1,datadir+=objects \
    merged

FUSE is supposed to serve only the metadata. On open, overlayfs follows the stub to the object and reads it directly, so no read goes through userspace and the page cache of an object is shared among all the containers using it.

It worked with single layer images and broke as soon as there was more than one layer: a stub in the lowest layer works, the same stub one layer above it fails with EPERM, and the kernel logs overlayfs: refusing to follow metacopy origin.

The overlayfs fix#

Following redirects from a lower layer into a data-only layer without metacopy=on was allowed in 6.16 by commit 5ef7bcdeecc9 (“ovl: relax redirect/metacopy requirements for lower -> data redirect”), exactly so that data-only layers could be used with userxattr. The problem is that ovl_lookup_layers() keeps walking the lower layers after it finds a metacopy file. If the stub is in the lowest layer the loop ends and the data is looked up in the data-only layers. If it is in any other layer, the next iteration calls ovl_check_follow_redirect():

1
2
3
4
5
6
7
8
9
static bool ovl_check_follow_redirect(struct ovl_lookup_data *d)
{
	struct ovl_fs *ofs = OVL_FS(d->sb);

	if (d->metacopy && !ofs->config.metacopy) {
		pr_warn_ratelimited("refusing to follow metacopy origin for (%pd2)\n", d->dentry);
		return false;
	}
	...

and since metacopy=on can’t be used with userxattr, the lookup fails with EPERM.

The same commit also changed what redirect_dir=nofollow means without showing it: with a data-only layer some redirects were followed while the mount options said redirect_dir=nofollow, or nothing when that was the default.

The fix, cbf4ccabcbfd (“ovl: follow lower to lowerdata redirects from any lower layer”), is in 7.4-rc3 and marked for stable. The lookup now stops at a lower metacopy file with an absolute redirect, when there are data-only layers and no metacopy=on, and looks the data up only in the data-only layers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
/*
 * Without metacopy=on, the absolute redirect of a lower metacopy file is
 * followed only into the data-only layers.
 */
static bool ovl_is_lowerdata_redirect(struct ovl_lookup_ctx *ctx,
				      struct ovl_lookup_data *d)
{
	struct ovl_fs *ofs = OVL_FS(d->sb);

	return ofs->numdatalayer && !ofs->config.metacopy && ctx->ctr &&
	       d->metacopy && d->absolute_redirect;
}

It also adds redirect_dir=followdata, which follows only redirects from a lower layer into a data-only layer. It is selected automatically when there are data-only layers and nothing more permissive was asked for, so the mount options now show what the kernel does, and redirect_dir=nofollow with data-only layers is refused since those layers can only be reached through redirects.

Older kernels#

Velarx still has to run on kernels without the fix, so the first rootless container with more than one layer mounts a small overlay on a detached tmpfs, with redirect_dir=followdata and a stub in a middle layer, and tries to read it. If it works, all the layers keep their stubs. If not, the FUSE server removes the stubs from every layer but the lowest and serves the files of the upper layers itself, from userspace. FUSE passthrough would help, but it needs CAP_SYS_ADMIN in the initial user namespace.

To see the cost, on a VM with the stock 7.3 kernel I made an image with a 512 MB file /low in its only layer, another image on top of it with a 512 MB file /top, and ran a rootless container from the second one. Its overlay looks like this (paths shortened):

1
overlay / ... lowerdir+=.../layer2,lowerdir+=.../layer1,datadir+=.../objects,upperdir=...,workdir=...,redirect_dir=nofollow,uuid=on,userxattr

There is the redirect_dir=nofollow from before, while the stubs of layer1 are followed into objects. Reading both files with a cold cache:

1
2
3
4
5
6
7
Benchmark 1: velarx exec fb /busybox cat /low
  Time (mean ± σ):     215.7 ms ±   6.2 ms    [User: 49.4 ms, System: 103.3 ms]
  Range (min … max):   206.7 ms … 227.3 ms    8 runs

Benchmark 2: velarx exec fb /busybox cat /top
  Time (mean ± σ):     314.1 ms ±  10.4 ms    [User: 54.7 ms, System: 101.3 ms]
  Range (min … max):   304.5 ms … 335.8 ms    8 runs

The file served by FUSE is about 46% slower to read, and its page cache belongs to the FUSE inode, so it is not shared with other containers. With the fix both files are read by overlayfs directly from the store.

The source is at github.com/giuseppe/velarx.