Velarx: a container engine with a chain of trust
Table of Contents
For the last few weeks I’ve been working on Velarx, a container engine written in Rust that is built around composefs from the start. I used AI heavily to prototype it, otherwise in a few weeks I wouldn’t have gotten to something that runs real images and that I could compare with Podman. Like composefs, it stores every file by its fs-verity digest. The name comes from the Latin velox, swift, and arx, citadel, and those are the two goals: it has to be fast, and what it runs must be verified from the image signature down to every page the kernel reads.
The store#
Every regular file of every layer is stored once under objects/,
named by its fs-verity digest, as in a composefs repository. A layer
is never unpacked into a directory. It is a table of contents and an
EROFS image with only the metadata, where each regular file is a stub
with trusted.overlay.redirect and trusted.overlay.metacopy pointing
to the object and carrying its digest. It is the layout from the
composefs sealing post,
with one EROFS image per layer instead of one per image, so a layer
shared by many images is built only once.
There isn’t a single fs-verity digest for a file: it depends on the hash algorithm, the Merkle tree block size and the salt.
Velarx uses the same parameters as composefs, SHA-256 (or SHA-512) over
4096-byte blocks and no salt, and the object names say it, as in
fsverity-sha256-12:<hex> where 12 is the log2 of the block size.
All the per-file work, hashing, enabling fs-verity and writing the EROFS images, is done when the pull ends. Starting a container then costs the same whatever the size of the image, there is nothing to unpack or chown, just a few mounts.
Everything else is in a single SQLite database: containers, images, layers, the tables of contents, every file and the layers referencing it, networks and IP allocations, ports, volumes, secrets, the build cache and the namespaces of the rootless user. Podman spreads the same information across the containers/storage JSON files and their lock files, the libpod database, the netavark configuration and its IPAM database and a secrets file. With one database, a change that touches several of these is one transaction, and garbage collection is just a query for the files that no layer references anymore.
Other differences from Podman#
There is no conmon. Each container has a velarx-monitor process that
holds its stdio, writes the log, records the exit status in the
database and releases the network itself when the container exits.
There is no pause process either. Rootless, all the processes of a user share one user namespace and one mount namespace for the boot, and nothing keeps them alive except the processes living in them. The database records the namespaces by the file handles of their nsfs files, which have a unique ID unlike inode numbers, so a command never joins a namespace that was recycled.
A daemon is not needed, each command can run the engine in its own process. When there is one, the REST API that the Podman and Docker clients use and the CRI server are separate processes that only translate their protocol and have no access to the store. The registry client and the decompression run in their own processes too, under Landlock and seccomp, so the code parsing what comes from the network never touches the store.
How fast is it#
I compared Velarx with Podman 5.8.9 on a Fedora 44 VM with a 7.4-rc2
kernel, 16 vCPUs and 4 GB of memory, both as root and with crun 1.30,
pulling from a registry on localhost. Podman doesn’t use fs-verity, so
to compare the same work both stores were on the XFS root file system,
where Velarx runs without it. The small image is alpine, one layer
of 3.9 MB (8.7 MB uncompressed), the big one python:3.12, 7 layers
and 412 MB (1.13 GB uncompressed), both with gzip layers as they are on
Docker Hub:
| operation | Velarx | Podman | Velarx/Podman |
|---|---|---|---|
| pull, small image, empty store | 169.9 ms ± 21.8 ms | 213.0 ms ± 8.5 ms | 0.80 |
| pull, big image, empty store | 2.850 s ± 159.2 ms | 3.852 s ± 177.1 ms | 0.74 |
run --rm small image |
52.1 ms ± 4.6 ms | 246.7 ms ± 6.9 ms | 0.21 |
run --rm --network none small image |
33.4 ms ± 2.9 ms | 148.1 ms ± 3.6 ms | 0.23 |
run --rm big image |
56.1 ms ± 7.1 ms | 252.2 ms ± 14.9 ms | 0.22 |
| create, start and rm | 144.7 ms ± 29.0 ms | 319.0 ms ± 8.1 ms | 0.45 |
rm |
31.9 ms ± 1.7 ms | 62.9 ms ± 1.9 ms | 0.51 |
rmi |
25.0 ms ± 1.5 ms | 38.9 ms ± 1.9 ms | 0.64 |
exec true |
18.0 ms ± 0.7 ms | 123.9 ms ± 3.1 ms | 0.15 |
Running python:3.12 costs the same as running alpine, and exec is
about 7 times faster.
The chain of trust#
The signature in policy.json covers the manifest. The manifest has
the digests of the config and of the layers, and for each layer an
io.containers.composefs.digest annotation with the fs-verity digest
of its EROFS image. The EROFS image has the fs-verity digest of each
file, and the kernel checks each file against it:
|
|
For this to work the images are pushed in a modified zstd:chunked
format. The table of contents of zstd:chunked already lists every file
and its chunks. Velarx adds an fsverity field with the fs-verity
digest of each regular file, written as sha256-12:<hex> so that the
algorithm and the block size are part of the value, and the
io.containers.composefs.digest annotation to each layer in the
manifest. With the digests in the
table of contents, a puller knows the name each file will have in the
store before downloading it, so it can skip what it already has, ask
other nodes for it, and verify each file as it arrives. With the EROFS
digest, the image the puller builds from the table of contents can be
compared with what the publisher signed. Images pushed by other tools
have neither, so they are pulled eagerly and the digests are computed
locally. They still run, but the chain stops at the manifest.
At every start the engine measures each layer image with
FS_IOC_MEASURE_VERITY, which takes constant time since the kernel
already has the Merkle tree, and compares it with the manifest. The
same open file is then passed to EROFS with FSCONFIG_SET_FD, so
nothing can be swapped between the check and the mount. At runtime
overlayfs refuses with EIO any file whose digest doesn’t match the
stub, and with verity=require also any file without one.
This doesn’t protect you from whoever owns the store, root or the rootless user can still edit the database and replace files. What they cannot do is change what a container runs without the start check or the kernel noticing, as long as the manifest is signed and the digests are required.
The cost of computing the Merkle trees is paid once, at pull time, when
every file pulled is sealed with fs-verity. On an ext4 with fs-verity
and verity=require, the cold pull of python:3.12 takes 5.2 s
instead of the 4.1 s it takes on the same ext4 without it, while
starting a container costs the same.
Peer-to-peer#
Another thing I wanted to experiment with is pulling images from peers.
Since every file is identified by its digest, it doesn’t matter who
sends it. velarx-p2p is an optional service that lets nodes pull
from each other before going to the registry:
|
|
What a peer sends goes through the same checks as what a registry sends, so a bad peer can make a pull slower but not wrong. Nodes are admitted with a pre-shared key or with certificates.
Each node tells the others which layers it has with a Bloom filter, so a pull knows whom to ask without asking everyone first. The interesting case is a rollout, where all the nodes pull the same image at the same time. A node shares each file as soon as it has verified it, without waiting for the rest of the layer, and each node fetches the files in a different order, so they trade files with each other instead of waiting for the first one to finish. With 20 nodes on 1 Gbit/s links, pulling 300 MB went from 47 to 25 seconds.
Rootless and FUSE#
As root, Velarx mounts the EROFS image of each layer, stacks them as
overlayfs lower layers and adds objects/ as a data-only layer with
datadir+. Rootless it can’t: EROFS cannot be mounted in a user
namespace, and with userxattr overlayfs reads user.overlay.*
instead of the trusted.overlay.* xattrs of the stubs. So rootless
Velarx serves the EROFS images through FUSE, one directory per layer,
translating the owners to the ids of the container and the
trusted.overlay.* xattrs to user.overlay.*:
|
|
FUSE is supposed to serve only the metadata. On open, overlayfs follows the stub to the object and reads it directly, so no read goes through userspace and the page cache of an object is shared among all the containers using it.
It worked with single layer images and broke as soon as there was more
than one layer: a stub in the lowest layer works, the same stub one
layer above it fails with EPERM, and the kernel logs overlayfs: refusing to follow metacopy origin.
The overlayfs fix#
Following redirects from a lower layer into a data-only layer without
metacopy=on was allowed in 6.16 by commit 5ef7bcdeecc9 (“ovl: relax
redirect/metacopy requirements for lower -> data redirect”), exactly so
that data-only layers could be used with userxattr. The problem is
that ovl_lookup_layers() keeps walking the lower layers after it
finds a metacopy file. If the stub is in the lowest layer the loop
ends and the data is looked up in the data-only layers. If it is in
any other layer, the next iteration calls
ovl_check_follow_redirect():
|
|
and since metacopy=on can’t be used with userxattr, the lookup
fails with EPERM.
The same commit also changed what redirect_dir=nofollow means without
showing it: with a data-only layer some redirects were followed while
the mount options said redirect_dir=nofollow, or nothing when that
was the default.
The fix,
cbf4ccabcbfd
(“ovl: follow lower to lowerdata redirects from any lower layer”), is
in 7.4-rc3 and marked for stable. The lookup now stops at a lower
metacopy file with an absolute redirect, when there are data-only
layers and no metacopy=on, and looks the data up only in the
data-only layers:
|
|
It also adds redirect_dir=followdata, which follows only redirects
from a lower layer into a data-only layer. It is selected
automatically when there are data-only layers and nothing more
permissive was asked for, so the mount options now show what the
kernel does, and redirect_dir=nofollow with data-only layers is
refused since those layers can only be reached through redirects.
Older kernels#
Velarx still has to run on kernels without the fix, so the first
rootless container with more than one layer mounts a small overlay on a
detached tmpfs, with redirect_dir=followdata and a stub in a middle
layer, and tries to read it. If it works, all the layers keep their
stubs. If not, the FUSE server removes the stubs from every layer but
the lowest and serves the files of the upper layers itself, from
userspace. FUSE passthrough would help, but it needs CAP_SYS_ADMIN
in the initial user namespace.
To see the cost, on a VM with the stock 7.3 kernel I made an image
with a 512 MB file /low in its only layer, another image on top of it
with a 512 MB file /top, and ran a rootless container from the second
one. Its overlay looks like this (paths shortened):
|
|
There is the redirect_dir=nofollow from before, while the stubs of
layer1 are followed into objects. Reading both files with a cold
cache:
|
|
The file served by FUSE is about 46% slower to read, and its page cache belongs to the FUSE inode, so it is not shared with other containers. With the fix both files are read by overlayfs directly from the store.
The source is at github.com/giuseppe/velarx.