For a while now we’ve had composefs support in the container-libs. For each OCI layer in an image, we build a read-only EROFS blob that can be verified with fs-verity and that the kernel mounts directly. It works well if you look at a single container, but the moment you start thinking about composefs as the storage for a real container host where many containers are running, we have a problem. Many of them share the same image layers but with composefs every one of those containers mounts the same EROFS blobs, and each mount is a separate world as far as the kernel is concerned. The bytes on disk are shared, but the memory used to cache them is not. That is one of the reasons composefs is not yet the default way to store and run containers, and it is what I want to talk about here.

The same image, mounted many times#

When you mount an EROFS blob you get a new superblock, and with it a fresh set of inodes for the files inside the image. The page cache is not really owned by the superblock, it hangs off each inode, but the end result is the same: mount the blob a second time and you get a second superblock with a second set of inodes, so every file inside the image is cached again from scratch. The backing file itself is only cached once, since both mounts open the same inode on the host, but everything EROFS builds on top of it, the superblock and all of its inodes, is duplicated per mount.

How much memory#

To put a number on it I mounted twenty times an EROFS 177M image based on Fedora and walked the whole tree of each mount, which is roughly what twenty containers sharing a base image would do. The loop drops the caches first so the starting point is clean, then mounts and traverses the same image twenty times:

1
2
3
4
5
6

sync; echo 3 > /proc/sys/vm/drop_caches
for i in $(seq 1 20); do
    mkdir /mnt/m$i
    mount -t erofs fedora.erofs /mnt/m$i
    find /mnt/m$i > /dev/null
done

Before and after the loop I sampled three numbers. How many distinct superblocks the mounts appear as, counted from /proc/self/mountinfo; how many erofs_inode objects the slab allocator is holding, from /proc/slabinfo; and the total Slab: figure from /proc/meminfo, which is where those inodes and superblocks actually live. The find matters here: without walking the tree the inodes are never instantiated and the inode cache stays empty. That gives:

1
2
3

Superblocks: 20
erofs_inode delta: +45864
Slab delta: +37448 kB

So twenty separate superblocks, an inode cache grown by more than forty thousand objects, and around 37MB of slab spent holding twenty copies of the same metadata. Multiply the image size and the container count by something realistic and it stops being a rounding error.

Doing it in userspace#

A kernel change wasn’t strictly required though. Because an EROFS mount can be backed by a file descriptor, a long-running daemon could open each image once, mount it, and hand the same mount out to every container that needs it, keeping the fd alive for as long as any container is using it. The deduplication would then happen entirely in userspace, and it would still rely on the fd based mounting from the first patch to tie the mount to the descriptor the daemon holds.

The problem is that it needs a daemon at all, and a daemon can be restarted. When that happens there is no way to retrieve those file descriptors again: the mounts the daemon was keeping alive are lost, and there is nothing to reconnect to.

Letting the kernel notice#

The fix is to let the kernel recognize that a new mount is backed by a file it already has a superblock for, and reuse that superblock instead of building another one. I proposed a superblock_share mount option to EROFS for file-backed mounts. When it is used, EROFS looks for an existing superblock backed by the same file, with compatible options, before allocating a new one. If it finds one the new mount just reuses the existing superblock.

There is nothing more to using it than using the new flag. Mount a file backed EROFS image with superblock_share, and any later mount of the same image that also asks for it reuses the first superblock instead of building a new one:

1
2

mount -t erofs -o superblock_share fedora.erofs /mnt/a
mount -t erofs -o superblock_share fedora.erofs /mnt/b

Here /mnt/b costs almost nothing on top of /mnt/a, since the two share a superblock and the inodes underneath it, so the files inside the image are only cached once.

The same twenty-mount test with the option on:

1
2
3

Superblocks: 1
erofs_inode delta: +2044
Slab delta: +284 kB

One superblock instead of twenty. The base image is now cached once no matter how many containers mount it, and the later mounts are also faster, because the first mount already instantiated the inodes and cached their contents, and the later mounts reuse them.

Turning it on#

Sharing is opt-in because a superblock that is shared among different mounts cannot be remounted with different options, as it would affect every mount. Since it is an opt-in flag, users of the flag are aware of this limitation and that MS_REMOUNT can’t be used on the EROFS mount anymore.

composefs will turn this on since it doesn’t need MS_REMOUNT and takes advantage of the memory deduplication.