← Explainers
Explainer · storage and sandboxes

Blocks to Sandboxes

How disks, filesystems, mounts and sandboxes fit together, built up one layer at a time.

The running example

A parent sandbox fans work out to many short-lived child sandboxes. Each child writes results the parent needs to read. The goal:

  • Own storage per childIt outlives the child.
  • 32+ at onceThe parent reads many at the same time.
  • Picked after bootThe parent doesn't know which ones up front.
  • Changes while runningThe set keeps changing.
child-1 child-2 child-3 … child-N writes disk 1 disk 2 disk 3 disk N reads 32+ parent

Each layer is a short animation. Use Next, or click a figure and press ← →.

Layer 0 · the drive

A drive is a row of numbered boxes

Every storage device speaks the same tiny language: numbered blocks of bytes.

FIG 0 · /dev/vdb, up closeStep 1 of 4
DRIVE = block device /dev/vdb 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 68 69 WRITE 5 READ 5 16 blocks drawn here. A 10 GB disk has 2,621,440 of them.
  1. A drive is a row of numbered blocks, 4 KB each. It has no idea what a file is.

  2. You can ask it two things. One: write these bytes to block 5.

    $ dd if=hi.bin of=/dev/vdb bs=4096 seek=5 count=1
  3. Two: read block 5. That's the whole interface.

    $ dd if=/dev/vdb bs=4096 skip=5 count=1
  4. Linux calls this a block device and names it under /dev. A VM's disks are /dev/vda, /dev/vdb, and so on.

    $ lsblk
    NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
    vda    253:0    0    2G  0 disk /
    vdb    253:16   0   10G  0 disk
Layer 1 · the filesystem

A filesystem is an index on top of the blocks

A file is some blocks of data plus an entry in a table that maps its name to those blocks.

FIG 1 · ext4, simplifiedStep 1 of 6
INDEX · LIVES IN BLOCKS 0–1 (empty) notes.txt → blocks 6, 7 notes.txt 8 KB of text 0 1 index 2 3 4 5 6 7 8 9 10 11 12 13 14 15 notes.txt 4 KB 4 KB entry 8 KB Real ext4 splits its index into a superblock, inodes and a free-block bitmap.
  1. Blocks have numbers, not names. Something has to remember where notes.txt lives.

  2. Reserve a few blocks for an index. Writing an empty index onto a drive is formatting.

    $ mkfs.ext4 /dev/vdb
  3. Saving notes.txt (8 KB) writes its bytes into two free blocks, 6 and 7.

  4. The index records it: notes.txt → blocks 6, 7.

    $ debugfs -R "stat /notes.txt" /dev/vdb | tail -2
    EXTENTS:
    (0-1):6-7
  5. Reading reverses it: look up the name, then read those blocks.

    $ cat notes.txt
  6. The index is the filesystem. Every layer above this one is about never letting two writers disagree about it.

Layer 2 · mounting

Mounting attaches a disk to the folder tree

Programs open files by path, so a disk's files need a place in the tree.

FIG 2 · mount /dev/vdb /mnt/dataStep 1 of 6
/ ├─ home ├─ tmp └─ mnt └─ data (empty) ← mount point ├─ notes.txt └─ results/ /dev/vdb ext4 notes.txt results/ KERNEL MOUNT TABLE /dev/vda/ext4 /dev/vdb/mnt/dataext4
  1. Linux has one folder tree, rooted at /. Every path lives in it.

    $ ls /
    home  mnt  tmp
  2. A new disk, /dev/vdb, has files on it, but nothing in the tree leads to them.

  3. Pick an empty folder to act as the doorway: the mount point.

    $ mkdir -p /mnt/data
  4. Mounting adds a row to the kernel's mount table. The disk's files appear under the doorway.

    $ mount /dev/vdb /mnt/data
    $ findmnt /mnt/data
    TARGET    SOURCE   FSTYPE OPTIONS
    /mnt/data /dev/vdb ext4   rw,relatime
  5. To open /mnt/data/notes.txt, the kernel walks the path, hits the mount, and continues in vdb's index.

    $ cat /mnt/data/notes.txt
  6. Unmounting deletes the row. The disk is untouched and can be mounted somewhere else.

    $ umount /mnt/data

A mount is a row in a table: this folder leads to that filesystem. Mounting something into a sandbox means adding a row to the table of whichever kernel the sandbox runs on.

Layer 3 · sharing a disk

One disk, two kernels: why sharing breaks

Attach one ordinary disk to two machines at once and it gets corrupted, however careful each one is.

FIG 3 · one disk, two machinesStep 1 of 6
Machine A its RAM copy of the index free: 9 10 11 free: 10 11 a.txt → 9 Machine B its RAM copy of the index free: 9 10 11 free: 10 11 b.txt → 9 shared disk INDEX ON DISK free: 9 10 11 a.txt → 9 b.txt → 9 9 b.txt index index a.txt b.txt one block disk ↔ one kernel
  1. The kernel keeps a copy of the index in RAM and writes changes back later. This is the page cache.

  2. Attach the same disk to machine B. B loads its own copy. Nothing keeps the two in sync.

  3. A saves a.txt. Its copy says block 9 is free, so A takes it.

  4. B saves b.txt. B's copy is stale, so it takes block 9 too.

  5. Both write back. Block 9 holds b.txt's bytes, and a.txt points at the wrong data. The disk is corrupted.

  6. Read-only doesn't fix it. A reader's cache goes stale and can catch blocks mid-write. ext4 and xfs assume one kernel per disk. Cluster filesystems like GFS2 and OCFS2 lift that rule with a lock manager, at a real cost.

Clouds enforce this. An EBS volume or a Persistent Disk attaches read-write to one machine at a time, unless you opt into multi-attach and bring a cluster filesystem.

Layer 4 · sandboxes

A sandbox controls what a program can see

There are two common shapes. They differ in where the wall sits.

FIG 4a · containers share one kernelStep 1 of 5
sandbox 1 / ├ app └ tmp = host /srv/sb1 agent.py cgroups · seccomp sandbox 2 / ├ srv └ tmp = host /srv/sb2 server.js cgroups · seccomp sandbox 3 / ├ data └ tmp = host /srv/sb3 test.sh cgroups · seccomp Linux kernel · one, shared by every sandbox above one kernel bug → every sandbox above is exposed open()
  1. Programs can't touch hardware. They ask the kernel for everything, from opening a file to starting a process. Each request is a syscall.

  2. A container is a normal process with a private view. Its mount namespace gives it its own tree and mount table, and its / is a folder on the host.

    $ unshare --mount --pid --fork --root=/srv/sb1 /bin/sh
  3. Cgroups cap its CPU and memory. Seccomp blocks risky syscalls.

    $ echo 4G > /sys/fs/cgroup/sb1/memory.max
  4. Many containers share one kernel. Each sees only its own tree and processes.

  5. That shared kernel is the weak spot. One bug in any of its hundreds of syscalls can expose every neighbor.

FIG 4b · microVMs get a kernel eachStep 1 of 4
microVM · child-7 agent.py its own guest kernel /dev/vda eth0 Firecracker fake hardware virtio-blk host kernel + KVM runs many microVMs side by side child-7.img a plain 10 GB file guest: write block 5 host: write at byte 5 × 4,096 = 20,480 0 20,480 10 GB Two walls to break: guest kernel, then Firecracker block 5
  1. A microVM gives each sandbox its own kernel. The hypervisor, Firecracker here, fakes a small computer with a CPU, memory, a network card and disks.

  2. The disk is a plain file on the host, child-7.img, which the guest sees as /dev/vda. Firecracker is told about it before boot.

    $ curl --unix-socket fc.sock -X PUT localhost/drives/rootfs \
        -d '{"drive_id":"rootfs","path_on_host":"child-7.img","is_root_device":true,"is_read_only":false}'
  3. The guest writes block 5, so Firecracker writes 4,096 bytes at offset 20,480 in child-7.img. Layers 0 to 2 still happen, inside the guest.

  4. Escaping means breaking the guest kernel and then the hypervisor. AWS Lambda, Fly.io and E2B use Firecracker. Modal uses gVisor, a kernel that runs as an ordinary process.

In a container, adding a disk means adding a mount-table row on a shared kernel. In a microVM, it means plugging a fake device into a computer that is already running, which is much harder.

Layer 5 · hotplug

Adding a disk to a running sandbox

Sometimes you only know which disks you need after a sandbox has booted. Adding a block device to a running VM is called hotplug.

FIG 5 · device slotsStep 1 of 4
ANY VM, AT BOOT WITH HOTPLUG · QEMU, CLOUD HYPERVISOR, EC2 FIRECRACKER · NO HOTPLUG parent VM · device slots vda · root disk vdb · scratch vdc · child-7.img empty can't add after boot vdd · child-8.img empty can't add after boot vde · child-9.img empty can't add after boot DISK FILES ON HOST child-5.img child-6.img child-7.img child-8.img child-9.img no free slots left device list frozen child-7 child-8 child-9 child-5 child-6 child-7
  1. Before boot, the host fixes the VM's device list. The guest kernel scans it once while starting up.

  2. With hotplug, the host adds a disk to the running VM. QEMU, Cloud Hypervisor and EC2 volumes support this. /dev/vdc appears and the parent mounts it.

    $ mount -o ro /dev/vdc /mnt/child-7
  3. Slots run out, and each disk needs a ritual: stop the child, detach, attach, mount, read, unmount, detach. The child must finish first, because of layer 3.

  4. Firecracker has no hotplug. Its device list is frozen at boot, so you can't pick disks later.

    $ curl --unix-socket fc.sock -X PUT localhost/drives/child8 -d '{…}'
    The requested operation is not supported after starting the microVM.

Shortcuts that skip the device dance

Containers: with mount propagation, anything the host mounts into a shared folder later appears inside the running container. Kubernetes calls this mountPropagation: HostToContainer.

QEMU and Cloud Hypervisor VMs: virtiofs shares a host folder into the guest, and the host can add to it at any time. Firecracker doesn't support virtiofs. Both shortcuts need everything on one host.

Layer 6 · network filesystems

Network filesystems put one server in charge

Keep exactly one kernel on the disk, and let everyone else talk to it over the network.

FIG 6 · one server, many mountsStep 1 of 6
child-1 mount /runs/42 child-2 mount /runs/42 closed child-3 mount /runs/42 file server the only kernel that touches the disk child-1/ child-2/✓ child-3/ + 997 more children parent mount /runs/42 write read
  1. One machine, the file server, owns the disk and the only real index. That satisfies layer 3.

  2. Sandboxes mount the server over the network. Inside, /runs/42 is just a folder.

    $ mount -t nfs files:/runs/42 /runs/42
  3. Child 2 writes /runs/42/child-2/out.json. The request goes to the server, which does the block work.

  4. Child 2 closes the file. NFS guarantees close-to-open consistency: whoever opens it next sees all of it.

  5. The parent reads the file from the server's single index. There are no stale copies.

  6. A network mount is just a connection. It works any time after boot, needs no device slot, and serves thousands of clients. The sandbox must allow NFS or FUSE mounts.

    $ mount-s3 my-bucket /runs        # FUSE over S3, one of several options

Managed NFS: AWS EFS, Google Filestore. Object storage presented as a folder through FUSE: JuiceFS, mountpoint-s3, gcsfuse.

Layer 7 · putting it together

Mount one place. Let the folders change.

Back to the running example. Give each child a folder inside one shared mount instead of its own disk, and every requirement becomes easy.

FIG 7 · parent + N children, one mountStep 1 of 6
QUEUE · N = 8 TASKS QUEUE · EMPTY writing done RUNNING · AT MOST X = 3 idle idle idle child-1 child-4 writing renamed ✓ child-2 writing child-3 writing 1 2 3 4 5 6 7 8 /runs/42/ one network share or bucket prefix child-1/✓ child-2/✓ child-3/✓ child-4/✓ child-5/✓ child-6/✓ child-7/✓ child-8/✓ parent mounted at boot: /runs/42
  1. The parent boots with one mount, /runs/42, and it never changes.

  2. N = 8 tasks, at most X = 3 running. Each child owns one folder and is its only writer.

  3. Child 1 writes out.json.tmp, then renames it to out.json. Renames are atomic, so the parent never reads half a file. S3 uploads are already atomic.

    $ produce_results > /runs/42/child-1/out.json.tmp
    $ mv /runs/42/child-1/out.json.tmp /runs/42/child-1/out.json
  4. Child 1 exits and its sandbox is destroyed. Its folder stays. Task 4 takes the free slot.

  5. Fast-forward until the queue drains. The parent's mounts never changed.

  6. The parent reads every result whenever it likes, through the same single mount for 8 children or 8,000. A lifecycle rule can expire old runs.

    $ ls /runs/42/*/out.json | wc -l
    8
The takeaway

Mounts are rigid and folders are cheap. Mount one shared place at boot and let folders come and go.

Four ways to build it

The columns are the four requirements from the top of the page.

ApproachOwn storage per child32+ at oncePicked after bootChanges while running
Block disk per child, attached to the parentOne kernel per disk, so each child must finish first. Impossible on Firecracker.yesslots run outhotplug onlyhotplug only
Host folder shared into the parentMount propagation or virtiofs. Needs provider support and one host.yesyesyesyes
Network filesystem, a folder per childNFS, EFS, JuiceFS. The sandbox must allow NFS or FUSE mounts.yesyesyesyes
Object storage, a prefix per childS3, R2, GCS. No mount at all; code uses an API instead of plain files.yesyesyesyes
Recap

Glossary

block
A fixed-size numbered box on a drive, usually 4 KB.
block device
Anything that reads and writes numbered blocks, like /dev/vda.
filesystem
The index that maps names to blocks. ext4 and xfs are examples.
formatting
Writing an empty index onto a drive with mkfs.
mount point
The empty folder where a filesystem gets attached.
mount table
The kernel's list of folder-to-filesystem rows.
page cache
The kernel's RAM copy of disk data and index.
namespace
A private view the kernel gives a process: its own mounts, processes or network.
cgroup
A CPU and memory limit on a group of processes.
hypervisor
Software that fakes hardware so a guest kernel can run. Firecracker is one.
virtio-blk
The fake disk device a VM guest talks to.
hotplug
Adding a device to a VM that is already running.
FUSE
Lets an ordinary program act as a filesystem.
close-to-open
NFS's promise that a closed file is fully visible to the next opener.