Linux Kernel Module Cheat

13.1. OverlayFS

OverlayFS is a filesystem merged in the Linux kernel in 3.18.

As the name suggests, OverlayFS allows you to merge multiple directories into one. The following minimal runnable examples should give you an intuition on how it works:

We are very interested in this filesystem because we are looking for a way to make host cross compiled executables appear on the guest root / without reboot.

This would have several advantages:

makes it faster to test modified guest programs
- not rebooting is fundamental for gem5, where the reboot is very costly.
- no need to regenerate the root filesystem at all and reboot
- overcomes the check_bin_arch problem as shown at: Section 26.8, “Buildroot rebuild is slow when the root filesystem is large”
we could keep the base root filesystem very small, which implies:
- less host disk usage, no need to copy the entire ./getvar out_rootfs_overlay_dir to the image again
- no need to worry about BR2_TARGET_ROOTFS_EXT2_SIZE

We can already make host files appear on the guest with 9P, but they appear on a subdirectory instead of the root.

If they would appear on the root instead, that would be even more awesome, because you would just use the exact same paths relative to the root transparently.

For example, we wouldn’t have to mess around with variables such as PATH and LD_LIBRARY_PATH.

The idea is to:

9P mount our overlay directory ./getvar out_rootfs_overlay_dir on the guest, which we already do at /mnt/9p/out_rootfs_overlay
then create an overlay with that directory and the root, and chroot into it.

I was unable to mount directly to / avoid the chroot: https://stackoverflow.com/questions/41119656/how-can-i-overlayfs-the-root-filesystem-on-linux https://unix.stackexchange.com/questions/316018/how-to-use-overlayfs-to-protect-the-root-filesystem ** https://unix.stackexchange.com/questions/420646/mount-root-as-overlayfs

We already have a prototype of this running from fstab on guest at /mnt/overlay, but it has the following shortcomings:

changes to underlying filesystems are not visible on the overlay unless you remount with mount -r remount /mnt/overlay, as mentioned on the kernel docs:
```
Changes to the underlying filesystems while part of a mounted overlay
filesystem are not allowed.  If the underlying filesystem is changed,
the behavior of the overlay is undefined, though it will not result in
a crash or deadlock.
```
This makes everything very inconvenient if you are inside chroot action. You would have to leave chroot, remount, then come back.
the overlay does not contain sub-filesystems, e.g. /proc. We would have to re-mount them. But should be doable with some automation.

Even more awesome than chroot would be to pivot_root, but I couldn’t get that working either: