Managing multiple OpenBSD systems with a single base install
I am working on an OpenBSD-based QubesOS TemplateVM, and have run
into a few problems.
In QubesOS, all volumes of a TemplateVM are persistent. AppVMs based
on a TemplateVM use a (copy of) the TemplateVM’s root partition,
but have their own private partition, which is set to zero when the
VM first boots up. Finally, DispVMs have no persistent storage at all.
This leads to a few difficulties. First, I found that I need to mount
various directories from the persistent volume (which I mounted at
/mnt/rw) over the root volumes (mounted at / and /usr in my case).
Linux AppVMs use bind mounts for that purpose, but OpenBSD doesn’t
have bind mounts, so I use NFS over loopback. Since that is slow,
I symlinked /home to /mnt/rw/home. Furthermore, when an AppVM first
starts up, /mnt/rw will not be mountable, as the underlying storage
will consist entirely of zeroes. I wrote a C program that uses
disklabel(5) ioctls to handle this. However, this breaks sysupgrade(8),
since /home is symlinked to /mnt/rw/home, which doesn’t exist until
/etc/rc.securelevel runs. I fixed this by monkeypatching sysupgrade(8)
to download into /var/_sysupgrade instead, but that is an ugly hack.
The other problem is that I do not know of a reliable way to make
the boot process fail if my /etc/rc.securelevel fails. This is
security-critical, as it does tasks like deleting SSH keys from
/etc/ssh so that they will be regenerated after /mnt/rw/export/etc/ssh
has been mounted over /etc/ssh. If this fails, there is a chance that
two AppVMs could share the same SSH host keys, which would be bad.
Furthermore, if mounting /mnt/rw fails, I would much prefer to drop
into a single-user shell than to try to continue. This is a trivial
patch to /etc/rc, but patching /etc/rc is strongly discouraged in
Are the solutions I came up with the best possible, or are there
better ones that I missed? Any help will be greatly appreciated.