10 KiB
How I put NixOS on my UDM (trashcan model) router
(Really its just a running NixOS on systemd-nspawn thing)
The UDM product line basically runs on Linux kernel and userland. It is a surprisingly normal device that allows you to SSH and run commands. It even has apt and systemd services installed. The only catch being that for the most part the file system structure is immutable with only a few exceptions like /data and /etc/systemd. Previous versions even had the Unifi services running on a podman container. On recent versions of the firmware podman was phased out but we got something that resembles a more complete system structure as opposed to a busybox-like system.
So basically its some kind of Debian-based Linux running on a headless ARM64 computer. Can we install and run stuff? Yes! In fact projects like https://github.com/unifi-utilities/unifios-utilities publish scripts to run general purpose programs and configurations on UDM. Be aware however that firmware upgrades might wipe the persistent data storage so don't put anything in there that you don't want to lose and preferably keep scripts so you can setup again after having its flash storage nuked by a major update.
I have the base UDM model. The first with the pill format that has been aparently replaced by the UDR. The UDR seems to have more features like Wifi6, bigger internal storage and even an SD card slot meant for vigilance camera footage storage but comes with a weaker CPU in comparison with the original UDM base. As far as I know the rack mountable models follow the same OS and file system structure.
Okay but why?
I'm gonna leave this to your imagination on why would you add services to your proprietary router applicance. To me its the fact that I don't really like running servers at home and I'm ultimately stuck with this router so why not put it to work maybe running a static webserver or something silly like Home Assistant. The truth of the matter is that I can't just leave things alone.
And if you can run Linux why would you run something that is not NixOS? Thats crazy and it doesn't make sense.
How do we root the UDM? What kind of jailbreak do I need?
No.
You enable SSH from the Controller UI, log into it as root with the password you set to the admin user. You just waltz in and start installing and configuring.
apt update && apt install systemd-container
Thats it. Kinda. The complicated part is modifying the programs to write into the persistent data directories while also making sure your stuff starts on boot and doesn't get wiped on minor firmware upgrades.
Debian packages are cached on persistent storage and reinstalled at boot. The systemd units are also kept and that gives us plenty of tools to work with.
Building the NixOS root image.
Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/
We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos contains the flake to build such an image and also publishes artifacts for AMD64 but not ARM64. I guess you could build this from an AMD64 machine but I haven't looked into building a cross platform environment (didn't needed to compile anything though). I have a recent macbook with UTM so I just downloaded one of the default Linux virtual machine images from the UTM page and installed the Nix runtime over the OS.
Make sure you have git and curl installed.
$ sh <(curl -L https://nixos.org/nix/install) --daemon
You need to start another terminal session.
$ git clone https://github.com/tfc/nspawn-nixos
$ cd nspawn-nixos
$ nix --extra-experimental-features 'nix-command flakes' build .
Optionally you could try to edit the configuration to generate an image with everything you need. In case you need something like Home Assistant, some compilation might be necessary and although I haven't tried compiling code on the UDM I suspect it might be a painful process due to CPU performance and space constraints.
The image will be available under
./result/tarball/nixos-system-aarch64-linux.tar.xz
. Use scp to send this to
the /data/ directory of the UDM.
Installing the image
First we create the folder structure:
mkdir -p /data/custom/machines
ln -s /data/custom/machines /var/lib/machines
Under normal circunstainces by now you would just run
machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz <machinename>
however the version of tar that is present in this system doesn't really like
the resulting tarball image. It will yeld errors like Directory renamed before its status could be extracted
.
Thankfully we can install bsdtar through apt install libarchive-tools
however
machinectl import-tar
is hardcoded to use the tar command. Adding a symlink
from /usr/bin/bsdtar
to /usr/local/bin/tar
won't work since some parameters
are used that are not supported in bsdtar. You could try writing a wrapper shell
script but just unpacking the tarball directly was sufficient.
mkdir /var/lib/machines/udmnixos
bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
Lets start the container.
# machinectl start udmnixos
# machinectl
MACHINE CLASS SERVICE OS VERSION ADDRESSES
udmnixos container systemd-nspawn nixos 23.11 192.168.168.88…
Good. Now we need to change the root password.
# machinectl shell udmnixos /usr/bin/env passwd
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
New password:
Retype new password:
passwd: password updated successfully
Connection to machine udmnixos terminated.
Finally we can login into the container.
# machinectl login nixos
# machinectl login udmnixos
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>
nixos login: root
Password:
[root@nixos:~]#
We haven't finished yet. By default the network is set to its own container network. We also don't have a DNS resolver configured. You can leave that session with CTRL+]]].
https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n
# machinectl stop udmnixos
# systemd-nspawn -M udmnixos
Directory /data/custom/machines/udmnixos doesn't look like it has an OS tree. Refusing.
Networking and Persistence
The first thing that needs to be addressed is the DNS configuration. The default setting that copies the /etc/resolv.conf from host won't work since it points to localhost. Either install resolved, netmask or set a static DNS config.
Also read the capabilities section if you want to do things like using VPNs like Tailscale.
As for the network method we have some options here.
- Run using the default network stack and map ports to the container. https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p
- Run using something akin to --network=host where the container has full access to the host network.
- Give the container its own independent interface through a bridge.
- Give the container its own independent interface through macvlan. https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network
Using --network-veth and port mapping
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
ResolvConf=off
[Network]
Port=tcp:2222:22
HERE
#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
Remember this will listen on ALL UDM interfaces so you might want to make sure the firewall rules will accomodate it.
# iptables -t nat -L -n -v | grep 2222
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
0 0 DNAT tcp -- * * 0.0.0.0/0 !127.0.0.0/8 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
Using the host network
This will give access to all the network interfaces. Any service that runs on the container will be accessible from the UDM interfaces without the need to map ports. The container will also have the same IP addresses as the UDM.
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
ResolvConf=off
[Network]
Private=off
VirtualEthernet=off
HERE
#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
Using a bridge to give the container its own interface
I had to give some capabilities to the container otherwise it wouldn't start. Feel free to find out what capabilities are needed from https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
Capability=CAP_NET_RAW,CAP_NET_ADMIN
ResolvConf=off
[Network]
Bridge=br2
Private=off
VirtualEthernet=off
HERE
#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
MACVLAN isolation and more
Here is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container to find out how to setup custom scripts.
Persistence
As far as I verified by rebooting the UDM many times to write this note all configurations were preserved. According to https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates although /etc/systemd and /data folders are preserved during firmware upgrades /var/ and /usr/ are not and there goes our packages and symlink. Please follow the steps on that page to setup persistence across firmware upgrades.