12 KiB
How I put NixOS on my UDM (trashcan model) router
a rare cursed fetch!
Content also available on https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md
Really it's just a running NixOS on systemd-nspawn thing.
The UDM product line basically runs on Linux kernel and userland. It is a surprisingly normal device that allows you to SSH and run commands. It even has apt and systemd services installed. The only catch being that for the most part the file system structure is immutable with only a few exceptions like /data and /etc/systemd. Previous versions even had the Unifi services running on a podman container. On recent versions of the firmware podman was phased out but we got something that resembles a more complete system structure as opposed to a busybox-like system.
So basically its some kind of Debian-based Linux running on a headless ARM64 computer. Can we install and run stuff? Yes! In fact projects like https://github.com/unifi-utilities/unifios-utilities publish scripts to run general purpose programs and configurations on UDM. Be aware however that firmware upgrades might wipe the persistent data storage so don't put anything in there that you don't want to lose and preferably keep scripts so you can setup again after having its flash storage nuked by a major update.
I have the base UDM model. The first with the pill format that has been aparently replaced by the UDR. The UDR seems to have more features like Wifi6, bigger internal storage and even an SD card slot meant for vigilance camera footage storage but comes with a weaker CPU in comparison with the original UDM base. As far as I know the rack mountable models follow the same OS and file system structure.
Okay but why?
I'm gonna leave this to your imagination on why would you add services to your proprietary router applicance. To me its the fact that I don't really like running servers at home and I'm ultimately stuck with this router so why not put it to work maybe running a static webserver or something silly like Home Assistant. The truth of the matter is that I can't just leave things alone.
And if you can run Linux why would you run something that is not NixOS? Thats crazy and it doesn't make sense.
How do we root the UDM? What kind of jailbreak do I need?
No.
You enable SSH from the Controller UI, log into it as root with the password you set to the admin user. You just waltz in and start installing and configuring.
# apt update && apt install systemd-container
Thats it. Kinda. The complicated part is modifying the programs to write into the persistent data directories while also making sure your stuff starts on boot and doesn't get wiped on minor firmware upgrades.
Building the NixOS root image.
Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/
We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos contains the flake to build such an image and also publishes artifacts for AMD64 but not ARM64. I guess you could build this from an AMD64 machine but I haven't looked into building a cross platform environment (didn't needed to compile anything though). I have a recent macbook with UTM so I just downloaded one of the default Linux virtual machine images from the UTM page and installed the Nix runtime over the OS.
Make sure you have git and curl installed.
$ sh <(curl -L https://nixos.org/nix/install) --daemon
You need to start another terminal session.
$ git clone https://github.com/tfc/nspawn-nixos
$ cd nspawn-nixos
$ nix --extra-experimental-features 'nix-command flakes' build .
Optionally you could try to edit the configuration to generate an image with everything you need. In case you need something like Home Assistant, some compilation might be necessary and although I haven't tried compiling code on the UDM I suspect it might be a painful process due to CPU performance and space constraints. Here is an example with Home Assistant, Caddy and Tailscale.
The image will be available under
./result/tarball/nixos-system-aarch64-linux.tar.xz. Use scp to send this to
the /data/ directory of the UDM.
Installing the image
First we create the folder structure:
# mkdir -p /data/custom/machines
# ln -s /data/custom/machines /var/lib/machines
Under normal circunstainces by now you would just run
machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz <machinename>
however the version of tar that is present in this system doesn't really like
the resulting tarball image. It will yeld errors like Directory renamed before its status could be extracted.
Thankfully we can install bsdtar through apt install libarchive-tools however
machinectl import-tar is hardcoded to use the tar command. Adding a symlink
from /usr/bin/bsdtar to /usr/local/bin/tar won't work since some parameters
are used that are not supported in bsdtar. You could try writing a wrapper shell
script but just unpacking the tarball directly was sufficient.
# mkdir /var/lib/machines/udmnixos
# bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
Lets start the container.
# machinectl start udmnixos
# machinectl
MACHINE CLASS SERVICE OS VERSION ADDRESSES
udmnixos container systemd-nspawn nixos 23.11 192.168.168.88…
Good. Now we need to change the root password.
# machinectl shell udmnixos /usr/bin/env passwd
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
New password:
Retype new password:
passwd: password updated successfully
Connection to machine udmnixos terminated.
Finally we can login into the container.
# machinectl login udmnixos
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>
nixos login: root
Password:
[root@nixos:~]#
We haven't finished yet. By default the network is set to its own container network. We also don't have a DNS resolver configured. You can leave that session with CTRL+]]].
https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n
# machinectl stop udmnixos
Networking and Persistence
The first thing that needs to be addressed is the DNS configuration. The default setting that copies the /etc/resolv.conf from host won't work since it points to localhost. Either install resolved, netmask or set a static DNS config.
As for the network method we have some options here.
- Run using the default network stack and map ports to the container.
- Run using something akin to --network=host where the container has full access to the host network.
- Give the container its own independent interface through a bridge.
- Give the container its own independent interface through macvlan.
Using --network-veth and port mapping
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
ResolvConf=off
[Network]
Port=tcp:2222:22
HERE
#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
Remember this will listen on ALL UDM interfaces so you might want to make sure the firewall rules will accomodate it.
# iptables -t nat -L -n -v | grep 2222
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
0 0 DNAT tcp -- * * 0.0.0.0/0 !127.0.0.0/8 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
Using the host network
This will give access to all the network interfaces. Any service that runs on the container will be accessible from the UDM interfaces without the need to map ports. The container will also have the same IP addresses as the UDM.
You might want to read about capabilities if you plan on running some VPN software like Wireguard or Tailscale.
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
#Daring are we?
#Capability=all
ResolvConf=off
[Network]
Private=off
VirtualEthernet=off
HERE
#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
Using a bridge to give the container its own interface
I had to give some capabilities to the container otherwise it wouldn't properly start. Replace the value of Bridge with the bridge corresponding to the UDM network you want to add. Normally these correspond to the VLAN id of that network. Use brctl show to find out.
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
Capability=CAP_NET_RAW,CAP_NET_ADMIN
ResolvConf=off
[Network]
Bridge=br2
Private=off
VirtualEthernet=off
HERE
#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
# machinectl login udmnixos
# machinectl login nixos
Failed to get login PTY: No machine 'nixos' known
root@UDM:/etc/systemd/nspawn# machinectl login udmnixos
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
<<< Welcome to NixOS 23.11.20240518.e7cc617 (aarch64) - pts/1 >>>
nixos login: root
Password:
[root@nixos:~]# ifconfig
host0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet [redacted] netmask 255.255.255.192 broadcast [redacted]
inet6 [redacted] prefixlen 64 scopeid 0x20<link>
inet6 [redacted] prefixlen 64 scopeid 0x0<global>
ether 92:01:4c:a7:a1:7b txqueuelen 1000 (Ethernet)
RX packets 2415 bytes 611986 (597.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 61 bytes 5337 (5.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
MACVLAN isolation and more
Here is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container to find out how to setup custom scripts.
Persistence
As far as I verified by rebooting the UDM many times to write this note all
configurations were preserved. According to the article on nspawn-containers on the unifies-utilities project
although /etc/systemd and /data folders are preserved during firmware upgrades /var/ and /usr/ are not and there goes our packages and symlink. Please follow the steps on that
page to setup persistence across firmware upgrades.