From a09c2872ea8867e3d556c3ab1a9e6483ca72c79a Mon Sep 17 00:00:00 2001 From: Iru Sensei Date: Sun, 19 May 2024 15:54:26 +0200 Subject: [PATCH] udm-nixos --- notes/UDM-NIXOS.md | 72 +++++++++++++++++++++++++++++++++------------- 1 file changed, 52 insertions(+), 20 deletions(-) diff --git a/notes/UDM-NIXOS.md b/notes/UDM-NIXOS.md index e5e6c7d..7a9284a 100644 --- a/notes/UDM-NIXOS.md +++ b/notes/UDM-NIXOS.md @@ -1,6 +1,8 @@ # How I put NixOS on my UDM (trashcan model) router -(Really its just a running NixOS on systemd-nspawn thing) +Content also available on [https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md](https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md) + +Really it's just a running NixOS on systemd-nspawn thing. The UDM product line basically runs on Linux kernel and userland. It is a surprisingly normal device that allows you to SSH and run commands. It even has @@ -46,19 +48,16 @@ You enable SSH from the Controller UI, log into it as root with the password you set to the admin user. You just waltz in and start installing and configuring. ``` -apt update && apt install systemd-container +# apt update && apt install systemd-container ``` Thats it. Kinda. The complicated part is modifying the programs to write into the persistent data directories while also making sure your stuff starts on boot and doesn't get wiped on minor firmware upgrades. -Debian packages are cached on persistent storage and reinstalled at boot. The -systemd units are also kept and that gives us plenty of tools to work with. - ## Building the NixOS root image. -Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/ +Might want to read first: [https://nixcademy.com/2023/08/29/nixos-nspawn/](https://nixcademy.com/2023/08/29/nixos-nspawn/) We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos contains the flake to build such an image and also publishes artifacts for AMD64 @@ -97,8 +96,8 @@ the /data/ directory of the UDM. First we create the folder structure: ``` -mkdir -p /data/custom/machines -ln -s /data/custom/machines /var/lib/machines +# mkdir -p /data/custom/machines +# ln -s /data/custom/machines /var/lib/machines ``` Under normal circunstainces by now you would just run @@ -114,8 +113,8 @@ are used that are not supported in bsdtar. You could try writing a wrapper shell script but just unpacking the tarball directly was sufficient. ``` -mkdir /var/lib/machines/udmnixos -bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos +# mkdir /var/lib/machines/udmnixos +# bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos ``` Lets start the container. @@ -175,15 +174,12 @@ The first thing that needs to be addressed is the DNS configuration. The default setting that copies the /etc/resolv.conf from host won't work since it points to localhost. Either install resolved, netmask or set a static DNS config. -Also read the capabilities section if you want to do things like using VPNs like -Tailscale. - As for the network method we have some options here. -- Run using the default network stack and map ports to the container. https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p +- [Run using the default network stack and map ports to the container](https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p). - Run using something akin to --network=host where the container has full access to the host network. - Give the container its own independent interface through a bridge. -- Give the container its own independent interface through macvlan. https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network +- [Give the container its own independent interface through macvlan](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network). ### Using --network-veth and port mapping @@ -219,13 +215,17 @@ This will give access to all the network interfaces. Any service that runs on the container will be accessible from the UDM interfaces without the need to map ports. The container will also have the same IP addresses as the UDM. -https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability= +You might want to read about [capabilities](https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=) if you plan on running some VPN +software like Wireguard or Tailscale. + ``` # mkdir -p /etc/systemd/nspawn # cat > /etc/systemd/nspawn/udmnixos.nspawn <>> + + +nixos login: root +Password: + +[root@nixos:~]# ifconfig +host0: flags=4163 mtu 1500 + inet [redacted] netmask 255.255.255.192 broadcast [redacted] + inet6 [redacted] prefixlen 64 scopeid 0x20 + inet6 [redacted] prefixlen 64 scopeid 0x0 + ether 92:01:4c:a7:a1:7b txqueuelen 1000 (Ethernet) + RX packets 2415 bytes 611986 (597.6 KiB) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 61 bytes 5337 (5.2 KiB) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 + +lo: flags=73 mtu 65536 + inet 127.0.0.1 netmask 255.0.0.0 + inet6 ::1 prefixlen 128 scopeid 0x10 + loop txqueuelen 1000 (Local Loopback) + RX packets 0 bytes 0 (0.0 B) + RX errors 0 dropped 0 overruns 0 frame 0 + TX packets 0 bytes 0 (0.0 B) + TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 + ``` ### MACVLAN isolation and more @@ -272,6 +304,6 @@ to find out how to setup custom scripts. ## Persistence As far as I verified by rebooting the UDM many times to write this note all -configurations were preserved. According to https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates -although /etc/systemd and /data folders are preserved during firmware upgrades /var/ and /usr/ are not and there goes our packages and symlink. Please follow the steps on that +configurations were preserved. According to [the article on nspawn-containers on the unifies-utilities project](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates) +although `/etc/systemd` and `/data` folders are preserved during firmware upgrades `/var/` and `/usr/` are not and there goes our packages and symlink. Please follow the steps on that page to setup persistence across firmware upgrades.