diff --git a/notes/UDM-NIXOS.md b/notes/UDM-NIXOS.md new file mode 100644 index 0000000..e5e6c7d --- /dev/null +++ b/notes/UDM-NIXOS.md @@ -0,0 +1,277 @@ +# How I put NixOS on my UDM (trashcan model) router + +(Really its just a running NixOS on systemd-nspawn thing) + +The UDM product line basically runs on Linux kernel and userland. It is a +surprisingly normal device that allows you to SSH and run commands. It even has +apt and systemd services installed. The only catch being that for the most part +the file system structure is immutable with only a few exceptions like /data and +/etc/systemd. Previous versions even had the Unifi services running on a podman +container. On recent versions of the firmware podman was phased out but we got +something that resembles a more complete system structure as opposed to a +busybox-like system. + +So basically its some kind of Debian-based Linux running on a headless ARM64 +computer. Can we install and run stuff? Yes! In fact projects like +https://github.com/unifi-utilities/unifios-utilities publish scripts to run +general purpose programs and configurations on UDM. Be aware however that +firmware upgrades might wipe the persistent data storage so don't put anything +in there that you don't want to lose and preferably keep scripts so you can +setup again after having its flash storage nuked by a major update. + +I have the base UDM model. The first with the pill format that has been +aparently replaced by the UDR. The UDR seems to have more features like Wifi6, +bigger internal storage and even an SD card slot meant for vigilance camera +footage storage but comes with a weaker CPU in comparison with the original +UDM base. As far as I know the rack mountable models follow the same OS and +file system structure. + + +## Okay but why? + +I'm gonna leave this to your imagination on why would you add services to your +proprietary router applicance. To me its the fact that I don't really like +running servers at home and I'm ultimately stuck with this router so why not +put it to work maybe running a static webserver or something silly like Home +Assistant. The truth of the matter is that I can't just leave things alone. + +And if you can run Linux why would you run something that is not NixOS? Thats +crazy and it doesn't make sense. + +## How do we root the UDM? What kind of jailbreak do I need? + +No. + +You enable SSH from the Controller UI, log into it as root with the password you +set to the admin user. You just waltz in and start installing and configuring. + +``` +apt update && apt install systemd-container +``` + +Thats it. Kinda. The complicated part is modifying the programs to write into +the persistent data directories while also making sure your stuff starts on +boot and doesn't get wiped on minor firmware upgrades. + +Debian packages are cached on persistent storage and reinstalled at boot. The +systemd units are also kept and that gives us plenty of tools to work with. + +## Building the NixOS root image. + +Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/ + +We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos +contains the flake to build such an image and also publishes artifacts for AMD64 +but not ARM64. I guess you could build this from an AMD64 machine but I haven't +looked into building a cross platform environment (didn't needed to compile +anything though). I have a recent macbook with UTM so I just downloaded one of +the default Linux virtual machine images from the UTM page and installed the +Nix runtime over the OS. + +Make sure you have git and curl installed. + +``` +$ sh <(curl -L https://nixos.org/nix/install) --daemon +``` + +You need to start another terminal session. + +``` +$ git clone https://github.com/tfc/nspawn-nixos +$ cd nspawn-nixos +$ nix --extra-experimental-features 'nix-command flakes' build . +``` + +Optionally you could try to edit the configuration to generate an image with +everything you need. In case you need something like Home Assistant, some +compilation might be necessary and although I haven't tried compiling code on +the UDM I suspect it might be a painful process due to CPU performance and +space constraints. + +The image will be available under +`./result/tarball/nixos-system-aarch64-linux.tar.xz`. Use scp to send this to +the /data/ directory of the UDM. + +## Installing the image + +First we create the folder structure: + +``` +mkdir -p /data/custom/machines +ln -s /data/custom/machines /var/lib/machines +``` + +Under normal circunstainces by now you would just run +`machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz ` +however the version of tar that is present in this system doesn't really like +the resulting tarball image. It will yeld errors like `Directory renamed before +its status could be extracted`. + +Thankfully we can install bsdtar through `apt install libarchive-tools` however +`machinectl import-tar` is hardcoded to use the tar command. Adding a symlink +from `/usr/bin/bsdtar` to `/usr/local/bin/tar` won't work since some parameters +are used that are not supported in bsdtar. You could try writing a wrapper shell +script but just unpacking the tarball directly was sufficient. + +``` +mkdir /var/lib/machines/udmnixos +bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos +``` + +Lets start the container. + +``` +# machinectl start udmnixos +# machinectl +MACHINE CLASS SERVICE OS VERSION ADDRESSES +udmnixos container systemd-nspawn nixos 23.11 192.168.168.88… + +``` + +Good. Now we need to change the root password. + +``` +# machinectl shell udmnixos /usr/bin/env passwd +Connected to machine udmnixos. Press ^] three times within 1s to exit session. +New password: +Retype new password: +passwd: password updated successfully +Connection to machine udmnixos terminated. +``` + +Finally we can login into the container. + +``` +# machinectl login nixos +# machinectl login udmnixos +Connected to machine udmnixos. Press ^] three times within 1s to exit session. + + +<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>> + + +nixos login: root +Password: + +[root@nixos:~]# +``` + +We haven't finished yet. By default the network is set to its own container +network. We also don't have a DNS resolver configured. You can leave that +session with CTRL+]]]. + +https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n + +``` +# machinectl stop udmnixos + +# systemd-nspawn -M udmnixos +Directory /data/custom/machines/udmnixos doesn't look like it has an OS tree. Refusing. +``` + +## Networking and Persistence + +The first thing that needs to be addressed is the DNS configuration. The default +setting that copies the /etc/resolv.conf from host won't work since it points to +localhost. Either install resolved, netmask or set a static DNS config. + +Also read the capabilities section if you want to do things like using VPNs like +Tailscale. + +As for the network method we have some options here. + +- Run using the default network stack and map ports to the container. https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p +- Run using something akin to --network=host where the container has full access to the host network. +- Give the container its own independent interface through a bridge. +- Give the container its own independent interface through macvlan. https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network + +### Using --network-veth and port mapping + +``` +# mkdir -p /etc/systemd/nspawn +# cat > /etc/systemd/nspawn/udmnixos.nspawn < /etc/systemd/nspawn/udmnixos.nspawn < /etc/systemd/nspawn/udmnixos.nspawn <