htdocs/notes/UDM-NIXOS.md

# How I put NixOS on my UDM (trashcan model) router

Content also available on [https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md](https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md)

Really it's just a running NixOS on systemd-nspawn thing.

The UDM product line basically runs on Linux kernel and userland. It is a 
surprisingly normal device that allows you to SSH and run commands. It even has 
apt and systemd services installed. The only catch being that for the most part
the file system structure is immutable with only a few exceptions like /data and
/etc/systemd. Previous versions even had the Unifi services running on a  podman
container. On recent versions of the firmware podman was phased out but we got
something that resembles a more complete system structure as opposed to a
busybox-like system.

So basically its some kind of Debian-based Linux running on a headless ARM64 
computer. Can we install and run stuff? Yes! In fact projects like 
https://github.com/unifi-utilities/unifios-utilities publish scripts to run 
general purpose programs and configurations on UDM. Be aware however that
firmware  upgrades might wipe the persistent data storage so don't put anything
in there that you don't want to lose and preferably keep scripts so you can
setup again after having its flash storage nuked by a major update.

I have the base UDM model. The first with the pill format that has been 
aparently replaced by the UDR. The UDR seems to have more features like Wifi6,
bigger internal storage and even an SD card slot meant for vigilance  camera
footage storage but comes with a weaker CPU in comparison with the original
UDM base. As far as I know the rack mountable models follow the same OS and
file system structure.


## Okay but why?

I'm gonna leave this to your imagination on why would you add services to your
proprietary router applicance. To me its the fact that I don't really like
running servers at home and I'm ultimately stuck with this router so why not
put it to work maybe running a static webserver or something silly like Home 
Assistant. The truth of the matter is that I can't just leave things alone.

And if you can run Linux why would you run something that is not NixOS? Thats
crazy and it doesn't make sense.

## How do we root the UDM? What kind of jailbreak do I need?

No.

You enable SSH from the Controller UI, log into it as root with the password you
set to the admin user. You just waltz in and start installing and configuring.

```
# apt update && apt install systemd-container 
```

Thats it. Kinda. The complicated part is modifying the programs to write into
the persistent data directories while also making sure your stuff starts on 
boot and doesn't get wiped on minor firmware upgrades.

## Building the NixOS root image.

Might want to read first: [https://nixcademy.com/2023/08/29/nixos-nspawn/](https://nixcademy.com/2023/08/29/nixos-nspawn/)

We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos
contains the flake to build such an image and also publishes artifacts for AMD64
but not  ARM64. I guess you could build this from an AMD64 machine but I haven't
looked into building a cross platform environment (didn't needed to compile
anything though). I have a recent macbook with UTM so I just downloaded one of
the default Linux virtual machine images from the UTM page and installed the 
Nix runtime over the OS.

Make sure you have git and curl installed. 

```
$ sh <(curl -L https://nixos.org/nix/install) --daemon
```

You need to start another terminal session. 

```
$ git clone https://github.com/tfc/nspawn-nixos
$ cd nspawn-nixos
$ nix --extra-experimental-features 'nix-command flakes' build .
```

Optionally you could try to edit the configuration to generate an image with
everything you need. In case you need something like Home Assistant, some 
compilation might be necessary and although I haven't tried compiling code on
the UDM I suspect it might be a painful process due to CPU performance and
space constraints. Here is an [example with Home Assistant, Caddy and Tailscale](https://code.despera.space/iru/nspawn-nixos/src/branch/main/configuration.nix).

The image will be available under
`./result/tarball/nixos-system-aarch64-linux.tar.xz`. Use scp to send this to
the /data/ directory of the UDM.

## Installing the image

First we create the folder structure:

```
# mkdir -p /data/custom/machines
# ln -s /data/custom/machines /var/lib/machines
```

Under normal circunstainces by now you would just run 
`machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz <machinename>`
however the version of tar that is present in this system doesn't really like
the resulting tarball image. It will yeld errors like `Directory renamed before
its status could be extracted`. 

Thankfully we can install bsdtar through `apt install libarchive-tools` however
`machinectl import-tar` is hardcoded to use the tar command. Adding a symlink
from `/usr/bin/bsdtar` to `/usr/local/bin/tar` won't work since some parameters
are used that are not supported in bsdtar. You could try writing a wrapper shell
script but just unpacking the tarball directly was sufficient.

```
# mkdir /var/lib/machines/udmnixos
# bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
```

Lets start the container.

```
# machinectl start udmnixos
# machinectl
MACHINE  CLASS     SERVICE        OS    VERSION ADDRESSES
udmnixos container systemd-nspawn nixos 23.11   192.168.168.88…

```

Good. Now we need to change the root password.

```
# machinectl shell udmnixos /usr/bin/env passwd
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
New password: 
Retype new password: 
passwd: password updated successfully
Connection to machine udmnixos terminated.
```

Finally we can login into the container.

```
# machinectl login nixos
# machinectl login udmnixos
Connected to machine udmnixos. Press ^] three times within 1s to exit session.


<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>


nixos login: root
Password: 

[root@nixos:~]# 
```

We haven't finished yet. By default the network is set to its own container
network. We also don't have a DNS resolver configured. You can leave that
session with CTRL+]]].

https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n

```
# machinectl stop udmnixos

# systemd-nspawn -M udmnixos
Directory /data/custom/machines/udmnixos doesn't look like it has an OS tree. Refusing.
```

## Networking and Persistence

The first thing that needs to be addressed is the DNS configuration. The default
setting that copies the /etc/resolv.conf from host won't work since it points to
localhost. Either install resolved, netmask or set a static DNS config.

As for the network method we have some options here. 

- [Run using the default network stack and map ports to the container](https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p). 
- Run using something akin to --network=host where the container has full access to the host network.
- Give the container its own independent interface through a bridge.
- [Give the container its own independent interface through macvlan](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network).

### Using --network-veth and port mapping

```
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
ResolvConf=off

[Network]
Port=tcp:2222:22
HERE

#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service

# machinectl start udmnixos
```

Remember this will listen on ALL UDM interfaces so you might want to make sure
the firewall rules will accomodate it.

```
# iptables -t nat -L -n -v | grep 2222
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0           !127.0.0.0/8          tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
```

### Using the host network 

This will give access to all the network interfaces. Any service that runs on
the container will be accessible from the UDM interfaces without the need to 
map ports. The container will also have the same IP addresses as the UDM. 

You might want to read about [capabilities](https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=) if you plan on running some VPN 
software like Wireguard or Tailscale.


```
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
#Daring are we?
#Capability=all
ResolvConf=off

[Network]
Private=off
VirtualEthernet=off
HERE

#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service

# machinectl start udmnixos
```

### Using a bridge to give the container its own interface

I had to give some capabilities to the container otherwise it wouldn't properly start. Replace the value of Bridge with the bridge corresponding to the UDM network you want to add. Normally these correspond to the VLAN id of that network. Use `brctl show` to find out.

```
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
Capability=CAP_NET_RAW,CAP_NET_ADMIN
ResolvConf=off

[Network]
Bridge=br2
Private=off
VirtualEthernet=off
HERE

#machinectl enable udmnixos
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service

# machinectl start udmnixos
# machinectl login udmnixos
# machinectl login nixos
Failed to get login PTY: No machine 'nixos' known
root@UDM:/etc/systemd/nspawn# machinectl login udmnixos
Connected to machine udmnixos. Press ^] three times within 1s to exit session.


<<< Welcome to NixOS 23.11.20240518.e7cc617 (aarch64) - pts/1 >>>


nixos login: root
Password: 

[root@nixos:~]# ifconfig 
host0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet [redacted]  netmask 255.255.255.192  broadcast  [redacted] 
        inet6  [redacted]   prefixlen 64  scopeid 0x20<link>
        inet6  [redacted]   prefixlen 64  scopeid 0x0<global>
        ether 92:01:4c:a7:a1:7b  txqueuelen 1000  (Ethernet)
        RX packets 2415  bytes 611986 (597.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 61  bytes 5337 (5.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

```

### MACVLAN isolation and more

Here is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container 
to find out how to setup custom scripts.

## Persistence

As far as I verified by rebooting the UDM many times to write this note all
configurations were preserved. According to [the article on nspawn-containers on the unifies-utilities project](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates)
although `/etc/systemd` and `/data` folders are preserved during firmware upgrades `/var/` and `/usr/` are not and there goes our packages and symlink. Please follow the steps on that 
page to setup persistence across firmware upgrades.