309 lines
12 KiB
Markdown
309 lines
12 KiB
Markdown
|
# How I put NixOS on my UDM (trashcan model) router
|
||
|
|
|
||
|
|
|
||
|
|
*a rare cursed fetch!*
|
||
|
|
|
||
|
|
Content also available on [https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md](https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md)
|
||
|
|
|
||
|
|
Really it's just a running NixOS on systemd-nspawn thing.
|
||
|
|
|
||
|
|
The UDM product line basically runs on Linux kernel and userland. It is a
|
||
|
|
surprisingly normal device that allows you to SSH and run commands. It even has
|
||
|
|
apt and systemd services installed. The only catch being that for the most part
|
||
|
|
the file system structure is immutable with only a few exceptions like /data and
|
||
|
|
/etc/systemd. Previous versions even had the Unifi services running on a podman
|
||
|
|
container. On recent versions of the firmware podman was phased out but we got
|
||
|
|
something that resembles a more complete system structure as opposed to a
|
||
|
|
busybox-like system.
|
||
|
|
|
||
|
|
So basically its some kind of Debian-based Linux running on a headless ARM64
|
||
|
|
computer. Can we install and run stuff? Yes! In fact projects like
|
||
|
|
https://github.com/unifi-utilities/unifios-utilities publish scripts to run
|
||
|
|
general purpose programs and configurations on UDM. Be aware however that
|
||
|
|
firmware upgrades might wipe the persistent data storage so don't put anything
|
||
|
|
in there that you don't want to lose and preferably keep scripts so you can
|
||
|
|
setup again after having its flash storage nuked by a major update.
|
||
|
|
|
||
|
|
I have the base UDM model. The first with the pill format that has been
|
||
|
|
aparently replaced by the UDR. The UDR seems to have more features like Wifi6,
|
||
|
|
bigger internal storage and even an SD card slot meant for vigilance camera
|
||
|
|
footage storage but comes with a weaker CPU in comparison with the original
|
||
|
|
UDM base. As far as I know the rack mountable models follow the same OS and
|
||
|
|
file system structure.
|
||
|
|
|
||
|
|
|
||
|
|
## Okay but why?
|
||
|
|
|
||
|
|
I'm gonna leave this to your imagination on why would you add services to your
|
||
|
|
proprietary router applicance. To me its the fact that I don't really like
|
||
|
|
running servers at home and I'm ultimately stuck with this router so why not
|
||
|
|
put it to work maybe running a static webserver or something silly like Home
|
||
|
|
Assistant. The truth of the matter is that I can't just leave things alone.
|
||
|
|
|
||
|
|
And if you can run Linux why would you run something that is not NixOS? Thats
|
||
|
|
crazy and it doesn't make sense.
|
||
|
|
|
||
|
|
## How do we root the UDM? What kind of jailbreak do I need?
|
||
|
|
|
||
|
|
No.
|
||
|
|
|
||
|
|
You enable SSH from the Controller UI, log into it as root with the password you
|
||
|
|
set to the admin user. You just waltz in and start installing and configuring.
|
||
|
|
|
||
|
|
```
|
||
|
|
# apt update && apt install systemd-container
|
||
|
|
```
|
||
|
|
|
||
|
|
Thats it. Kinda. The complicated part is modifying the programs to write into
|
||
|
|
the persistent data directories while also making sure your stuff starts on
|
||
|
|
boot and doesn't get wiped on minor firmware upgrades.
|
||
|
|
|
||
|
|
## Building the NixOS root image.
|
||
|
|
|
||
|
|
Might want to read first: [https://nixcademy.com/2023/08/29/nixos-nspawn/](https://nixcademy.com/2023/08/29/nixos-nspawn/)
|
||
|
|
|
||
|
|
We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos
|
||
|
|
contains the flake to build such an image and also publishes artifacts for AMD64
|
||
|
|
but not ARM64. I guess you could build this from an AMD64 machine but I haven't
|
||
|
|
looked into building a cross platform environment (didn't needed to compile
|
||
|
|
anything though). I have a recent macbook with UTM so I just downloaded one of
|
||
|
|
the default Linux virtual machine images from the UTM page and installed the
|
||
|
|
Nix runtime over the OS.
|
||
|
|
|
||
|
|
Make sure you have git and curl installed.
|
||
|
|
|
||
|
|
```
|
||
|
|
$ sh <(curl -L https://nixos.org/nix/install) --daemon
|
||
|
|
```
|
||
|
|
|
||
|
|
You need to start another terminal session.
|
||
|
|
|
||
|
|
```
|
||
|
|
$ git clone https://github.com/tfc/nspawn-nixos
|
||
|
|
$ cd nspawn-nixos
|
||
|
|
$ nix --extra-experimental-features 'nix-command flakes' build .
|
||
|
|
```
|
||
|
|
|
||
|
|
Optionally you could try to edit the configuration to generate an image with
|
||
|
|
everything you need. In case you need something like Home Assistant, some
|
||
|
|
compilation might be necessary and although I haven't tried compiling code on
|
||
|
|
the UDM I suspect it might be a painful process due to CPU performance and
|
||
|
|
space constraints. Here is an [example with Home Assistant, Caddy and Tailscale](https://code.despera.space/iru/nspawn-nixos/src/branch/main/configuration.nix).
|
||
|
|
|
||
|
|
The image will be available under
|
||
|
|
`./result/tarball/nixos-system-aarch64-linux.tar.xz`. Use scp to send this to
|
||
|
|
the /data/ directory of the UDM.
|
||
|
|
|
||
|
|
## Installing the image
|
||
|
|
|
||
|
|
First we create the folder structure:
|
||
|
|
|
||
|
|
```
|
||
|
|
# mkdir -p /data/custom/machines
|
||
|
|
# ln -s /data/custom/machines /var/lib/machines
|
||
|
|
```
|
||
|
|
|
||
|
|
Under normal circunstainces by now you would just run
|
||
|
|
`machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz <machinename>`
|
||
|
|
however the version of tar that is present in this system doesn't really like
|
||
|
|
the resulting tarball image. It will yeld errors like `Directory renamed before
|
||
|
|
its status could be extracted`.
|
||
|
|
|
||
|
|
Thankfully we can install bsdtar through `apt install libarchive-tools` however
|
||
|
|
`machinectl import-tar` is hardcoded to use the tar command. Adding a symlink
|
||
|
|
from `/usr/bin/bsdtar` to `/usr/local/bin/tar` won't work since some parameters
|
||
|
|
are used that are not supported in bsdtar. You could try writing a wrapper shell
|
||
|
|
script but just unpacking the tarball directly was sufficient.
|
||
|
|
|
||
|
|
```
|
||
|
|
# mkdir /var/lib/machines/udmnixos
|
||
|
|
# bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
|
||
|
|
```
|
||
|
|
|
||
|
|
Lets start the container.
|
||
|
|
|
||
|
|
```
|
||
|
|
# machinectl start udmnixos
|
||
|
|
# machinectl
|
||
|
|
MACHINE CLASS SERVICE OS VERSION ADDRESSES
|
||
|
|
udmnixos container systemd-nspawn nixos 23.11 192.168.168.88…
|
||
|
|
|
||
|
|
```
|
||
|
|
|
||
|
|
Good. Now we need to change the root password.
|
||
|
|
|
||
|
|
```
|
||
|
|
# machinectl shell udmnixos /usr/bin/env passwd
|
||
|
|
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
||
|
|
New password:
|
||
|
|
Retype new password:
|
||
|
|
passwd: password updated successfully
|
||
|
|
Connection to machine udmnixos terminated.
|
||
|
|
```
|
||
|
|
|
||
|
|
Finally we can login into the container.
|
||
|
|
|
||
|
|
```
|
||
|
|
# machinectl login udmnixos
|
||
|
|
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
||
|
|
|
||
|
|
|
||
|
|
<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>
|
||
|
|
|
||
|
|
|
||
|
|
nixos login: root
|
||
|
|
Password:
|
||
|
|
|
||
|
|
[root@nixos:~]#
|
||
|
|
```
|
||
|
|
|
||
|
|
We haven't finished yet. By default the network is set to its own container
|
||
|
|
network. We also don't have a DNS resolver configured. You can leave that
|
||
|
|
session with CTRL+]]].
|
||
|
|
|
||
|
|
https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n
|
||
|
|
|
||
|
|
```
|
||
|
|
# machinectl stop udmnixos
|
||
|
|
```
|
||
|
|
|
||
|
|
## Networking and Persistence
|
||
|
|
|
||
|
|
The first thing that needs to be addressed is the DNS configuration. The default
|
||
|
|
setting that copies the /etc/resolv.conf from host won't work since it points to
|
||
|
|
localhost. Either install resolved, netmask or set a static DNS config.
|
||
|
|
|
||
|
|
As for the network method we have some options here.
|
||
|
|
|
||
|
|
- [Run using the default network stack and map ports to the container](https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p).
|
||
|
|
- Run using something akin to --network=host where the container has full access to the host network.
|
||
|
|
- Give the container its own independent interface through a bridge.
|
||
|
|
- [Give the container its own independent interface through macvlan](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network).
|
||
|
|
|
||
|
|
### Using --network-veth and port mapping
|
||
|
|
|
||
|
|
```
|
||
|
|
# mkdir -p /etc/systemd/nspawn
|
||
|
|
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
||
|
|
[Exec]
|
||
|
|
Boot=on
|
||
|
|
ResolvConf=off
|
||
|
|
|
||
|
|
[Network]
|
||
|
|
Port=tcp:2222:22
|
||
|
|
HERE
|
||
|
|
|
||
|
|
#machinectl enable udmnixos
|
||
|
|
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
||
|
|
|
||
|
|
# machinectl start udmnixos
|
||
|
|
```
|
||
|
|
|
||
|
|
Remember this will listen on ALL UDM interfaces so you might want to make sure
|
||
|
|
the firewall rules will accomodate it.
|
||
|
|
|
||
|
|
```
|
||
|
|
# iptables -t nat -L -n -v | grep 2222
|
||
|
|
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
|
||
|
|
0 0 DNAT tcp -- * * 0.0.0.0/0 !127.0.0.0/8 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
|
||
|
|
```
|
||
|
|
|
||
|
|
### Using the host network
|
||
|
|
|
||
|
|
This will give access to all the network interfaces. Any service that runs on
|
||
|
|
the container will be accessible from the UDM interfaces without the need to
|
||
|
|
map ports. The container will also have the same IP addresses as the UDM.
|
||
|
|
|
||
|
|
You might want to read about [capabilities](https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=) if you plan on running some VPN
|
||
|
|
software like Wireguard or Tailscale.
|
||
|
|
|
||
|
|
|
||
|
|
```
|
||
|
|
# mkdir -p /etc/systemd/nspawn
|
||
|
|
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
||
|
|
[Exec]
|
||
|
|
Boot=on
|
||
|
|
#Daring are we?
|
||
|
|
#Capability=all
|
||
|
|
ResolvConf=off
|
||
|
|
|
||
|
|
[Network]
|
||
|
|
Private=off
|
||
|
|
VirtualEthernet=off
|
||
|
|
HERE
|
||
|
|
|
||
|
|
#machinectl enable udmnixos
|
||
|
|
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
||
|
|
|
||
|
|
# machinectl start udmnixos
|
||
|
|
```
|
||
|
|
|
||
|
|
### Using a bridge to give the container its own interface
|
||
|
|
|
||
|
|
I had to give some capabilities to the container otherwise it wouldn't properly start. Replace the value of Bridge with the bridge corresponding to the UDM network you want to add. Normally these correspond to the VLAN id of that network. Use `brctl show` to find out.
|
||
|
|
|
||
|
|
```
|
||
|
|
# mkdir -p /etc/systemd/nspawn
|
||
|
|
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
||
|
|
[Exec]
|
||
|
|
Boot=on
|
||
|
|
Capability=CAP_NET_RAW,CAP_NET_ADMIN
|
||
|
|
ResolvConf=off
|
||
|
|
|
||
|
|
[Network]
|
||
|
|
Bridge=br2
|
||
|
|
Private=off
|
||
|
|
VirtualEthernet=off
|
||
|
|
HERE
|
||
|
|
|
||
|
|
#machinectl enable udmnixos
|
||
|
|
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
||
|
|
|
||
|
|
# machinectl start udmnixos
|
||
|
|
# machinectl login udmnixos
|
||
|
|
# machinectl login nixos
|
||
|
|
Failed to get login PTY: No machine 'nixos' known
|
||
|
|
root@UDM:/etc/systemd/nspawn# machinectl login udmnixos
|
||
|
|
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
||
|
|
|
||
|
|
|
||
|
|
<<< Welcome to NixOS 23.11.20240518.e7cc617 (aarch64) - pts/1 >>>
|
||
|
|
|
||
|
|
|
||
|
|
nixos login: root
|
||
|
|
Password:
|
||
|
|
|
||
|
|
[root@nixos:~]# ifconfig
|
||
|
|
host0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
|
||
|
|
inet [redacted] netmask 255.255.255.192 broadcast [redacted]
|
||
|
|
inet6 [redacted] prefixlen 64 scopeid 0x20<link>
|
||
|
|
inet6 [redacted] prefixlen 64 scopeid 0x0<global>
|
||
|
|
ether 92:01:4c:a7:a1:7b txqueuelen 1000 (Ethernet)
|
||
|
|
RX packets 2415 bytes 611986 (597.6 KiB)
|
||
|
|
RX errors 0 dropped 0 overruns 0 frame 0
|
||
|
|
TX packets 61 bytes 5337 (5.2 KiB)
|
||
|
|
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
|
||
|
|
|
||
|
|
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
|
||
|
|
inet 127.0.0.1 netmask 255.0.0.0
|
||
|
|
inet6 ::1 prefixlen 128 scopeid 0x10<host>
|
||
|
|
loop txqueuelen 1000 (Local Loopback)
|
||
|
|
RX packets 0 bytes 0 (0.0 B)
|
||
|
|
RX errors 0 dropped 0 overruns 0 frame 0
|
||
|
|
TX packets 0 bytes 0 (0.0 B)
|
||
|
|
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
|
||
|
|
|
||
|
|
```
|
||
|
|
|
||
|
|
### MACVLAN isolation and more
|
||
|
|
|
||
|
|
Here is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container
|
||
|
|
to find out how to setup custom scripts.
|
||
|
|
|
||
|
|
## Persistence
|
||
|
|
|
||
|
|
As far as I verified by rebooting the UDM many times to write this note all
|
||
|
|
configurations were preserved. According to [the article on nspawn-containers on the unifies-utilities project](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates)
|
||
|
|
although `/etc/systemd` and `/data` folders are preserved during firmware upgrades `/var/` and `/usr/` are not and there goes our packages and symlink. Please follow the steps on that
|
||
|
|
page to setup persistence across firmware upgrades.
|