278 lines
10 KiB
Markdown
278 lines
10 KiB
Markdown
# How I put NixOS on my UDM (trashcan model) router
|
|
|
|
(Really its just a running NixOS on systemd-nspawn thing)
|
|
|
|
The UDM product line basically runs on Linux kernel and userland. It is a
|
|
surprisingly normal device that allows you to SSH and run commands. It even has
|
|
apt and systemd services installed. The only catch being that for the most part
|
|
the file system structure is immutable with only a few exceptions like /data and
|
|
/etc/systemd. Previous versions even had the Unifi services running on a podman
|
|
container. On recent versions of the firmware podman was phased out but we got
|
|
something that resembles a more complete system structure as opposed to a
|
|
busybox-like system.
|
|
|
|
So basically its some kind of Debian-based Linux running on a headless ARM64
|
|
computer. Can we install and run stuff? Yes! In fact projects like
|
|
https://github.com/unifi-utilities/unifios-utilities publish scripts to run
|
|
general purpose programs and configurations on UDM. Be aware however that
|
|
firmware upgrades might wipe the persistent data storage so don't put anything
|
|
in there that you don't want to lose and preferably keep scripts so you can
|
|
setup again after having its flash storage nuked by a major update.
|
|
|
|
I have the base UDM model. The first with the pill format that has been
|
|
aparently replaced by the UDR. The UDR seems to have more features like Wifi6,
|
|
bigger internal storage and even an SD card slot meant for vigilance camera
|
|
footage storage but comes with a weaker CPU in comparison with the original
|
|
UDM base. As far as I know the rack mountable models follow the same OS and
|
|
file system structure.
|
|
|
|
|
|
## Okay but why?
|
|
|
|
I'm gonna leave this to your imagination on why would you add services to your
|
|
proprietary router applicance. To me its the fact that I don't really like
|
|
running servers at home and I'm ultimately stuck with this router so why not
|
|
put it to work maybe running a static webserver or something silly like Home
|
|
Assistant. The truth of the matter is that I can't just leave things alone.
|
|
|
|
And if you can run Linux why would you run something that is not NixOS? Thats
|
|
crazy and it doesn't make sense.
|
|
|
|
## How do we root the UDM? What kind of jailbreak do I need?
|
|
|
|
No.
|
|
|
|
You enable SSH from the Controller UI, log into it as root with the password you
|
|
set to the admin user. You just waltz in and start installing and configuring.
|
|
|
|
```
|
|
apt update && apt install systemd-container
|
|
```
|
|
|
|
Thats it. Kinda. The complicated part is modifying the programs to write into
|
|
the persistent data directories while also making sure your stuff starts on
|
|
boot and doesn't get wiped on minor firmware upgrades.
|
|
|
|
Debian packages are cached on persistent storage and reinstalled at boot. The
|
|
systemd units are also kept and that gives us plenty of tools to work with.
|
|
|
|
## Building the NixOS root image.
|
|
|
|
Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/
|
|
|
|
We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos
|
|
contains the flake to build such an image and also publishes artifacts for AMD64
|
|
but not ARM64. I guess you could build this from an AMD64 machine but I haven't
|
|
looked into building a cross platform environment (didn't needed to compile
|
|
anything though). I have a recent macbook with UTM so I just downloaded one of
|
|
the default Linux virtual machine images from the UTM page and installed the
|
|
Nix runtime over the OS.
|
|
|
|
Make sure you have git and curl installed.
|
|
|
|
```
|
|
$ sh <(curl -L https://nixos.org/nix/install) --daemon
|
|
```
|
|
|
|
You need to start another terminal session.
|
|
|
|
```
|
|
$ git clone https://github.com/tfc/nspawn-nixos
|
|
$ cd nspawn-nixos
|
|
$ nix --extra-experimental-features 'nix-command flakes' build .
|
|
```
|
|
|
|
Optionally you could try to edit the configuration to generate an image with
|
|
everything you need. In case you need something like Home Assistant, some
|
|
compilation might be necessary and although I haven't tried compiling code on
|
|
the UDM I suspect it might be a painful process due to CPU performance and
|
|
space constraints.
|
|
|
|
The image will be available under
|
|
`./result/tarball/nixos-system-aarch64-linux.tar.xz`. Use scp to send this to
|
|
the /data/ directory of the UDM.
|
|
|
|
## Installing the image
|
|
|
|
First we create the folder structure:
|
|
|
|
```
|
|
mkdir -p /data/custom/machines
|
|
ln -s /data/custom/machines /var/lib/machines
|
|
```
|
|
|
|
Under normal circunstainces by now you would just run
|
|
`machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz <machinename>`
|
|
however the version of tar that is present in this system doesn't really like
|
|
the resulting tarball image. It will yeld errors like `Directory renamed before
|
|
its status could be extracted`.
|
|
|
|
Thankfully we can install bsdtar through `apt install libarchive-tools` however
|
|
`machinectl import-tar` is hardcoded to use the tar command. Adding a symlink
|
|
from `/usr/bin/bsdtar` to `/usr/local/bin/tar` won't work since some parameters
|
|
are used that are not supported in bsdtar. You could try writing a wrapper shell
|
|
script but just unpacking the tarball directly was sufficient.
|
|
|
|
```
|
|
mkdir /var/lib/machines/udmnixos
|
|
bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
|
|
```
|
|
|
|
Lets start the container.
|
|
|
|
```
|
|
# machinectl start udmnixos
|
|
# machinectl
|
|
MACHINE CLASS SERVICE OS VERSION ADDRESSES
|
|
udmnixos container systemd-nspawn nixos 23.11 192.168.168.88…
|
|
|
|
```
|
|
|
|
Good. Now we need to change the root password.
|
|
|
|
```
|
|
# machinectl shell udmnixos /usr/bin/env passwd
|
|
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
|
New password:
|
|
Retype new password:
|
|
passwd: password updated successfully
|
|
Connection to machine udmnixos terminated.
|
|
```
|
|
|
|
Finally we can login into the container.
|
|
|
|
```
|
|
# machinectl login nixos
|
|
# machinectl login udmnixos
|
|
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
|
|
|
|
|
<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>
|
|
|
|
|
|
nixos login: root
|
|
Password:
|
|
|
|
[root@nixos:~]#
|
|
```
|
|
|
|
We haven't finished yet. By default the network is set to its own container
|
|
network. We also don't have a DNS resolver configured. You can leave that
|
|
session with CTRL+]]].
|
|
|
|
https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n
|
|
|
|
```
|
|
# machinectl stop udmnixos
|
|
|
|
# systemd-nspawn -M udmnixos
|
|
Directory /data/custom/machines/udmnixos doesn't look like it has an OS tree. Refusing.
|
|
```
|
|
|
|
## Networking and Persistence
|
|
|
|
The first thing that needs to be addressed is the DNS configuration. The default
|
|
setting that copies the /etc/resolv.conf from host won't work since it points to
|
|
localhost. Either install resolved, netmask or set a static DNS config.
|
|
|
|
Also read the capabilities section if you want to do things like using VPNs like
|
|
Tailscale.
|
|
|
|
As for the network method we have some options here.
|
|
|
|
- Run using the default network stack and map ports to the container. https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p
|
|
- Run using something akin to --network=host where the container has full access to the host network.
|
|
- Give the container its own independent interface through a bridge.
|
|
- Give the container its own independent interface through macvlan. https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network
|
|
|
|
### Using --network-veth and port mapping
|
|
|
|
```
|
|
# mkdir -p /etc/systemd/nspawn
|
|
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
|
[Exec]
|
|
Boot=on
|
|
ResolvConf=off
|
|
|
|
[Network]
|
|
Port=tcp:2222:22
|
|
HERE
|
|
|
|
#machinectl enable udmnixos
|
|
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
|
|
|
# machinectl start udmnixos
|
|
```
|
|
|
|
Remember this will listen on ALL UDM interfaces so you might want to make sure
|
|
the firewall rules will accomodate it.
|
|
|
|
```
|
|
# iptables -t nat -L -n -v | grep 2222
|
|
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
|
|
0 0 DNAT tcp -- * * 0.0.0.0/0 !127.0.0.0/8 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
|
|
```
|
|
|
|
### Using the host network
|
|
|
|
This will give access to all the network interfaces. Any service that runs on
|
|
the container will be accessible from the UDM interfaces without the need to
|
|
map ports. The container will also have the same IP addresses as the UDM.
|
|
|
|
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
|
|
|
|
```
|
|
# mkdir -p /etc/systemd/nspawn
|
|
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
|
[Exec]
|
|
Boot=on
|
|
ResolvConf=off
|
|
|
|
[Network]
|
|
Private=off
|
|
VirtualEthernet=off
|
|
HERE
|
|
|
|
#machinectl enable udmnixos
|
|
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
|
|
|
# machinectl start udmnixos
|
|
```
|
|
|
|
### Using a bridge to give the container its own interface
|
|
|
|
I had to give some capabilities to the container otherwise it wouldn't start. Feel free to find out what capabilities are needed from
|
|
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
|
|
|
|
```
|
|
# mkdir -p /etc/systemd/nspawn
|
|
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
|
[Exec]
|
|
Boot=on
|
|
Capability=CAP_NET_RAW,CAP_NET_ADMIN
|
|
ResolvConf=off
|
|
|
|
[Network]
|
|
Bridge=br2
|
|
Private=off
|
|
VirtualEthernet=off
|
|
HERE
|
|
|
|
#machinectl enable udmnixos
|
|
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
|
|
|
# machinectl start udmnixos
|
|
```
|
|
|
|
### MACVLAN isolation and more
|
|
|
|
Here is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container
|
|
to find out how to setup custom scripts.
|
|
|
|
## Persistence
|
|
|
|
As far as I verified by rebooting the UDM many times to write this note all
|
|
configurations were preserved. According to https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates
|
|
although /etc/systemd and /data folders are preserved during firmware upgrades /var/ and /usr/ are not and there goes our packages and symlink. Please follow the steps on that
|
|
page to setup persistence across firmware upgrades.
|