udm-nixos
This commit is contained in:
parent
82e511a2ed
commit
f9fbd200d7
277
notes/UDM-NIXOS.md
Normal file
277
notes/UDM-NIXOS.md
Normal file
|
|
@ -0,0 +1,277 @@
|
|||
# How I put NixOS on my UDM (trashcan model) router
|
||||
|
||||
(Really its just a running NixOS on systemd-nspawn thing)
|
||||
|
||||
The UDM product line basically runs on Linux kernel and userland. It is a
|
||||
surprisingly normal device that allows you to SSH and run commands. It even has
|
||||
apt and systemd services installed. The only catch being that for the most part
|
||||
the file system structure is immutable with only a few exceptions like /data and
|
||||
/etc/systemd. Previous versions even had the Unifi services running on a podman
|
||||
container. On recent versions of the firmware podman was phased out but we got
|
||||
something that resembles a more complete system structure as opposed to a
|
||||
busybox-like system.
|
||||
|
||||
So basically its some kind of Debian-based Linux running on a headless ARM64
|
||||
computer. Can we install and run stuff? Yes! In fact projects like
|
||||
https://github.com/unifi-utilities/unifios-utilities publish scripts to run
|
||||
general purpose programs and configurations on UDM. Be aware however that
|
||||
firmware upgrades might wipe the persistent data storage so don't put anything
|
||||
in there that you don't want to lose and preferably keep scripts so you can
|
||||
setup again after having its flash storage nuked by a major update.
|
||||
|
||||
I have the base UDM model. The first with the pill format that has been
|
||||
aparently replaced by the UDR. The UDR seems to have more features like Wifi6,
|
||||
bigger internal storage and even an SD card slot meant for vigilance camera
|
||||
footage storage but comes with a weaker CPU in comparison with the original
|
||||
UDM base. As far as I know the rack mountable models follow the same OS and
|
||||
file system structure.
|
||||
|
||||
|
||||
## Okay but why?
|
||||
|
||||
I'm gonna leave this to your imagination on why would you add services to your
|
||||
proprietary router applicance. To me its the fact that I don't really like
|
||||
running servers at home and I'm ultimately stuck with this router so why not
|
||||
put it to work maybe running a static webserver or something silly like Home
|
||||
Assistant. The truth of the matter is that I can't just leave things alone.
|
||||
|
||||
And if you can run Linux why would you run something that is not NixOS? Thats
|
||||
crazy and it doesn't make sense.
|
||||
|
||||
## How do we root the UDM? What kind of jailbreak do I need?
|
||||
|
||||
No.
|
||||
|
||||
You enable SSH from the Controller UI, log into it as root with the password you
|
||||
set to the admin user. You just waltz in and start installing and configuring.
|
||||
|
||||
```
|
||||
apt update && apt install systemd-container
|
||||
```
|
||||
|
||||
Thats it. Kinda. The complicated part is modifying the programs to write into
|
||||
the persistent data directories while also making sure your stuff starts on
|
||||
boot and doesn't get wiped on minor firmware upgrades.
|
||||
|
||||
Debian packages are cached on persistent storage and reinstalled at boot. The
|
||||
systemd units are also kept and that gives us plenty of tools to work with.
|
||||
|
||||
## Building the NixOS root image.
|
||||
|
||||
Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/
|
||||
|
||||
We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos
|
||||
contains the flake to build such an image and also publishes artifacts for AMD64
|
||||
but not ARM64. I guess you could build this from an AMD64 machine but I haven't
|
||||
looked into building a cross platform environment (didn't needed to compile
|
||||
anything though). I have a recent macbook with UTM so I just downloaded one of
|
||||
the default Linux virtual machine images from the UTM page and installed the
|
||||
Nix runtime over the OS.
|
||||
|
||||
Make sure you have git and curl installed.
|
||||
|
||||
```
|
||||
$ sh <(curl -L https://nixos.org/nix/install) --daemon
|
||||
```
|
||||
|
||||
You need to start another terminal session.
|
||||
|
||||
```
|
||||
$ git clone https://github.com/tfc/nspawn-nixos
|
||||
$ cd nspawn-nixos
|
||||
$ nix --extra-experimental-features 'nix-command flakes' build .
|
||||
```
|
||||
|
||||
Optionally you could try to edit the configuration to generate an image with
|
||||
everything you need. In case you need something like Home Assistant, some
|
||||
compilation might be necessary and although I haven't tried compiling code on
|
||||
the UDM I suspect it might be a painful process due to CPU performance and
|
||||
space constraints.
|
||||
|
||||
The image will be available under
|
||||
`./result/tarball/nixos-system-aarch64-linux.tar.xz`. Use scp to send this to
|
||||
the /data/ directory of the UDM.
|
||||
|
||||
## Installing the image
|
||||
|
||||
First we create the folder structure:
|
||||
|
||||
```
|
||||
mkdir -p /data/custom/machines
|
||||
ln -s /data/custom/machines /var/lib/machines
|
||||
```
|
||||
|
||||
Under normal circunstainces by now you would just run
|
||||
`machinectl import-tar /data/nixos-system-aarch64-linux.tar.xz <machinename>`
|
||||
however the version of tar that is present in this system doesn't really like
|
||||
the resulting tarball image. It will yeld errors like `Directory renamed before
|
||||
its status could be extracted`.
|
||||
|
||||
Thankfully we can install bsdtar through `apt install libarchive-tools` however
|
||||
`machinectl import-tar` is hardcoded to use the tar command. Adding a symlink
|
||||
from `/usr/bin/bsdtar` to `/usr/local/bin/tar` won't work since some parameters
|
||||
are used that are not supported in bsdtar. You could try writing a wrapper shell
|
||||
script but just unpacking the tarball directly was sufficient.
|
||||
|
||||
```
|
||||
mkdir /var/lib/machines/udmnixos
|
||||
bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
|
||||
```
|
||||
|
||||
Lets start the container.
|
||||
|
||||
```
|
||||
# machinectl start udmnixos
|
||||
# machinectl
|
||||
MACHINE CLASS SERVICE OS VERSION ADDRESSES
|
||||
udmnixos container systemd-nspawn nixos 23.11 192.168.168.88…
|
||||
|
||||
```
|
||||
|
||||
Good. Now we need to change the root password.
|
||||
|
||||
```
|
||||
# machinectl shell udmnixos /usr/bin/env passwd
|
||||
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
||||
New password:
|
||||
Retype new password:
|
||||
passwd: password updated successfully
|
||||
Connection to machine udmnixos terminated.
|
||||
```
|
||||
|
||||
Finally we can login into the container.
|
||||
|
||||
```
|
||||
# machinectl login nixos
|
||||
# machinectl login udmnixos
|
||||
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
|
||||
|
||||
|
||||
<<< Welcome to NixOS 23.11.20240115.b8dd8be (aarch64) - pts/1 >>>
|
||||
|
||||
|
||||
nixos login: root
|
||||
Password:
|
||||
|
||||
[root@nixos:~]#
|
||||
```
|
||||
|
||||
We haven't finished yet. By default the network is set to its own container
|
||||
network. We also don't have a DNS resolver configured. You can leave that
|
||||
session with CTRL+]]].
|
||||
|
||||
https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-n
|
||||
|
||||
```
|
||||
# machinectl stop udmnixos
|
||||
|
||||
# systemd-nspawn -M udmnixos
|
||||
Directory /data/custom/machines/udmnixos doesn't look like it has an OS tree. Refusing.
|
||||
```
|
||||
|
||||
## Networking and Persistence
|
||||
|
||||
The first thing that needs to be addressed is the DNS configuration. The default
|
||||
setting that copies the /etc/resolv.conf from host won't work since it points to
|
||||
localhost. Either install resolved, netmask or set a static DNS config.
|
||||
|
||||
Also read the capabilities section if you want to do things like using VPNs like
|
||||
Tailscale.
|
||||
|
||||
As for the network method we have some options here.
|
||||
|
||||
- Run using the default network stack and map ports to the container. https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p
|
||||
- Run using something akin to --network=host where the container has full access to the host network.
|
||||
- Give the container its own independent interface through a bridge.
|
||||
- Give the container its own independent interface through macvlan. https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network
|
||||
|
||||
### Using --network-veth and port mapping
|
||||
|
||||
```
|
||||
# mkdir -p /etc/systemd/nspawn
|
||||
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
||||
[Exec]
|
||||
Boot=on
|
||||
ResolvConf=off
|
||||
|
||||
[Network]
|
||||
Port=tcp:2222:22
|
||||
HERE
|
||||
|
||||
#machinectl enable udmnixos
|
||||
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
||||
|
||||
# machinectl start udmnixos
|
||||
```
|
||||
|
||||
Remember this will listen on ALL UDM interfaces so you might want to make sure
|
||||
the firewall rules will accomodate it.
|
||||
|
||||
```
|
||||
# iptables -t nat -L -n -v | grep 2222
|
||||
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
|
||||
0 0 DNAT tcp -- * * 0.0.0.0/0 !127.0.0.0/8 tcp dpt:2222 ADDRTYPE match dst-type LOCAL to:192.168.206.200:22
|
||||
```
|
||||
|
||||
### Using the host network
|
||||
|
||||
This will give access to all the network interfaces. Any service that runs on
|
||||
the container will be accessible from the UDM interfaces without the need to
|
||||
map ports. The container will also have the same IP addresses as the UDM.
|
||||
|
||||
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
|
||||
|
||||
```
|
||||
# mkdir -p /etc/systemd/nspawn
|
||||
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
||||
[Exec]
|
||||
Boot=on
|
||||
ResolvConf=off
|
||||
|
||||
[Network]
|
||||
Private=off
|
||||
VirtualEthernet=off
|
||||
HERE
|
||||
|
||||
#machinectl enable udmnixos
|
||||
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
||||
|
||||
# machinectl start udmnixos
|
||||
```
|
||||
|
||||
### Using a bridge to give the container its own interface
|
||||
|
||||
I had to give some capabilities to the container otherwise it wouldn't start. Feel free to find out what capabilities are needed from
|
||||
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
|
||||
|
||||
```
|
||||
# mkdir -p /etc/systemd/nspawn
|
||||
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
|
||||
[Exec]
|
||||
Boot=on
|
||||
Capability=CAP_NET_RAW,CAP_NET_ADMIN
|
||||
ResolvConf=off
|
||||
|
||||
[Network]
|
||||
Bridge=br2
|
||||
Private=off
|
||||
VirtualEthernet=off
|
||||
HERE
|
||||
|
||||
#machinectl enable udmnixos
|
||||
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
|
||||
|
||||
# machinectl start udmnixos
|
||||
```
|
||||
|
||||
### MACVLAN isolation and more
|
||||
|
||||
Here is where some custom configuration might be needed. Read https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container
|
||||
to find out how to setup custom scripts.
|
||||
|
||||
## Persistence
|
||||
|
||||
As far as I verified by rebooting the UDM many times to write this note all
|
||||
configurations were preserved. According to https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates
|
||||
although /etc/systemd and /data folders are preserved during firmware upgrades /var/ and /usr/ are not and there goes our packages and symlink. Please follow the steps on that
|
||||
page to setup persistence across firmware upgrades.
|
||||
Loading…
Reference in a new issue