iru/htdocs
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

udm-nixos

Browse source
This commit is contained in:
Iru Sensei 2024-05-19 15:54:26 +02:00
parent f9fbd200d7
commit a09c2872ea
1 changed files with 52 additions and 20 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

72
notes/UDM-NIXOS.md
View file

@ -1,6 +1,8 @@
# How I put NixOS on my UDM (trashcan model) router
(Really its just a running NixOS on systemd-nspawn thing)
Content also available on [https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md](https://code.despera.space/iru/htdocs/src/branch/main/notes/UDM-NIXOS.md)
Really it's just a running NixOS on systemd-nspawn thing.
The UDM product line basically runs on Linux kernel and userland. It is a
surprisingly normal device that allows you to SSH and run commands. It even has
@ -46,19 +48,16 @@ You enable SSH from the Controller UI, log into it as root with the password you
set to the admin user. You just waltz in and start installing and configuring.
```
apt update && apt install systemd-container
# apt update && apt install systemd-container
```
Thats it. Kinda. The complicated part is modifying the programs to write into
the persistent data directories while also making sure your stuff starts on
boot and doesn't get wiped on minor firmware upgrades.
Debian packages are cached on persistent storage and reinstalled at boot. The
systemd units are also kept and that gives us plenty of tools to work with.
## Building the NixOS root image.
Might want to read first: https://nixcademy.com/2023/08/29/nixos-nspawn/
Might want to read first: [https://nixcademy.com/2023/08/29/nixos-nspawn/](https://nixcademy.com/2023/08/29/nixos-nspawn/)
We need a NixOS tarball image. TFC's https://github.com/tfc/nspawn-nixos
contains the flake to build such an image and also publishes artifacts for AMD64
@ -97,8 +96,8 @@ the /data/ directory of the UDM.
First we create the folder structure:
```
mkdir -p /data/custom/machines
ln -s /data/custom/machines /var/lib/machines
# mkdir -p /data/custom/machines
# ln -s /data/custom/machines /var/lib/machines
```
Under normal circunstainces by now you would just run
@ -114,8 +113,8 @@ are used that are not supported in bsdtar. You could try writing a wrapper shell
script but just unpacking the tarball directly was sufficient.
```
mkdir /var/lib/machines/udmnixos
bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
# mkdir /var/lib/machines/udmnixos
# bsdtar Jxvfp /data/nixos-system-aarch64-linux.tar.xz -C /var/lib/machines/udmnixos
```
Lets start the container.
@ -175,15 +174,12 @@ The first thing that needs to be addressed is the DNS configuration. The default
setting that copies the /etc/resolv.conf from host won't work since it points to
localhost. Either install resolved, netmask or set a static DNS config.
Also read the capabilities section if you want to do things like using VPNs like
Tailscale.
As for the network method we have some options here.
- Run using the default network stack and map ports to the container. https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p
- [Run using the default network stack and map ports to the container](https://www.freedesktop.org/software/systemd/man/latest/systemd-nspawn.html#-p).
- Run using something akin to --network=host where the container has full access to the host network.
- Give the container its own independent interface through a bridge.
- Give the container its own independent interface through macvlan. https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network
- [Give the container its own independent interface through macvlan](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-2a-configure-the-container-to-use-an-isolated-macvlan-network).
### Using --network-veth and port mapping
@ -219,13 +215,17 @@ This will give access to all the network interfaces. Any service that runs on
the container will be accessible from the UDM interfaces without the need to
map ports. The container will also have the same IP addresses as the UDM.
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
You might want to read about [capabilities](https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=) if you plan on running some VPN
software like Wireguard or Tailscale.
```
# mkdir -p /etc/systemd/nspawn
# cat > /etc/systemd/nspawn/udmnixos.nspawn <<HERE
[Exec]
Boot=on
#Daring are we?
#Capability=all
ResolvConf=off
[Network]
@ -241,8 +241,7 @@ Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixo
### Using a bridge to give the container its own interface
I had to give some capabilities to the container otherwise it wouldn't start. Feel free to find out what capabilities are needed from
https://www.freedesktop.org/software/systemd/man/latest/systemd.nspawn.html#Capability=
I had to give some capabilities to the container otherwise it wouldn't properly start. Replace the value of Bridge with the bridge corresponding to the UDM network you want to add. Normally these correspond to the VLAN id of that network. Use `brctl show` to find out.
```
# mkdir -p /etc/systemd/nspawn
@ -262,6 +261,39 @@ HERE
Created symlink /etc/systemd/system/machines.target.wants/systemd-nspawn@udmnixos.service → /lib/systemd/system/systemd-nspawn@.service
# machinectl start udmnixos
# machinectl login udmnixos
# machinectl login nixos
Failed to get login PTY: No machine 'nixos' known
root@UDM:/etc/systemd/nspawn# machinectl login udmnixos
Connected to machine udmnixos. Press ^] three times within 1s to exit session.
<<< Welcome to NixOS 23.11.20240518.e7cc617 (aarch64) - pts/1 >>>
nixos login: root
Password:
[root@nixos:~]# ifconfig
host0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet [redacted] netmask 255.255.255.192 broadcast [redacted]
inet6 [redacted] prefixlen 64 scopeid 0x20<link>
inet6 [redacted] prefixlen 64 scopeid 0x0<global>
ether 92:01:4c:a7:a1:7b txqueuelen 1000 (Ethernet)
RX packets 2415 bytes 611986 (597.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 61 bytes 5337 (5.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
```
### MACVLAN isolation and more
@ -272,6 +304,6 @@ to find out how to setup custom scripts.
## Persistence
As far as I verified by rebooting the UDM many times to write this note all
configurations were preserved. According to https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates
although /etc/systemd and /data folders are preserved during firmware upgrades /var/ and /usr/ are not and there goes our packages and symlink. Please follow the steps on that
configurations were preserved. According to [the article on nspawn-containers on the unifies-utilities project](https://github.com/unifi-utilities/unifios-utilities/tree/main/nspawn-container#step-3-configure-persistence-across-firmware-updates)
although `/etc/systemd` and `/data` folders are preserved during firmware upgrades `/var/` and `/usr/` are not and there goes our packages and symlink. Please follow the steps on that
page to setup persistence across firmware upgrades.