Added documentation and instructions for nixOS jail

[templehasfallen]

Added documentation and instructions for nixOS jail

• Added documentation for deploying a nixOS based jail
• Updated compatibility table

[Jip-Hop]

Cool! Looks good. By the way I think you can add the bridge interface in the create statement: jlmkr create --distro=nixos --release=23.11 nixos-jail --network-bridge=br1.

[templehasfallen]

Cool! Looks good. By the way I think you can add the bridge interface in the create statement: jlmkr create --distro=nixos --release=23.11 nixos-jail --network-bridge=br1.

You're right, I missed it. Updated

[Jip-Hop]

Thanks for the update!

My idea behind the templates are to provide, if possible, a config template file which can help the user setup a jail in a specific, reproducible, way with as little user interaction as possible.

See for example the docker template: https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker

The readme is almost empty. It only contains the create command.

Do you think you could setup the nixos jail by using the features of the config file? Particularly the initial_setup= comes to mind.

[templehasfallen]

Thanks for the update!

My idea behind the templates are to provide, if possible, a config template file which can help the user setup a jail in a specific, reproducible, way with as little user interaction as possible.

See for example the docker template: https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker

The readme is almost empty. It only contains the create command.

Do you think you could setup the nixos jail by using the features of the config file? Particularly the initial_setup= comes to mind.

While a part of it could be done (replacing/changing the configuration.nix), the OS will still have to be rebuilt manually. The issue with that is because, since host networking is used in the initial setup phase, it will either:

1. Fail to rebuild if CAP_NET_ADMIN is disabled
2. Mess with the hosts iptables table if CAP_NET_ADMIN is enabled (guaranteed to be destructive for the host).

What could be done is editing/adding/replacing the configuration.nix (maybe pulling it from this repo and replacing it?) and once set up, just having to call nixos-rebuild switch inside the jail.

I'll have a look into the above scenario and do some tests next week unless someone gets to it before me.

[Lockszmith-GH]

I was curious so I tried this just now on my Cobia system. This is the first jail I'm installing on this one.
I was able to start the jail

Following the instruction the installation output is:

USE THIS SCRIPT AT YOUR OWN RISK!
IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.
Creating jail nixos-jail with default config.
Overriding distro config value with nixos.
Overriding release config value with 23.11.
Overriding systemd_nspawn_user_args config value with ['--network-bridge=br0'].
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Nixos 23.11 x86_64 (20240419_01:00) container.

WARNING: DISTRO NOT SUPPORTED

Chosen distro appears not to use systemd...

You probably will not get a shell with:
machinectl shell nixos-jail

You may get a shell with this command:
nsenter -t $(machinectl show nixos-jail -p Leader --value) -a /bin/sh -l

Read about the downsides of nsenter:
https://github.com/systemd/systemd/issues/12785#issuecomment-503019081

Using this distro with jlmkr.py is NOT recommended.

Autostart has been disabled.
You need to start this jail manually.

I think it would be wise to include an explanation about the DISTRO NOT SUPPORTED and that it's expected. Or even provide an example of the output so it will look familiar.

Thanks for sharing this, I've been really interested exploring NixOS, and this is the perfect starting point for me.

[Jip-Hop]

I think it would be wise to include an explanation about the DISTRO NOT SUPPORTED and that it's expected.

Thanks for pointing this out!

Jailmaker checks if the init system is systemd. In the case of nixOS the init system is also systemd:

[root@nixos:~]# /sbin/init --version 

<<< NixOS Stage 2 >>>

booting system configuration /nix/store/901sbqb9ys72xjps9c9y6yy5s3l4fxch-nixos-system-nixos-23.11.6418.e402c3eb6d88
running activation script...
setting up /etc...
starting systemd...
systemd 254 (254.10)
+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

[root@nixos:~]# ls -la /sbin/init
lrwxrwxrwx 1 root root 91 Apr 19 16:21 /sbin/init -> /nix/store/901sbqb9ys72xjps9c9y6yy5s3l4fxch-nixos-system-nixos-23.11.6418.e402c3eb6d88/init

But it can't be determined from the filepath that the init system is indeed systemd. So the current check fails and throws this warning while it shouldn't...

If we provide a template for nixOS then I think this check also requires some work...

[Jip-Hop]

just having to call nixos-rebuild switch inside the jail.

Perhaps the initial_setup could also insert a task which would execute nixos-rebuild switch the first time the jail starts? I don't know how nixOS works and if that's possible but here's a generic example of how to add a first boot task via systemd. At least then nixos-rebuild switch will run with --network-bridge=br1 in effect.

[Jip-Hop]

I also ran into NixOS/nixpkgs#63028 when removing the nixOS jail:

jlmkr remove nixos-jail

CAUTION: Type "nixos-jail" to confirm jail deletion!

nixos-jail

Wait for nixos-jail to stop.
Cleaning up: jails/nixos-jail.
Traceback (most recent call last):
  File "/mnt/data/jailmaker/jlmkr.py", line 2122, in <module>
    main()
  File "/mnt/data/jailmaker/jlmkr.py", line 2117, in main
    sys.exit(func(**args))
             ^^^^^^^^^^^^
  File "/mnt/data/jailmaker/jlmkr.py", line 1553, in remove_jail
    cleanup(jail_path)
  File "/mnt/data/jailmaker/jlmkr.py", line 768, in cleanup
    shutil.rmtree(jail_path, onerror=_onerror)
  File "/usr/lib/python3.11/shutil.py", line 732, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.11/shutil.py", line 660, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.11/shutil.py", line 666, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/mnt/data/jailmaker/jlmkr.py", line 765, in _onerror
    raise exc_value
  File "/usr/lib/python3.11/shutil.py", line 664, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
PermissionError: [Errno 1] Operation not permitted: 'empty'

My jails are created in plain directories, not datasets.

I had to run chattr -i jails/nixos-jail/rootfs/var/empty/ before I could remove the nixOS jail.

[templehasfallen]

I also ran into NixOS/nixpkgs#63028 when removing the nixOS jail:

I had to run chattr -i jails/nixos-jail/rootfs/var/empty/ before I could remove the nixOS jail.

Interesting. I would say making v1.1.4 with datasets a requirement would make sense. Possibly even a check within the script. Or alternatively an additional step for the remove command if its directories & nixOS?

just having to call nixos-rebuild switch inside the jail.

Perhaps the initial_setup could also insert a task which would execute nixos-rebuild switch the first time the jail starts? I don't know how nixOS works and if that's possible but here's a generic example of how to add a first boot task via systemd. At least then nixos-rebuild switch will run with --network-bridge=br1 in effect.

This is kinda tricky, specifically for nixos, because whatever task of this type you'd add would only be applied once you actually run nixos-rebuild switch.

Thanks for sharing this, I've been really interested exploring NixOS, and this is the perfect starting point for me.

Happy to do so. I'm also just starting to learn NixOS :)

If we provide a template for nixOS then I think this check also requires some work...

Yes I agree. Couldn't have been that easy after all

[templehasfallen]

Its more trickier than thought so far.

The following is the issue.

The nixOS image pulled is barebones, with directories and configuration created @ first boot. This means stuff such as /root, /usr don't exist yet. Attempting to run any kind of initial_setup will fail from systemd-nspawn because /usr does not exist yet.

The error is:

Directory /mnt/path/to/jails/nixostest7/rootfs doesn't look like it has an OS tree (/usr/ directory is missing). Refusing.

which seems to be a safety check from systemd-nspawn as mentioned in paragraph 6

As a safety check systemd-nspawn will verify the existence of /usr/lib/os-release or /etc/os-release in the container tree before booting a container (see os-release(5)). It might be necessary to add this file to the container tree manually if the OS of the container is too old to contain this file out-of-the-box.

The directories in question and the config /etc/nixos/configuration.nix are created during first boot (thanks to the people @ nixOS discord).

The main problem is bypassing/finding a way around the systemd-nspawn safety check.
I thought about placing the initial/setup file in a different location maybe (from what I see it is placed in /root/initial_setup) but I don't understand why the container will start when you don't pass and initial setup and won't if you do. Maybe because the rest of the systemd-run and systemd-nspawn arguments aren't specified during initial setup?

From what I can see there isn't really any option to run it via ExecPreStart only once directly via systemd.

Only option I see to get this working currently is actually adding support to jlmkr via a function/if block.

But I'll dig into it more to see if I can find a workaround.

[Jip-Hop]

You may be able to modify the extracted rootfs from the pre_start_hook. If you can figure out, by looking inside the rootfs, if it's the first boot, then I think you may be able to modify the nixOS configuration even before it booted and you may not even need to run nixos-rebuild switch. This is me brainstorming. Pure speculation, I have 0 experience with nixOS.

[Jip-Hop]

Attempting to run any kind of initial_setup will fail from systemd-nspawn because /usr does not exist yet.

Since jailmaker v1.4.0 the initial_setup runs once the jail has started for the first time and the init is ready.

[templehasfallen]

Attempting to run any kind of initial_setup will fail from systemd-nspawn because /usr does not exist yet.

Since jailmaker v1.4.0 the initial_setup runs once the jail has started for the first time and the init is ready.

I've tan some with the bew version, still unable to get it to fully work but making progress.

I'm trying to edit the config in place with the initial setup and then running nixos-rebuild switch which seems to complete successfully, but then config doesn't add up.

[dkowis]

Thanks for the update!
My idea behind the templates are to provide, if possible, a config template file which can help the user setup a jail in a specific, reproducible, way with as little user interaction as possible.
See for example the docker template: https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker
The readme is almost empty. It only contains the create command.
Do you think you could setup the nixos jail by using the features of the config file? Particularly the initial_setup= comes to mind.

While a part of it could be done (replacing/changing the configuration.nix), the OS will still have to be rebuilt manually. The issue with that is because, since host networking is used in the initial setup phase, it will either:

1. Fail to rebuild if `CAP_NET_ADMIN` is disabled

2. Mess with the hosts iptables table if `CAP_NET_ADMIN` is enabled (guaranteed to be destructive for the host).

What could be done is editing/adding/replacing the configuration.nix (maybe pulling it from this repo and replacing it?) and once set up, just having to call nixos-rebuild switch inside the jail.

Is this something that would be accepted into nixos-hardware so it can just be used as a module for this "hardware"?

I don't know if it'd accept the bits from the config file, but maybe something could be made to do that...

There's a lot I don't know, and I'm experimenting with it also :)

[Lockszmith-GH]

This one is interesting:

The process to actually get this to work is:

1. Install with default setup (changing initial_setup here messes the process)
2. Start the jail - it will initialize the nixos setup
3. Stop the jail
4. Modify the /etc/nixos/configuration.nix file to support DHCP (the setup @templehasfallen provided in the instructions)
5. Start the jail

I'll spend some time later to figure out what is happening in 2. maybe there is a modification pre_start_hook can make to setup the networking correctly before starting the first time.

[dkowis]

I do that step that @Lockszmith-GH mentioned. It's the same basic steps that I do with my virtual machines when I set them up with NixOS as well. I've got a flake based repo with the configuration for the jail that is working quite well.

I don't mind that I had to do a bit more manual configuration, it works perfectly, does what it's supposed to over reboots, I'm very happy with it.

[Jip-Hop]

I think using initial_setup doesn't work currently because the code assumes /root to always exist and puts the setup script in there. Should be an easy fix.

And perhaps the reason that networking isn't working on the initial boot is that the .network config files are assumed to be in a default location (lib/systemd/network/, etc/systemd/network) which is not appropriate for nixos. Perhaps this is solvable in a neat generic way, by following where the /sbin/init points to and then locate the network files relative this this location instead of the absolute path currently used.

I'd be open to changes to jlmkr.py, especially if they don't introduce OS specific logic.

[Lockszmith-GH]

I was able to use pre_start_hook to setup:

• copy a custom configuration.nix file (form the template location) to /etc/nixos
• copy jail-init.script and place it in the root dir.

The jail-init.script will initalize the NIC via DHCP and runs nixos-rebuild switch to prep the system for the first time, if everything is successfull it moves itself to /tmp.

@Jip-Hop - you're hunch was right about the initial error I got.
Creating /root within the rootfs resolved the error. However, while the jail starts, it does not start the initial_setup, it just states "Tried to run the following..." (see full output below)

I think the last issue we are facing here is to figure out how to run initial_setup.
Maybe we need a pre-network initial_setup.

The semi-automated process I have now

With SCALE_POOL_ROOT = the nixos rootfs path and my $SCALE_POOL_ROOT/jailmaker/templates/nixos/ has 3 files:

• config - the jailmkr template
• configuration.nix - containing the configuration from the documentation in this PR.
• jail-init.script - the network initialization and NixOs rebuild steps

The steps of the process are:

1. Install using my customized nixos config template with a custom configuraiton.nix in the template dir

   jlmkr create --config $SCALE_POOL_ROOT/jailmaker/templates/nixos/config nixos

2. Start the jail (auto start isn't allowed, so this step is mandatory)
   Also create the /root dir in the rootfs.

   install -m 700 -d $SCALE_POOL_ROOT/jailmaker/jails/nixos/rootfs/root
   jlmkr start nixos

3. Shell into the jail

   jlmkr shell nixos

   a. Inside the jail run the jail-init.script:
   bash /jail-init.script

At this point NixOS is ready and can be restarted and it will get it's IP address from DHCP.
Of course, if you'd like - you can configure it with a static IP by uncommenting the static IP section of the configuration.nix file.

Invocation output

Click to expand

# jlmkr create --config $SCALE_POOL_ROOT/jailmaker/templates/nixos/config nixos && install -m 700 -d $SCALE_POOL_ROOT/jailmaker/jails/nixos/rootfs/root && jailmkr start nixos
USE THIS SCRIPT AT YOUR OWN RISK!
IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.
Creating jail nixos from config template /mnt/pool/jailmaker/templates/nixos/config.

TIP: Run `jlmkr.py create` without any arguments for interactive config.
Or use CLI args to override the default options.
For more info, run: `jlmkr.py create --help`

Creating ZFS Dataset pool/jailmaker/jails/nixos
Using image from local cache
Unpacking the rootfs
---
You just created an Nixos 23.11 x86_64 (20240620_01:02) container.

WARNING: DISTRO NOT SUPPORTED

Chosen distro appears not to use systemd...

You probably will not get a shell with:
machinectl shell nixos

You may get a shell with this command:
nsenter -t $(machinectl show nixos -p Leader --value) -a /bin/sh -l

Read about the downsides of nsenter:
https://github.com/systemd/systemd/issues/12785#issuecomment-503019081

Using this distro with jlmkr.py is NOT recommended.

Autostart has been disabled.
You need to start this jail manually.

Starting jail nixos with the following command:

systemd-run --collect --property=Delegate=yes --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=TasksMax=infinity --property=Type=notify --setenv=SYSTEMD_NSPAWN_LOCK=0 --property=KillMode=mixed --unit=jlmkr-nixos --working-directory=./jails/nixos '--description=My nspawn jail nixos [created with jailmaker]' --property=ExecStartPre=/mnt/szmedia/jailmaker/jails/nixos/.ExecStartPre -- systemd-nspawn --bind-ro=/sys/module --boot --inaccessible=/sys/module/apparmor --quiet --keep-unit --machine=nixos --directory=rootfs --notify-ready=yes --network-bridge=br0

Running as unit: jlmkr-nixos.service
About to run the initial setup.
Waiting for networking in the jail to be ready.
Please wait (this may take 90s in case of bridge networking with STP is enabled)...
Tried to run the following commands inside the jail:
 #!/usr/bin/bash
 set -euo pipefail
 [ -r /jail-init.script ] && bash -e /jail-init.script

# jailmkr shell nixos

Failed to run initial setup... you may want to stop and remove the jail and try again.
[root@nixos:~]# bash /jail-init.script
Requesting DHCP lease...
sending commands to dhcpcd process
Checking for a LAN connection on 192.168...
Time: .........10........20........30........40........50........60........70........80 seconds
Wait: ..................................................................

IP Assigned(192.168.222.117)
Checking for connection with the Web...
Time: .........10........20........30........40........50........60........70........80 seconds
Wait: 136.56.63.176

Can reach the Web

building Nix...
building the system configuration...
these 51 derivations will be built:
  /nix/store/064swcmyh8kf9djy89s5s6ylydvlb25b-unit-script-container_-post-start.drv
  /nix/store/4km8xgb3hjbgjcq8g87nhynax9w7v312-etc-sysctl.d-60-nixos.conf.drv
  /nix/store/0gdkp4yf8rpdw1ij2f3z8h0yfxwldhdq-X-Restart-Triggers-systemd-sysctl.drv
  /nix/store/0xbrnslanjpfyc2w8lcz69zqvgy5v2hy-ensure-all-wrappers-paths-exist.drv
  /nix/store/rbpdj3vmn0ar8jizqzvm936s22q0dldb-container-init.drv
  /nix/store/1ha9w74v2n21svqd7x4bvk0nrfbsvjvs-unit-script-container_-start.drv
  /nix/store/xb194c7g3xpykyikxd0h1y7dn7anvg5m-set-environment.drv
  /nix/store/1lpn686rfcbdcvw3bkip7j1nyhf30875-etc-profile.drv
  /nix/store/5yzhvqdq2csasnb2hhi0n5pbj6qx68ym-builder.pl.drv
  /nix/store/23nc6abg43r4irvppdrzs2ng9idj9rwq-perl-5.38.2-env.drv
  /nix/store/2s32c6vanhhk9457vmzbg4z5pw9b7f2d-unit-script-container_-pre-start.drv
  /nix/store/4crg85s6bbm7p0b8ahpl8j63wl7macs6-nixos-container.drv
  /nix/store/4566i23r8g34f9clkdpy6yrwql73a9jc-reload-container.drv
  /nix/store/6xhmszkx2vj7vhmqqw4agav2hvzpll92-etc-systemd-networkd.conf.drv
  /nix/store/4qkjddmdpd6lni0gs8s1h0n0a1fqxm32-X-Restart-Triggers-systemd-networkd.drv
  /nix/store/ijy99blkgm44x5z05g7ca78gbd9n2a15-etc-systemd-resolved.conf.drv
  /nix/store/56b9a05kh4qs04r8l13pv6149ljl8cnk-X-Restart-Triggers-systemd-resolved.drv
  /nix/store/lhkw2f702bkgwk117fg627j0vs0v70fr-command-not-found.drv
  /nix/store/jaqj8lkqsplr102p5rypqghsay9m8pzg-system-path.drv
  /nix/store/jndsd89lkp89b04f7xd2rfnhb7v9glci-dbus-1.drv
  /nix/store/sqfcgfp95psy71cpws51kaq8b6ppd8ba-X-Restart-Triggers-dbus.drv
  /nix/store/5bgm1xjcca8dz1ywbb9f21k1b62gnvz2-unit-dbus.service.drv
  /nix/store/5fxb0xvbbxriknkd2k43ah7ipm3jcynz-local-cmds.drv
  /nix/store/5si9zpn75ggnmgdmrxmdk11nacr4fk71-unit-container-.service.drv
  /nix/store/67kjnylsaanzqr0nlxm18f10ckfn20sb-etc-resolvconf.conf.drv
  /nix/store/86h8i1v8zl69ksg4w4x4qplvf63nsycj-etc-nix-registry.json.drv
  /nix/store/xqvnh5242vzvd072dn6pqddgsjkbykf1-unit-40-host0.network.drv
  /nix/store/rs7wif13j05zvfxd1p0rlazp1vsc5k36-X-Reload-Triggers-systemd-networkd.drv
  /nix/store/8n2cj6zlpdjr9gp2nb3lfvvi8x1vrrj8-unit-systemd-networkd.service.drv
  /nix/store/9zbca7slsccz6a0sb0mwinfgpnpyl1w9-unit-systemd-network-wait-online-.service.drv
  /nix/store/amah5ww5y5b1mgsvillmsklrixh0s9nj-unit-systemd-vconsole-setup.service-disabled.drv
  /nix/store/d4x034nzh1gsmg40x3bd012kbrqhc08m-unit-systemd-networkd-wait-online.service.drv
  /nix/store/fckd8hpr4g92b7cir19vfib689phrk9m-unit-network-local-commands.service.drv
  /nix/store/i8g01lkvnm4qlsba6n5yv74jn2kldq6y-unit-systemd-sysctl.service.drv
  /nix/store/k5l3gvc4kjic8ib3h0ix7v6lkbp8ahvd-unit-dbus.service.drv
  /nix/store/mliq87756jgcfavj5726w8fv8dzcqyqh-unit-systemd-networkd.socket.drv
  /nix/store/q46f7qzjclj8n5rdyicdxrgczqcxvv2c-unit-autovt-.service-disabled.drv
  /nix/store/qzq0bjqvnmxlwkha2cq52lng1v5ch77g-unit-multi-user.target.drv
  /nix/store/zrsnwk21v8l2d0rf04h0s90lq342qlam-unit-systemd-resolved.service.drv
  /nix/store/8kspd3b9xlplf04g8zxjqr06ffsjy5gs-system-units.drv
  /nix/store/aqr1iv3clbd13nk9wj7fsjwx3q7f0xkm-users-groups.json.drv
  /nix/store/ckcsya240ahd7jzcqv7hs1b2z99jskb5-issue.drv
  /nix/store/fa6m7x962p8aaxdj5z3gpghib7wjlfhr-etc-nsswitch.conf.drv
  /nix/store/gfq1q1pn8a6qpwnc58fp63nahq03b72l-etc-pam-environment.drv
  /nix/store/vvadv6brhjyiljba05fpxysmzpmy8z37-user-units.drv
  /nix/store/vzs0i4w46na1bhsx26x63hgv6jgq19ld-etc-bashrc.drv
  /nix/store/ws7a83bqgn0nvs7vv43hynbzgv974vk5-etc-man_db.conf.drv
  /nix/store/n07s46ywwk39wbnarmrpzrbxrh8nafgi-etc.drv
  /nix/store/ngkkp7zap1jvyqf29hdw8sl0incwiv48-init-script-builder.sh.drv
  /nix/store/ykcs66m9w6dyc3m6hnh36wcyigj3acww-stage-2-init.sh.drv
  /nix/store/fjfazh4kxf39jjrvn7jw2sqlfnns2ngm-nixos-system-nixos-23.11.7665.03d771e513ce.drv
these 86 paths will be fetched (74.42 MiB download, 373.69 MiB unpacked):
  /nix/store/k693mwkvxwxc4rszhhidg2d73bzprm6y-acl-2.3.1-doc
  /nix/store/3cdjn43dzddjvnm8wnd4xlxds7igawpv-acl-2.3.1-man
  /nix/store/sg2vzvyc14cly74lr04spy4ni8r06sxg-attr-2.5.1-doc
  /nix/store/zjyz211jsvbhyz4n7z57w9g2h2gca2j9-attr-2.5.1-man
  /nix/store/1r2kxwnz1j0wblb4j3xx9g6xyr1p9wj5-bash-interactive-5.2-p15-doc
  /nix/store/7144dl9ph7x39b61ib85d2vjxi6jxnb6-bash-interactive-5.2-p15-info
  /nix/store/iwlwcp2a6jmzl8gp44rg7lp00arvsc82-bash-interactive-5.2-p15-man
  /nix/store/9sdkyh5hh2hkk1m808wwg9cafz7zj2a3-bind-9.18.24-man
  /nix/store/w0z7wgmhyb31108slp66lhmqanhbnwfw-binutils-2.40
  /nix/store/x4lpw6bn1y31641izylfrxpmxgyb509d-binutils-2.40-lib
  /nix/store/ls7wal9y40rjl21p6bzrx657k9wbil4j-binutils-wrapper-2.40
  /nix/store/zk2nr0q7n27fx3aqvc5d52xjp6avf5pj-bzip2-1.0.8-man
  /nix/store/2yc5j4fnf0mfc50vzh11p0syjgi7zazk-coreutils-full-9.3-info
  /nix/store/kfwy0fgbysvapm11pwb07qq02v9phh7z-curl-8.4.0-man
  /nix/store/43axaapcv95bjn803693zvz6cxw3y9dg-dbus-1.14.10-doc
  /nix/store/l03qqcvhjgx3jrmllzvc8nszvlgdxyjj-dbus-1.14.10-man
  /nix/store/hr35zmy28hayf0b23anszlxcjkdwvb0m-desktop-file-utils-0.26
  /nix/store/j9zpq9clx98pwf7sir7szm82pqn151p2-die-hook
  /nix/store/pz08bz8vp9x7bkg7jnll7g6v4f1nrb07-diffutils-3.10-info
  /nix/store/bkky00l9i47dls8bbc4f7h0lnkgxqlka-e2fsprogs-1.47.0-info
  /nix/store/4xch31flaniaqng2pq73vkncsnd11r87-e2fsprogs-1.47.0-man
  /nix/store/3aangam5afdh7945bds12j2vh6v0kv8s-expand-response-params
  /nix/store/qh16lscqbqv33rhk9bhdc6m2nin2i64v-file-5.45
  /nix/store/bp1p3i95b562907i808k3pz77ha25vaz-find-xml-catalogs-hook
  /nix/store/37p0n99fqcddw9rzhl0h0kcgpx8cf0ll-findutils-4.9.0-info
  /nix/store/89c4cb77x51a6pcq5vkwb75phlx96q83-gawk-5.2.2-info
  /nix/store/6ayk80gg7a02bxrl34lh6jn85sbmf22z-gawk-5.2.2-man
  /nix/store/bmcypdzbidmpam7np26hn25gs7idf88v-gcc-12.3.0
  /nix/store/shdkxxdlpv66wl33rh9d01zfypaka1dn-gcc-wrapper-12.3.0
  /nix/store/4ss0cyy9bw327gfmhjik2ya9rwih074r-getconf-glibc-2.38-77
  /nix/store/n2x4y1n23m8xnq1gpl3f9vf43vfifcf9-glib-2.78.6
  /nix/store/i58yz1rxjxpha40l17hgg7cz62jck9q3-glibc-2.38-77-dev
  /nix/store/jwksvfzwywqscxi17ndyq1g4by6cjfrl-gmp-6.3.0
  /nix/store/nzahvria6cvyb3kcn3wpn0m64xmjpq8m-gnu-config-2023-09-19
  /nix/store/jqk4hc789yjw5si01dfkzqd37avvffii-gnugrep-3.11-info
  /nix/store/hm6hj5ylwa35cdwb9d35nbql5k7qwnir-gnumake-4.4.1
  /nix/store/pvcs4gsvibmsplxyp7p1ykp9n2zb17s0-gnused-4.9-info
  /nix/store/kp628rsilcnadawk3pfh7vx0cjlixajd-gnutar-1.35-info
  /nix/store/8mhz9sszz4n09ifax207sqpd79hwdmn1-groff-1.23.0
  /nix/store/rbz9shkgnynakk6f7zllp3jpffw0dhgh-gzip-1.13-info
  /nix/store/4srywpbmg0bl6kng9y2mkd5jgmin5pi4-gzip-1.13-man
  /nix/store/i8avwbj70p1f31656a93qpzn46f7kd41-hicolor-icon-theme-0.17
  /nix/store/8khkbhgsl2g9sg8xvj5d0cfmmg28kvxq-isl-0.20
  /nix/store/713423fia8n9gqmnm3dbvsg7cbg4vz9q-less-643-man
  /nix/store/r2cs7pyba2r2rfdyfgv4yaxgswq5kbx6-lesspipe-2.10
  /nix/store/vzgz7dni5a62j7p9slvjzi73ly78jm2a-libcap-2.69-doc
  /nix/store/8n6fysacfchw7m7ylvq6766kgqasgkm3-libcap-2.69-man
  /nix/store/j8rl2pr50jpjsipwr5xwvj10cqmls6qh-libmpc-1.3.1
  /nix/store/mbpw0l9fsri5i63nrqx5nxqw4x6pz0rr-libpipeline-1.5.7
  /nix/store/kaapjd4y8ppa63h2c2illyy1pm0yqg70-libressl-3.8.4-man
  /nix/store/737gl69d38r8d86wvkfk1bxnsvpxww58-libselinux-3.3
  /nix/store/d65q3vwxyyxa8y18axsfhbrk14l6xpv4-libxslt-1.1.38
  /nix/store/hcszkiv5y5znrhx0cd8yi286brcnlb6d-libxslt-1.1.38-bin
  /nix/store/sin431s4g4m621f2ac35zwmd0pgrqy9n-linux-headers-6.5
  /nix/store/c6yrzi876ginxzrzx5qmi6z3nv0gnsy8-linux-pam-1.5.2-doc
  /nix/store/458s8ksz6ikvh5dxqcj4bcd3mrbnj700-linux-pam-1.5.2-man
  /nix/store/fsyii1arm1gz36jjbpvbp8r97pqzicw9-lndir-1.0.4
  /nix/store/rxwbhj6snisrqxvmlllndsgm8xjb09c3-make-binary-wrapper-hook
  /nix/store/rcjjq20sac3mmh304cl7pw6xw817jqim-man-db-2.11.2
  /nix/store/5avnrn9hwigp0sggzkf0s5z5mr0g9pgh-man-db-2.11.2-doc
  /nix/store/002zhxss9avpvbsjkl323r2gbdn2kw4r-mpfr-4.2.1
  /nix/store/ic0l57wr8gw37kfwz7fn8jm0zimfzpyy-nano-7.2-info
  /nix/store/jcs7az4cnilqirh5x6kzm0l85206acg8-ncurses-6.4-man
  /nix/store/ijqiysgli4x4lghvfcwpc8r8ggadwn3n-nix-2.18.1-doc
  /nix/store/xsc031j4y9jg7in4m1fzfxpyvbkg48sb-patchelf-0.15.0
  /nix/store/w0m6l34znp98cyi8v35l8dxmxh1sp8wg-pcre-8.45
  /nix/store/0sr9jxvk9ynb7q3zics43a4dq8pm78r0-perl5.38.2-DBD-SQLite-1.74
  /nix/store/ncrl41z6mfpyfmc46fy3yr0l2wqm0gyl-perl5.38.2-DBI-1.643
  /nix/store/ihl0gpqzlid2pc8cv5cm36rsjy8wfs5h-perl5.38.2-String-ShellQuote-1.04
  /nix/store/hdx2aw4b1cdy6r7x65zhvia0iinlz53i-rsync-3.2.7
  /nix/store/l4kizq8sh41askjawpqasjgky8rwqnq8-shadow-4.14.2-man
  /nix/store/12f878dy1sxvffiaa4w26w9r7xwd6k70-shared-mime-info-2.4
  /nix/store/5pqgpdd933c8adab6ha0l1fk311ljyhx-sound-theme-freedesktop-0.8
  /nix/store/3ji0dxhhrhg1hpf2zp80jf9w0ls71yyn-stdenv-linux
  /nix/store/qrnj3wvbs1ysmv2b773bfflq1jxlgqnr-strace-6.9
  /nix/store/cl5h3rmlylbx7aqld00w6njd5hrr12dg-system-generators
  /nix/store/ifl31fys6fx6mxp9c2m5i8970xvy3kim-system-shutdown
  /nix/store/7b5vj6w0cdd0pqs499z3340ga71v8dnl-systemd-254.10-man
  /nix/store/am8ymky6wyspna12q4fsaz2hbmgx5j7n-texinfo-7.0.3
  /nix/store/f4xmyvgy61azay4rd4b481ddvmg21m37-texinfo-interactive-7.0.3
  /nix/store/wjwpl3zmj26hzq2mxqq728jxjzf2ixpw-update-autotools-gnu-config-scripts-hook
  /nix/store/q7hrqb8l4znz7hhz07906wp6v4k7bdps-util-linux-2.39.2-man
  /nix/store/1fbyv4cjc2v0paymlxr7r2ik66w3zdrd-xxHash-0.8.2
  /nix/store/59ivxahjjapxw4yp26f5mdd5c3p8gd6k-xz-5.4.4-doc
  /nix/store/lpmb6y3gd25blxv7bvlgn7d4jlvrjszm-xz-5.4.4-man
  /nix/store/f93zpvx7xdz29n736ysdc6h5660d2xx6-zstd-1.5.5-man
copying path '/nix/store/7b5vj6w0cdd0pqs499z3340ga71v8dnl-systemd-254.10-man' from 'https://cache.nixos.org'...
copying path '/nix/store/q7hrqb8l4znz7hhz07906wp6v4k7bdps-util-linux-2.39.2-man' from 'https://cache.nixos.org'...
copying path '/nix/store/3cdjn43dzddjvnm8wnd4xlxds7igawpv-acl-2.3.1-man' from 'https://cache.nixos.org'...
copying path '/nix/store/k693mwkvxwxc4rszhhidg2d73bzprm6y-acl-2.3.1-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/sg2vzvyc14cly74lr04spy4ni8r06sxg-attr-2.5.1-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/zjyz211jsvbhyz4n7z57w9g2h2gca2j9-attr-2.5.1-man' from 'https://cache.nixos.org'...
copying path '/nix/store/1r2kxwnz1j0wblb4j3xx9g6xyr1p9wj5-bash-interactive-5.2-p15-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/9sdkyh5hh2hkk1m808wwg9cafz7zj2a3-bind-9.18.24-man' from 'https://cache.nixos.org'...
copying path '/nix/store/7144dl9ph7x39b61ib85d2vjxi6jxnb6-bash-interactive-5.2-p15-info' from 'https://cache.nixos.org'...
copying path '/nix/store/zk2nr0q7n27fx3aqvc5d52xjp6avf5pj-bzip2-1.0.8-man' from 'https://cache.nixos.org'...
copying path '/nix/store/kfwy0fgbysvapm11pwb07qq02v9phh7z-curl-8.4.0-man' from 'https://cache.nixos.org'...
copying path '/nix/store/2yc5j4fnf0mfc50vzh11p0syjgi7zazk-coreutils-full-9.3-info' from 'https://cache.nixos.org'...
copying path '/nix/store/iwlwcp2a6jmzl8gp44rg7lp00arvsc82-bash-interactive-5.2-p15-man' from 'https://cache.nixos.org'...
copying path '/nix/store/l03qqcvhjgx3jrmllzvc8nszvlgdxyjj-dbus-1.14.10-man' from 'https://cache.nixos.org'...
copying path '/nix/store/43axaapcv95bjn803693zvz6cxw3y9dg-dbus-1.14.10-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/pz08bz8vp9x7bkg7jnll7g6v4f1nrb07-diffutils-3.10-info' from 'https://cache.nixos.org'...
copying path '/nix/store/x4lpw6bn1y31641izylfrxpmxgyb509d-binutils-2.40-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/j9zpq9clx98pwf7sir7szm82pqn151p2-die-hook' from 'https://cache.nixos.org'...
copying path '/nix/store/bkky00l9i47dls8bbc4f7h0lnkgxqlka-e2fsprogs-1.47.0-info' from 'https://cache.nixos.org'...
copying path '/nix/store/4xch31flaniaqng2pq73vkncsnd11r87-e2fsprogs-1.47.0-man' from 'https://cache.nixos.org'...
copying path '/nix/store/3aangam5afdh7945bds12j2vh6v0kv8s-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/qh16lscqbqv33rhk9bhdc6m2nin2i64v-file-5.45' from 'https://cache.nixos.org'...
copying path '/nix/store/bp1p3i95b562907i808k3pz77ha25vaz-find-xml-catalogs-hook' from 'https://cache.nixos.org'...
copying path '/nix/store/37p0n99fqcddw9rzhl0h0kcgpx8cf0ll-findutils-4.9.0-info' from 'https://cache.nixos.org'...
copying path '/nix/store/4ss0cyy9bw327gfmhjik2ya9rwih074r-getconf-glibc-2.38-77' from 'https://cache.nixos.org'...
copying path '/nix/store/89c4cb77x51a6pcq5vkwb75phlx96q83-gawk-5.2.2-info' from 'https://cache.nixos.org'...
copying path '/nix/store/jwksvfzwywqscxi17ndyq1g4by6cjfrl-gmp-6.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/6ayk80gg7a02bxrl34lh6jn85sbmf22z-gawk-5.2.2-man' from 'https://cache.nixos.org'...
copying path '/nix/store/nzahvria6cvyb3kcn3wpn0m64xmjpq8m-gnu-config-2023-09-19' from 'https://cache.nixos.org'...
copying path '/nix/store/hm6hj5ylwa35cdwb9d35nbql5k7qwnir-gnumake-4.4.1' from 'https://cache.nixos.org'...
copying path '/nix/store/jqk4hc789yjw5si01dfkzqd37avvffii-gnugrep-3.11-info' from 'https://cache.nixos.org'...
copying path '/nix/store/kp628rsilcnadawk3pfh7vx0cjlixajd-gnutar-1.35-info' from 'https://cache.nixos.org'...
copying path '/nix/store/8mhz9sszz4n09ifax207sqpd79hwdmn1-groff-1.23.0' from 'https://cache.nixos.org'...
copying path '/nix/store/pvcs4gsvibmsplxyp7p1ykp9n2zb17s0-gnused-4.9-info' from 'https://cache.nixos.org'...
copying path '/nix/store/rbz9shkgnynakk6f7zllp3jpffw0dhgh-gzip-1.13-info' from 'https://cache.nixos.org'...
copying path '/nix/store/4srywpbmg0bl6kng9y2mkd5jgmin5pi4-gzip-1.13-man' from 'https://cache.nixos.org'...
copying path '/nix/store/i8avwbj70p1f31656a93qpzn46f7kd41-hicolor-icon-theme-0.17' from 'https://cache.nixos.org'...
copying path '/nix/store/713423fia8n9gqmnm3dbvsg7cbg4vz9q-less-643-man' from 'https://cache.nixos.org'...
copying path '/nix/store/vzgz7dni5a62j7p9slvjzi73ly78jm2a-libcap-2.69-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/8n6fysacfchw7m7ylvq6766kgqasgkm3-libcap-2.69-man' from 'https://cache.nixos.org'...
copying path '/nix/store/mbpw0l9fsri5i63nrqx5nxqw4x6pz0rr-libpipeline-1.5.7' from 'https://cache.nixos.org'...
copying path '/nix/store/kaapjd4y8ppa63h2c2illyy1pm0yqg70-libressl-3.8.4-man' from 'https://cache.nixos.org'...
copying path '/nix/store/d65q3vwxyyxa8y18axsfhbrk14l6xpv4-libxslt-1.1.38' from 'https://cache.nixos.org'...
copying path '/nix/store/c6yrzi876ginxzrzx5qmi6z3nv0gnsy8-linux-pam-1.5.2-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/458s8ksz6ikvh5dxqcj4bcd3mrbnj700-linux-pam-1.5.2-man' from 'https://cache.nixos.org'...
copying path '/nix/store/sin431s4g4m621f2ac35zwmd0pgrqy9n-linux-headers-6.5' from 'https://cache.nixos.org'...
copying path '/nix/store/fsyii1arm1gz36jjbpvbp8r97pqzicw9-lndir-1.0.4' from 'https://cache.nixos.org'...
copying path '/nix/store/ic0l57wr8gw37kfwz7fn8jm0zimfzpyy-nano-7.2-info' from 'https://cache.nixos.org'...
copying path '/nix/store/5avnrn9hwigp0sggzkf0s5z5mr0g9pgh-man-db-2.11.2-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/w0z7wgmhyb31108slp66lhmqanhbnwfw-binutils-2.40' from 'https://cache.nixos.org'...
copying path '/nix/store/jcs7az4cnilqirh5x6kzm0l85206acg8-ncurses-6.4-man' from 'https://cache.nixos.org'...
copying path '/nix/store/002zhxss9avpvbsjkl323r2gbdn2kw4r-mpfr-4.2.1' from 'https://cache.nixos.org'...
copying path '/nix/store/8khkbhgsl2g9sg8xvj5d0cfmmg28kvxq-isl-0.20' from 'https://cache.nixos.org'...
copying path '/nix/store/ijqiysgli4x4lghvfcwpc8r8ggadwn3n-nix-2.18.1-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/xsc031j4y9jg7in4m1fzfxpyvbkg48sb-patchelf-0.15.0' from 'https://cache.nixos.org'...
copying path '/nix/store/hcszkiv5y5znrhx0cd8yi286brcnlb6d-libxslt-1.1.38-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/w0m6l34znp98cyi8v35l8dxmxh1sp8wg-pcre-8.45' from 'https://cache.nixos.org'...
copying path '/nix/store/ncrl41z6mfpyfmc46fy3yr0l2wqm0gyl-perl5.38.2-DBI-1.643' from 'https://cache.nixos.org'...
copying path '/nix/store/ihl0gpqzlid2pc8cv5cm36rsjy8wfs5h-perl5.38.2-String-ShellQuote-1.04' from 'https://cache.nixos.org'...
copying path '/nix/store/l4kizq8sh41askjawpqasjgky8rwqnq8-shadow-4.14.2-man' from 'https://cache.nixos.org'...
copying path '/nix/store/5pqgpdd933c8adab6ha0l1fk311ljyhx-sound-theme-freedesktop-0.8' from 'https://cache.nixos.org'...
copying path '/nix/store/j8rl2pr50jpjsipwr5xwvj10cqmls6qh-libmpc-1.3.1' from 'https://cache.nixos.org'...
copying path '/nix/store/qrnj3wvbs1ysmv2b773bfflq1jxlgqnr-strace-6.9' from 'https://cache.nixos.org'...
copying path '/nix/store/cl5h3rmlylbx7aqld00w6njd5hrr12dg-system-generators' from 'https://cache.nixos.org'...
copying path '/nix/store/ifl31fys6fx6mxp9c2m5i8970xvy3kim-system-shutdown' from 'https://cache.nixos.org'...
copying path '/nix/store/am8ymky6wyspna12q4fsaz2hbmgx5j7n-texinfo-7.0.3' from 'https://cache.nixos.org'...
copying path '/nix/store/rcjjq20sac3mmh304cl7pw6xw817jqim-man-db-2.11.2' from 'https://cache.nixos.org'...
copying path '/nix/store/i58yz1rxjxpha40l17hgg7cz62jck9q3-glibc-2.38-77-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/f4xmyvgy61azay4rd4b481ddvmg21m37-texinfo-interactive-7.0.3' from 'https://cache.nixos.org'...
copying path '/nix/store/wjwpl3zmj26hzq2mxqq728jxjzf2ixpw-update-autotools-gnu-config-scripts-hook' from 'https://cache.nixos.org'...
copying path '/nix/store/737gl69d38r8d86wvkfk1bxnsvpxww58-libselinux-3.3' from 'https://cache.nixos.org'...
copying path '/nix/store/1fbyv4cjc2v0paymlxr7r2ik66w3zdrd-xxHash-0.8.2' from 'https://cache.nixos.org'...
copying path '/nix/store/59ivxahjjapxw4yp26f5mdd5c3p8gd6k-xz-5.4.4-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/lpmb6y3gd25blxv7bvlgn7d4jlvrjszm-xz-5.4.4-man' from 'https://cache.nixos.org'...
copying path '/nix/store/f93zpvx7xdz29n736ysdc6h5660d2xx6-zstd-1.5.5-man' from 'https://cache.nixos.org'...
copying path '/nix/store/3ji0dxhhrhg1hpf2zp80jf9w0ls71yyn-stdenv-linux' from 'https://cache.nixos.org'...
copying path '/nix/store/0sr9jxvk9ynb7q3zics43a4dq8pm78r0-perl5.38.2-DBD-SQLite-1.74' from 'https://cache.nixos.org'...
copying path '/nix/store/n2x4y1n23m8xnq1gpl3f9vf43vfifcf9-glib-2.78.6' from 'https://cache.nixos.org'...
copying path '/nix/store/r2cs7pyba2r2rfdyfgv4yaxgswq5kbx6-lesspipe-2.10' from 'https://cache.nixos.org'...
building '/nix/store/5yzhvqdq2csasnb2hhi0n5pbj6qx68ym-builder.pl.drv'...
building '/nix/store/rbpdj3vmn0ar8jizqzvm936s22q0dldb-container-init.drv'...
building '/nix/store/0xbrnslanjpfyc2w8lcz69zqvgy5v2hy-ensure-all-wrappers-paths-exist.drv'...
building '/nix/store/ws7a83bqgn0nvs7vv43hynbzgv974vk5-etc-man_db.conf.drv'...
building '/nix/store/86h8i1v8zl69ksg4w4x4qplvf63nsycj-etc-nix-registry.json.drv'...
building '/nix/store/fa6m7x962p8aaxdj5z3gpghib7wjlfhr-etc-nsswitch.conf.drv'...
building '/nix/store/gfq1q1pn8a6qpwnc58fp63nahq03b72l-etc-pam-environment.drv'...
building '/nix/store/67kjnylsaanzqr0nlxm18f10ckfn20sb-etc-resolvconf.conf.drv'...
Checking that Nix store paths of all wrapped programs exist... OK
copying path '/nix/store/ls7wal9y40rjl21p6bzrx657k9wbil4j-binutils-wrapper-2.40' from 'https://cache.nixos.org'...
copying path '/nix/store/hdx2aw4b1cdy6r7x65zhvia0iinlz53i-rsync-3.2.7' from 'https://cache.nixos.org'...
copying path '/nix/store/hr35zmy28hayf0b23anszlxcjkdwvb0m-desktop-file-utils-0.26' from 'https://cache.nixos.org'...
copying path '/nix/store/bmcypdzbidmpam7np26hn25gs7idf88v-gcc-12.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/12f878dy1sxvffiaa4w26w9r7xwd6k70-shared-mime-info-2.4' from 'https://cache.nixos.org'...
building '/nix/store/4km8xgb3hjbgjcq8g87nhynax9w7v312-etc-sysctl.d-60-nixos.conf.drv'...
building '/nix/store/6xhmszkx2vj7vhmqqw4agav2hvzpll92-etc-systemd-networkd.conf.drv'...
building '/nix/store/ijy99blkgm44x5z05g7ca78gbd9n2a15-etc-systemd-resolved.conf.drv'...
building '/nix/store/ngkkp7zap1jvyqf29hdw8sl0incwiv48-init-script-builder.sh.drv'...
building '/nix/store/ckcsya240ahd7jzcqv7hs1b2z99jskb5-issue.drv'...
building '/nix/store/5fxb0xvbbxriknkd2k43ah7ipm3jcynz-local-cmds.drv'...
building '/nix/store/4crg85s6bbm7p0b8ahpl8j63wl7macs6-nixos-container.drv'...
building '/nix/store/xb194c7g3xpykyikxd0h1y7dn7anvg5m-set-environment.drv'...
building '/nix/store/xqvnh5242vzvd072dn6pqddgsjkbykf1-unit-40-host0.network.drv'...
building '/nix/store/q46f7qzjclj8n5rdyicdxrgczqcxvv2c-unit-autovt-.service-disabled.drv'...
building '/nix/store/qzq0bjqvnmxlwkha2cq52lng1v5ch77g-unit-multi-user.target.drv'...
building '/nix/store/4qkjddmdpd6lni0gs8s1h0n0a1fqxm32-X-Restart-Triggers-systemd-networkd.drv'...
building '/nix/store/56b9a05kh4qs04r8l13pv6149ljl8cnk-X-Restart-Triggers-systemd-resolved.drv'...
building '/nix/store/0gdkp4yf8rpdw1ij2f3z8h0yfxwldhdq-X-Restart-Triggers-systemd-sysctl.drv'...
building '/nix/store/fckd8hpr4g92b7cir19vfib689phrk9m-unit-network-local-commands.service.drv'...
building '/nix/store/064swcmyh8kf9djy89s5s6ylydvlb25b-unit-script-container_-post-start.drv'...
building '/nix/store/2s32c6vanhhk9457vmzbg4z5pw9b7f2d-unit-script-container_-pre-start.drv'...
building '/nix/store/rs7wif13j05zvfxd1p0rlazp1vsc5k36-X-Reload-Triggers-systemd-networkd.drv'...
building '/nix/store/1lpn686rfcbdcvw3bkip7j1nyhf30875-etc-profile.drv'...
building '/nix/store/4566i23r8g34f9clkdpy6yrwql73a9jc-reload-container.drv'...
building '/nix/store/ykcs66m9w6dyc3m6hnh36wcyigj3acww-stage-2-init.sh.drv'...
building '/nix/store/1ha9w74v2n21svqd7x4bvk0nrfbsvjvs-unit-script-container_-start.drv'...
building '/nix/store/9zbca7slsccz6a0sb0mwinfgpnpyl1w9-unit-systemd-network-wait-online-.service.drv'...
building '/nix/store/d4x034nzh1gsmg40x3bd012kbrqhc08m-unit-systemd-networkd-wait-online.service.drv'...
building '/nix/store/mliq87756jgcfavj5726w8fv8dzcqyqh-unit-systemd-networkd.socket.drv'...
building '/nix/store/zrsnwk21v8l2d0rf04h0s90lq342qlam-unit-systemd-resolved.service.drv'...
building '/nix/store/i8g01lkvnm4qlsba6n5yv74jn2kldq6y-unit-systemd-sysctl.service.drv'...
building '/nix/store/amah5ww5y5b1mgsvillmsklrixh0s9nj-unit-systemd-vconsole-setup.service-disabled.drv'...
building '/nix/store/aqr1iv3clbd13nk9wj7fsjwx3q7f0xkm-users-groups.json.drv'...
building '/nix/store/8n2cj6zlpdjr9gp2nb3lfvvi8x1vrrj8-unit-systemd-networkd.service.drv'...
building '/nix/store/5si9zpn75ggnmgdmrxmdk11nacr4fk71-unit-container-.service.drv'...
copying path '/nix/store/shdkxxdlpv66wl33rh9d01zfypaka1dn-gcc-wrapper-12.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/rxwbhj6snisrqxvmlllndsgm8xjb09c3-make-binary-wrapper-hook' from 'https://cache.nixos.org'...
building '/nix/store/23nc6abg43r4irvppdrzs2ng9idj9rwq-perl-5.38.2-env.drv'...
created 481 symlinks in user environment
building '/nix/store/lhkw2f702bkgwk117fg627j0vs0v70fr-command-not-found.drv'...
building '/nix/store/vzs0i4w46na1bhsx26x63hgv6jgq19ld-etc-bashrc.drv'...
building '/nix/store/jaqj8lkqsplr102p5rypqghsay9m8pzg-system-path.drv'...
created 5987 symlinks in user environment
building '/nix/store/jndsd89lkp89b04f7xd2rfnhb7v9glci-dbus-1.drv'...
building '/nix/store/sqfcgfp95psy71cpws51kaq8b6ppd8ba-X-Restart-Triggers-dbus.drv'...
building '/nix/store/5bgm1xjcca8dz1ywbb9f21k1b62gnvz2-unit-dbus.service.drv'...
building '/nix/store/k5l3gvc4kjic8ib3h0ix7v6lkbp8ahvd-unit-dbus.service.drv'...
building '/nix/store/vvadv6brhjyiljba05fpxysmzpmy8z37-user-units.drv'...
building '/nix/store/8kspd3b9xlplf04g8zxjqr06ffsjy5gs-system-units.drv'...
building '/nix/store/n07s46ywwk39wbnarmrpzrbxrh8nafgi-etc.drv'...
building '/nix/store/fjfazh4kxf39jjrvn7jw2sqlfnns2ngm-nixos-system-nixos-23.11.7665.03d771e513ce.drv'...
stat: cannot read file system information for '/boot': No such file or directory
WARNING: /boot being on a different filesystem not supported by init-script-builder.sh
stopping the following units: dhcpcd.service, firewall.service, sshd.socket, systemd-sysctl.service
activating the configuration...
removing group ‘sshd’
removing group ‘dhcpcd’
removing user ‘sshd’
removing user ‘dhcpcd’
setting up /etc...
removing obsolete symlink ‘/etc/dhcpcd.exit-hook’...
removing obsolete symlink ‘/etc/ssh/moduli’...
removing obsolete symlink ‘/etc/ssh/sshd_config’...
removing obsolete symlink ‘/etc/systemd/network/40-eth0.link’...
removing obsolete symlink ‘/etc/pam.d/sshd’...
reloading user units for root...
setting up tmpfiles
reloading the following units: dbus.service
starting the following units: systemd-sysctl.service
the following new units were started: systemd-networkd-wait-online.service, systemd-networkd.service, systemd-networkd.socket, systemd-resolved.service
All Done

The Files

Click to expand

My tempaltes/nixos/config file:

startup=0
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
# Turning off seccomp filtering improves performance at the expense of security
seccomp=1

# Use macvlan networking to provide an isolated network namespace,
# so docker can manage firewall rules
# Alternatively use --network-macvlan=eno1 instead of --network-bridge
# Ensure to change eno1/br1 to the interface name you want to use
# You may want to add additional options here, e.g. bind mounts
systemd_nspawn_user_args=--network-bridge=br0
#   --resolv-conf=bind-host
#   --system-call-filter='add_key keyctl bpf'

# Script to run on the HOST before starting the jail
# Load kernel module and config kernel settings required for docker
pre_start_hook=#!/usr/bin/bash
    set -euo pipefail
    echo 'PRE_START_HOOK - NixOS Initialization'
    JAIL_ROOT="$PWD"
    JAIL_ROOTFS="$JAIL_ROOT/rootfs"
    JAIL_SRC="$JAIL_ROOT/../../templates/nixos"

    if [ ! -e $JAIL_ROOT/initialized ]; then
        install -m 0755 -d "$JAIL_ROOTFS/etc"
        install -m 0755 -d "$JAIL_ROOTFS/etc/nixos"

        install -m 0644 -T $JAIL_SRC/configuration.nix $JAIL_ROOTFS/etc/nixos/configuration.nix
        install -m 0600 -T $JAIL_SRC/jail-init.script  $JAIL_ROOTFS/jail-init.script
        date -Iseconds > $JAIL_ROOT/initialized
    fi

# Only used while creating the jail
distro=nixos
release=23.11

# NOTE: for some reason this currently does not run, it is recommended you run this manually after first boot.
initial_setup=#!/usr/bin/bash
  set -euo pipefail
  [ -r /jail-init.script ] && bash -e /jail-init.script

My tempaltes/nixos/configuration.nix file:

# Initial /etc/nixos/configuration.nix file for Jail

{
  config,
  lib,
  pkgs,
  modulesPath,
  ...
}:
let
  hostName = "nixos";
  interface = "host0";
in
{
  imports = [
  ];

  boot.isContainer = true;
  boot.loader.initScript.enable = true;

  networking.hostName = "${hostName}";

  # Enable networkd
  systemd.network.enable = true;
  networking.useNetworkd = true;

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault false;
  networking.interfaces."${interface}".useDHCP = lib.mkDefault true;
  networking.useHostResolvConf = false;
  networking.firewall.enable = false;

# # configure static networking through systemd
# systemd.network.networks."10-host" = {
#   # how can I make this generic enough
#   matchConfig.Name = "${interface}";
#   address = [
#     "192.168.1.125/24"
#   ];
#   gateway = [
#     "192.168.1.254"
#   ];
#   dns = [
#     "192.168.1.53"
#   ];
#   domains = [
#     "example.com"
#   ];
# };

# nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";

  system.stateVersion = "23.11";
}

My tempaltes/nixos/jail-init.script file:

#!/usr/bin/bash

set -e

install -m 0644 -T <(echo "nameserver 1.1.1.1") "$JAIL_ROOTFS/etc/resolv.conf"
install -m 0644 -T /dev/null "$JAIL_ROOTFS/etc/dhcpcd.conf"

if [ "down" == $(cat /sys/class/net/host0/operstate) ]; then
    echo "Requesting DHCP lease..."
    dhcpcd
    sleep 3s
fi

[ "up" == $(cat /sys/class/net/host0/operstate) ]

echo "Checking for a LAN connection on 192.168..."
printf 'Time: .'
printf '........%s0' {1..8}
printf ' seconds \nWait: '
# wait for gateway connection (if STP is on, this can take ~60 seconds)
while true; do
    read _ _ gateway _ _ _ _ _ ipaddress _ < <(ip route list match 0/0) || true
    [ "${gateway#192\.168}" = "${gateway}" ] || break
    printf '.'; sleep 1;
done
printf '\n\nIP Assigned(%s)\n' "$ipaddress"

echo "Checking for connection with the Web..."
printf 'Time: .'
printf '........%s0' {1..8}
printf ' seconds \nWait: '
until curl https://api.ipify.org 2>/dev/null; do printf '.'; sleep 1; done; printf '\n\nCan reach the Web\n'

nixos-rebuild switch

# System is ready, move the init script into /tmp - this way, after
# reboot it will be deleted
mv $0 /tmp

echo "All Done"

If you find the reason the initial_setup didn't run, the entirety of jail-init.script can move into initial_setup.

[Jip-Hop]

Please add the jail-init.script code to initial_setup and turn on tracing with set -x somewhere at the top. Then we can see where it's going wrong. Because it looks like it's now actually executing the script from within /root. We're getting closer! :)

[Lockszmith-GH]

Damn, this was a finicky one.

I tried with set -x and never got anywhere, then I looked at the initial_setup code in your script, the invocation code has:

                "--property=After=network-online.target",
                "--property=Wants=network-online.target",

I thought that removing these will allow the script to run, and I was right.
(BTW, I changed the initial_setup_file_jailed_path assignment from "/root/jlmkr-initial-setup" to "/etc/jlmkr-initial-setup"

But then it failed immedietly.

Further investigation revealed that NixOS LXC image doesn't put anything in /bin and /usr/bin except /bin/sh and /usr/bin/env 😬 , bash and (ls and which and) everyhting else is /run/current-system/sw/bin - and that wasn't in the $PATH😒.

Setting the shebang to #! /run/current-system/sw/bin/bash and sourcing . /etc/bashrc as the first step resolved that issue, and now the initialization process is completley automated.

So, the following is required:

1. jlmkr-initial-setup must be written into a location that exists. (/etc should be safe, but creating /root might do the trick as well)
2. We need the ability to run initial-setup without networking.
   a. Option 1: Add a pre-networking-initial-setup step.
   b. Option 2: Add a switch that ommits the Wait and After `arguments.
3. Never assume bash exists (sh is still in /bin/sh).
4. Never assume the systemd-run envrionment is initialized with basic variables (like $PATH).

Also note that, --quiet makes it hard to debug, so maybe a --verbose switch that removes it.
And one more thing - add --collect to the systemd-run execution, this will make it possible to execute systemd-run from the jail's shell internally repeadely for troubleshooting.

The initial_setup config section

click to expand

initial_setup=#!/run/current-system/sw/bin/bash
    . /etc/bashrc
    set -e

    install -m 0644 -T <(echo "nameserver 1.1.1.1") "$JAIL_ROOTFS/etc/resolv.conf"
    install -m 0644 -T /dev/null "$JAIL_ROOTFS/etc/dhcpcd.conf"

    if [ "down" == $(cat /sys/class/net/host0/operstate) ]; then
        echo "Requesting DHCP lease..."
        dhcpcd
        sleep 3s
    fi

    [ "up" == $(cat /sys/class/net/host0/operstate) ]

    echo "Checking for a LAN connection on 192.168..."
    printf 'Time: .'
    printf '........%s0' {1..8}
    printf ' seconds \nWait: '
    # wait for gateway connection (if STP is on, this can take ~60 seconds)
    while true; do
        read _ _ gateway _ _ _ _ _ ipaddress _ < <(ip route list match 0/0) || true
        [ "${gateway#192\.168}" = "${gateway}" ] || break
        printf '.'; sleep 1;
    done
    printf '\n\nIP Assigned(%s)\n' "$ipaddress"

    echo "Checking for connection with the Web..."
    printf 'Time: .'
    printf '........%s0' {1..8}
    printf ' seconds \nWait: '
    until curl https://api.ipify.org 2>/dev/null; do printf '.'; sleep 1; done; printf '\n\nCan reach the Web\n'

    nixos-rebuild switch

    # System is ready, move the init script into /tmp - this way, after
    # reboot it will be deleted
    mv $0 /tmp

    echo "All Done"

My modified version of the initial-setup code in jlmkr.py

click to expand

Changes I've made:

• use /etc instead of /root
• comment out --quiet
• print the systemd-run command before executing
• add --collect to systemd-run
• only remove the script if the run was successful.

    initial_setup = False

    # If there's no machine-id, then this the first time the jail is started
    if not os.path.exists(os.path.join(jail_rootfs_path, "etc/machine-id")) and (
        initial_setup := config.my_get("initial_setup")
    ):
        if not initial_setup.startswith("#!"):
            initial_setup = "#!/bin/sh\n" + initial_setup

        initial_setup_file_jailed_path = "/etc/jlmkr-initial-setup"
        initial_setup_file_host_path = os.path.abspath(
            jail_rootfs_path + initial_setup_file_jailed_path
        )

        # Write a script file to call during initial setup
        print(initial_setup, file=open(initial_setup_file_host_path, "w"))
        stat_chmod(initial_setup_file_host_path, 0o700)

        # Ensure the jail init system is ready before we start the initial_setup
        systemd_nspawn_additional_args += [
            "--notify-ready=yes",
        ]

    cmd = [
        "systemd-run",
        *shlex.split(config.my_get("systemd_run_default_args")),
        *systemd_run_additional_args,
        "--",
        "systemd-nspawn",
        *shlex.split(config.my_get("systemd_nspawn_default_args")),
        *systemd_nspawn_additional_args,
        *shlex.split(config.my_get("systemd_nspawn_user_args")),
    ]

    print(
        dedent(
            f"""
        Starting jail {jail_name} with the following command:

        {shlex.join(cmd)}
    """
        )
    )

    returncode = subprocess.run(cmd).returncode
    if returncode != 0:
        eprint(
            dedent(
                f"""
            Failed to start jail {jail_name}...
            In case of a config error, you may fix it with:
            {COMMAND_NAME} edit {jail_name}
        """
            )
        )

        return returncode

    # Handle initial setup after jail is up and running (for the first time)
    if initial_setup:
        print("About to run the initial setup.")
        print("Waiting for networking in the jail to be ready.")
        print("Please wait (this may take 90s in case of bridge networking with STP is enabled)...")

        init_cmd = [
                "--",
                "systemd-run",
                f"--unit={os.path.basename(initial_setup_file_jailed_path)}",
                #"--quiet",
                "--collect",
                "--pipe",
                "--wait",
                "--service-type=exec",
                #"--property=After=network-online.target",
                #"--property=Wants=network-online.target",
                initial_setup_file_jailed_path,
            ]
        print(
            dedent(
                f"""
            Running initial_setup within the {jail_name} jail with the following command:

            {shlex.join(init_cmd)}
        """
            )
        )

        returncode = exec_jail(
            jail_name,
            init_cmd,
        )

        if returncode != 0:
            eprint("Tried to run the following commands inside the jail:")
            eprint(initial_setup)
            eprint()
            eprint(
                f"""{RED}{BOLD}Failed to run initial setup... you may want to stop and remove the jail and try again.{NORMAL}"""
            )
            return returncode
        else:
            print(f"Done with initial setup of jail {jail_name}!")
            # Cleanup the initial_setup_file_host_path
            if initial_setup_file_host_path:
                Path(initial_setup_file_host_path).unlink(missing_ok=True)

    return returncode

[Jip-Hop]

@Lockszmith-GH @templehasfallen @dkowis
The easiest way I found to setup nixos seems to be:

./jlmkr.py create --distro=nixos --release=24.05 nixos --network-bridge=br1 \
    --resolv-conf=bind-host --bind-ro=./lxd.nix:/etc/nixos/lxd.nix
echo '{ ... }:{}' > ./jails/nixos/lxd.nix
./jlmkr.py start nixos
sleep 5
./jlmkr.py shell nixos /bin/sh -c 'nixos-rebuild switch'

This will create a new jail with the latest nixos release using bridge networking.

It will also create an empty config file, which is then bind mounted, since the default LXC configuration.nix expects this file to exist. LXC would normally mount that file containing the hostname.

The configuration.nix also includes lxc-container.nix, which is made to specifically support nspawn too.

Sleep a little until the network is ready. You may have to increase the sleep time if you have STP enabled on your bridge interface. And then run nixos-rebuild switch to prove we can rebuild the system again. It will build and restart. I think from there on you have a normal nixos system and you can do whatever you want. 🙂