stdenv: single make jobserver across multiple nix builds

[pennae]

Motivation for this change

make -jN -lN in stdenv is a very blunt instrument. it works well when max-jobs=1, but as nix-level paralellism increases it becomes increasingly deficient. starting from a low-load situation we start max-jobs * N compilers, loadavg goes through the roof, the -lN load limit kicks in and inhibits new compilers starting until loadavg has fallen below N—at which point all make instances spawn a lot of new compilers and loadavg goes through the roof again. this oscillation leaves the system underutilized in low phases and overcommitted in high phases.

testing the current stdenv against a jobserver with 26 tokens on a 12C/24T machine shows that parallel builds of llvm_{8..11} run about 7% faster (35:52min for stdenv, 33:30min with jobserver), a larger build of llvm{5..13} is about about 11% faster (1:27h for stdenv, 1:17h with jobserver). (removing the -l from stdenv also improves utilization but is less efficient. preliminary testing here shows that -l${1.5 * N} may be a good alternative to -lN as used currently, #141266 could be a good vector to go for that instead of this whole mess. [more testing says that -l2N would be a minimum to get better utilization, but so far every -l setting we've tried has produced some underutilization except excessive large numbers like 6N or higher])

nothing in here should be regarded as a final suggestion in any way, it's more of a "hey look, this might just work". as such it's extremely rough around the edges, eg to use the jobserver the experimenter currently has to bring a /jobserver fifo filled with tokens into the nix sandbox:

nix-build -E 'with import ./. {}; pkgs.callPackage ./pkgs/os-specific/linux/nixos-jobserver {}'
touch /tmp/jobserver
sudo ./result -t 24 -g nixbld /tmp/jobserver&
nix build --extra-sandbox-paths "/jobserver=/tmp/jobserver" #...

is this something worth pursuing? a 10% speedup for hydra does seem tempting

todos before this is more generally usable:

• re-add $NIX_BUILD_CORES support (ioctl on the jobserver fd should do it)
• maybe port from cuse to fuse to not require root for everything (mmap problems may be solved by reporting jobserver file size as 0 at all times, needs testing)
• usability of tools (error messages, runtime configuration, etc)
• add support for other build systems (cargo can honor MAKEFLAGS, maybe others too)
• nixos module
• make configurable which env var(s) jscall adds jobserver information to
• metrics?
• <your suggestion here>

cc @vcunat @Artturin

[pennae]

the token loss problem could be solved on linux with a little fuse filesystem that emulates pipes, tracks how many tokens were taken from each file description of a pipe, and returns missing tokens when a pipe file description is released. darwin also has a fuse implementation, so there may just be a chance that if this works on linux it could also work reasonably well on darwin at some point.

[pennae]

updated with a CUSE-based jobserver fifo-alike. this variant survives nix-daemon killing builds hard without losing tokens, and in theory it could be implemented as a FUSE filesystem too (eg for darwin, or if the CUSE variant is deemed too exotic). at the moment it's CUSE mostly because FUSE doesn't seem to have a way to forbid mmapping of files, and while mmapping a token fd wouldn't break anything it would consume 4k tokens in one fell swoop and not give them back for a good while.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-global-jobserver-for-all-nix-builds/15923/1

[pennae]

updated with support for cargo, reintroduced per-build core limits, and added a nixos module. getting there 🤞

[expipiplus1]

Did this slip in by mistake?

[pennae]

nope. without this change this package will fail to build trying to build stdenv.jscall (because after this override stdenv.cc is a list, which is illegal. the jscall drv interpolates stdenv.cc into a bash string directly, thus failing here)