Prevent ninja starting new job when physical memory become low

[adelplanque]

I have a big project where

• 80% of jobs use a little amount of memory ==> jobs must be limited by CPU ressource
• 20% of jobs use large amount of memory ==> this jobs must be limited by memory ressouce

So like -l flags, it should be nice to have a -m flag :

ninja -m 1G

==> don't start new job if physical available memory is below 1G

[metti]

See #660 and #1354 for previous attempts.

[kunaltyagi]

Any chance of the PR getting merged? Or is the addition of another cmdline flag a blocking change? It'd be great to not have ninja || ninja -j1 as the build step 😄

[jhasse]

Any chance of the PR getting merged?

Not any time soon, sorry.

Or is the addition of another cmdline flag a blocking change?

I would prefer something like --keep-free-memory instead of -m.

I also wonder if this could be implemented in a GNU make jobserver.

[cmorty]

@jhasse : Any chance to move this forward with --keep-free-memory? I have some ugly template stuff that needs GB, while the rest needs far less than 100MB. Of course it would be perfect if a requirement could be bound to the job/jobpool, but not spawning new jobs below a certain limit should remove most of the pain.

[jhasse]

@cmorty I'm not working on this.

Ideally I think this should be fixed by giving the kernel more information about what's happening: If the kernel knew that the processes that ninja spawns don't have to run in parallel, it could swap enough of them out if memory is low.

[chen3feng]

Slow down the speed of job starting when the memory limit is near is also a considerable help.

[greg7mdp]

This would be great to have! What's the holdup?

[rustyx]

This is a must-have when doing LTO with clang on a modern CPU.
Running 32 c++ processes in parallel takes <4 GB RAM, but 32 ld processes at the end take 100 GB RAM, causing my system to lock up :(
And reducing parallelism to -j8 takes forever to compile.
At least need a separate argument to be able to specify compiler and linker parallelism separately (combinations can be computed relative to each other).