[project/cmake] add auto linker selection based on availability

[xmkg]

mold and lld are usually several magnitudes faster compared to the default linker of the toolchains. This PR adds the configure_linker CMake function to automatically configure which linker to use based on availability of the "mold" and "lld" in the environment. The function is no-op below CMake 3.29 since the current implementation depends on a rather recent CMake variable, CMAKE_LINKER_TYPE.

The user can disable the auto linker selection by explicitly specifying the CMAKE_LINKER_TYPE at configure step, i.e.,:

cmake -DCMAKE_LINKER_TYPE=DEFAULT -S . -B build

The configure_linker is intended to be only run in non-CI environments. The is_running_in_ci determines whether the project is being built in a CI environment.

MULTI-1912

[xmkg]

A crude test to demonstrate the link speed difference.

Test steps:

• Build the project from scratch to a clean build folder
• Remove only the build/bin/multipass_tests (one of the usual suspects that has looong linking time)
• Trigger build again, nearly all of the time will be spent on linking since all object files are ready

-- Results --

Baseline: Default linker (GNU ld)
12.42 seconds

> cmake -DCMAKE_BUILD_TYPE=Debug -GNinja -DCMAKE_LINKER_TYPE=DEFAULT -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -S . -B build
> cmake --build build/ --parallel 32
> rm build/bin/multipass_tests 
> time cmake --build build/ --parallel 32 --target multipass_tests
[0/2] Re-checking globbed directories...
[1/1] Linking CXX executable bin/multipass_tests

[0/2] Re-checking globbed directories...
[1/1] Linking CXX executable bin/multipass_tests

real    0m12,426s
user    0m8,445s
sys     0m3,980s

Contender 1: mold
1.1 seconds (11.29x faster than the baseline, 1.22x faster than lld)

> cmake -DCMAKE_BUILD_TYPE=Debug -GNinja -DCMAKE_LINKER_TYPE=MOLD -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -S . -B build
> cmake --build build/ --parallel 32
> rm build/bin/multipass_tests 
> time cmake --build build/ --parallel 32 --target multipass_tests
[0/2] Re-checking globbed directories...
[1/1] Linking CXX executable bin/multipass_tests

real    0m1,105s
user    0m0,100s
sys     0m0,028s

Contender 2: lld
1.35 seconds (9.2x faster than the baseline)

> cmake -DCMAKE_BUILD_TYPE=Debug -GNinja -DCMAKE_LINKER_TYPE=LLD -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -S . -B build
> cmake --build build/ --parallel 32
> rm build/bin/multipass_tests 
> time cmake --build build/ --parallel 32 --target multipass_tests
[0/2] Re-checking globbed directories...
[1/1] Linking CXX executable bin/multipass_tests

real    0m1,351s
user    0m4,301s
sys     0m2,776s

Conclusion:
Without a doubt, "mold" and "lld" blow the "ld" out of the water.

[ricab]

Not a review, but this is a great idea! It should speed up iterating a lot without affecting our confidence on what we ship. 👍

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.34%. Comparing base (7cb6b2b) to head (afab3c7).
Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4013   +/-   ##
=======================================
  Coverage   89.34%   89.34%           
=======================================
  Files         260      260           
  Lines       14738    14738           
=======================================
  Hits        13167    13167           
  Misses       1571     1571           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andrei-toterman]

Hmm, I see that Linux CI runs log Running in CI environment? FALSE. Perhaps the env var you're looking for is not seen inside the snapcraft LXD container?

[xmkg]

Hmm, I see that Linux CI runs log Running in CI environment? FALSE. Perhaps the env var you're looking for is not seen inside the snapcraft LXD container?

Yes, that's the case. I opted to set the CMAKE_LINKER_TYPE in snapcraft.yaml instead of forwarding the environment variables. This kinda eliminates the point of having the check, but the code needs to ensure that the same type of linker is used for all CI jobs. What do you think? Should we forward the environment variable instead? That's a reasonable approach, too.

[andrei-toterman]

It's not a big deal, but I think it is a bit confusing that the CI logs report that the build was not running in CI. So if it isn't too difficult to fix, I'd like that. IMO, if forwarding the env var to snapcraft would not introduce any other issue, I think that approach would be more consistent.

[xmkg]

I think that approach would be more consistent.

Yeah, at least we can ditch the CI patchfile changes. Updated the branch, let's see if it works.

[xmkg]

It's not a big deal, but I think it is a bit confusing that the CI logs report that the build was not running in CI. So if it isn't too difficult to fix, I'd like that. IMO, if forwarding the env var to snapcraft would not introduce any other issue, I think that approach would be more consistent.

Done, the CI is now injecting the GITHUB_ACTIONS environment variable to build-environment. Also noticed that some of the previously injected variables are not using correct "is-set" tests, fixed them too.

[ricab]

Not an actual review, but I just want to add a couple of remarks

• First, this is a great idea, thanks a lot!
• Second, should we add a very brief line to our build instructions?

[xmkg]

• Second, should we add a very brief line to our build instructions?

Done!

[andrei-toterman]

Great, thanks!

[xmkg]

@levkropp It's ready for a secondary review, lemme know if it looks good to you :)

[levkropp]

Great work on this @xmkg ! I have tested building and I am seeing the speedup :)

[0/2] Re-checking globbed directories...
[1/2] Linking CXX executable bin/multipass_tests

real    0m1.019s
user    0m0.048s
sys     0m0.028s