Instrumenting with -Zbuild-std fails to link with musl

[brunocodutra]

cargo-pgo works great with -Zbuild-std for target x86_64-unknown-linux-gnu, but I have been unable to get it to work with musl. This is specific to the presence of the -Zbuild-std swtich, without it musl links just fine.

cargo pgo instrument build -- -Zbuild-std=core,alloc,std,panic_abort -Zbuild-std-features=panic_immediate_abort --target=x86_64-unknown-linux-musl

error: linking with `cc` failed: exit status: 12: cinder(bin)                           
  |
  = note:  "cc" "-m64" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/rcrt1.o" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crti.o" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtbeginS.o" "<1 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "-lunwind" "-lc" "/tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib" "/cinder/target/x86_64-unknown-linux-musl/release/deps/libcompiler_builtins-c07f7650b44b7d80.rlib" "-L" "/tmp/rustcF2cQyl/raw-dylibs" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-nostartfiles" "-L" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained" "-L" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/cinder/target/x86_64-unknown-linux-musl/release/deps/cinder-65a9da1bcc1fb533" "-static-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-all" "-nodefaultlibs" "-u" "__llvm_profile_runtime" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtendS.o" "<sysroot>/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingValue.o): in function `InstProfPopcountll':
          /checkout/src/llvm-project/compiler-rt/include/profile/InstrProfData.inc:961:(.text.InstProfPopcountll+0x5): undefined reference to `__popcountdi2'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/include/profile/InstrProfData.inc:961:(.text.InstrProfGetRangeRepValue+0x15): undefined reference to `__popcountdi2'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/include/profile/InstrProfData.inc:961:(.text.InstrProfIsSingleValRange+0xb): undefined reference to `__popcountdi2'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingValue.o): in function `lprofSetupValueProfiler':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingValue.c:46:(.text.lprofSetupValueProfiler+0x44): undefined reference to `atoi'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingValue.o): in function `InstProfPopcountll':
          /checkout/src/llvm-project/compiler-rt/include/profile/InstrProfData.inc:961:(.text.__llvm_profile_instrument_memop+0x1e): undefined reference to `__popcountdi2'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `fileWriter':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:353:(.text.fileWriter+0x85): undefined reference to `fseek'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `getProfileFileSizeForMerging':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:393:(.text.getProfileFileSizeForMerging+0x12): undefined reference to `fseek'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:398:(.text.getProfileFileSizeForMerging+0x1f): undefined reference to `ftell'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:401:(.text.getProfileFileSizeForMerging+0x2e): undefined reference to `fseek'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `mmapForContinuousMode':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:266:(.text.mmapForContinuousMode.isra.0+0xb4): undefined reference to `fileno'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `doProfileMerging':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:471:(.text.writeFile+0xc9): undefined reference to `fileno'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `openFileForMerging':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:516:(.text.writeFile+0xf2): undefined reference to `fseek'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `setupIOBuffer':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:382:(.text.writeFile+0x19c): undefined reference to `atoi'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `openFileForMerging':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:515:(.text.writeFile+0x21c): undefined reference to `fileno'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `initializeProfileForContinuousMode':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:688:(.text.initializeProfileForContinuousMode+0x13b): undefined reference to `ftell'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:689:(.text.initializeProfileForContinuousMode+0x143): undefined reference to `getpagesize'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `parseFilenamePattern':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:850:(.text.parseAndSetFilename+0x23e): undefined reference to `getpagesize'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `__llvm_profile_set_file_object':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:1234:(.text.__llvm_profile_set_file_object+0xe1): undefined reference to `fileno'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:1255:(.text.__llvm_profile_set_file_object+0x1e0): undefined reference to `fileno'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingFile.o): in function `setupIOBuffer':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:382:(.text.__llvm_profile_set_file_object+0x214): undefined reference to `atoi'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:382:(.text.__llvm_write_custom_profile+0x2a4): undefined reference to `atoi'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofGetHostName':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:136:(.text.lprofGetHostName+0x12): undefined reference to `uname'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofLockFileHandle':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:206:(.text.lprofLockFileHandle+0x7): undefined reference to `fileno'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofUnlockFileHandle':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:216:(.text.lprofUnlockFileHandle+0x7): undefined reference to `fileno'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofGetFileContentBuffer':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:330:(.text.lprofGetFileContentBuffer+0x1b): undefined reference to `fileno'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:348:(.text.lprofGetFileContentBuffer+0x7c): undefined reference to `ftell'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:332:(.text.lprofGetFileContentBuffer+0xc4): undefined reference to `fileno'
          /usr/bin/ld: /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:357:(.text.lprofGetFileContentBuffer+0x144): undefined reference to `feof'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofGetPathPrefix':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:396:(.text.lprofGetPathPrefix+0x48): undefined reference to `atoi'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofFindLastDirSeparator':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:448:(.text.lprofFindLastDirSeparator+0x6): undefined reference to `strrchr'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofReleaseMemoryPagesToOS':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:490:(.text.lprofReleaseMemoryPagesToOS+0xd): undefined reference to `getpagesize'
          /usr/bin/ld: /tmp/rustcF2cQyl/libprofiler_builtins-66af106da44a9e82.rlib(3868e1b678fe2595-InstrProfilingUtil.o): in function `lprofAtExit':
          /checkout/src/llvm-project/compiler-rt/lib/profile/InstrProfilingUtil.c:573:(.text.lprofAtExit+0x1): undefined reference to `atexit'
          collect2: error: ld returned 1 exit status

[Kobzol]

Hi, does it work with just cargo build, without cargo-pgo?

[brunocodutra]

Yes, I've just tested with the following.

cargo build --release -Zbuild-std=core,alloc,std,panic_abort -Zbuild-std-features=panic_immediate_abort --target=x86_64-unknown-linux-musl

[Kobzol]

I investigated this a bit and I think that the issue is profiler_builtins, which is required for PGO. When you compile for x64, the profiler_builtins crate is already available from the sysroot, so it is used. But it's not available for cross-compilation, so you have to build it from scratch. That means you'll have to add profiler_builtins to the -Zbuild-std crate list, and then find a way to actually compile it, which won't be so trivial, because it actually requires LLVM sources, IIRC.

The reproducer for your issue should be this:

RUSTFLAGS="-Cprofile-generate=/tmp" cargo build --release -Zbuild-std=core,alloc,std,panic_abort -Zbuild-std-features=panic_immediate_abort --target=x86_64-unknown-linux-musl

[brunocodutra]

Thanks for looking into it so promptly. I understand the problem now. I'll keep an eye on the rfc.