cross-compiling to aarch64-unkown-linux-{gnu,musl} from x86_64-linux

[viperML]

Hello!

I am trying to do a cross-build to ARM64 from my standard x86_64 PC, but failing in the process.

I have set up this minimal reproducer: https://github.com/viperML/cross-naersk

$ nix build github:viperML/cross-naersk#legacyPackages.x86_64-linux.cross-naersk.aarch64-unknown-linux-gnu -L
cross-naersk> .......
cross-naersk> cargo build $cargo_release -j "$NIX_BUILD_CORES" --message-format=$cargo_message_format
cross-naersk>    Compiling cross-naersk v0.1.0 (/build/source)
cross-naersk> error: could not compile `cross-naersk` due to 2 previous errors
cross-naersk> error: linking with `cc` failed: exit status: 1

The same linking errors appear by trying to do nix develop + cargo build manually.

I am already creating a toolchain with fenix that includes the target, and passing the env variable CARGO_BUILD_TARGET, but I think the problem is that the build environment uses some cc/binutils that is not suitable for the task.

Moreover, the examples cover how to build to a different target (-musl) of the same architecture, but not between architectures (Utimately I want to crossbuild to ARM64 and statically link against musl, but first thing first...)

EDIT:
I added a new branch that uses pkgsCross and callPackages naersk directly. It successfully crossbuilds a rustPlatorm-based package, while fails to find cc with naersk

$ nix build github:viperML/cross-naersk/pkgsCross#legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform.cross-naersk -L
...
cross-naersk-deps>    Compiling cross-naersk v0.1.0 (/build/dummy-src)
cross-naersk-deps> error: could not compile `cross-naersk` due to 2 previous errors
cross-naersk-deps> error: linker `cc` not found

$ nix build github:viperML/cross-naersk/pkgsCross#legacyPackages.x86_64-linux.pkgsCross.aarch64-multiplatform.cross-rustplatform -L

(OK)

[codyps]

I had similar issues in getting cross compilation from x86_64-darwin to x86_64-linux working using naersk.

After many variations, I found that carefully setting some env variables to point to the right cross compiler for cc seems to work. Here's the example setup I got working.

and inline `flake.nix` for posterity

{
  inputs = {
    nixpkgs.url = "nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    naersk = {
      url = "github:nix-community/naersk";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    fenix = {
      url = "github:nix-community/fenix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, flake-utils, naersk, nixpkgs, fenix }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = ((import nixpkgs) {
          inherit system;
        });

        crossPkgs = ((import nixpkgs) {
          inherit system;
          crossSystem = {
            config = "x86_64-unknown-linux-gnu";
          };
        });

        toolchain = with fenix.packages.${crossPkgs.stdenv.buildPlatform.system};
          combine [
            minimal.rustc
            minimal.cargo
            targets.${crossPkgs.stdenv.targetPlatform.config}.latest.rust-std
          ];

        naersk' = pkgs.callPackage naersk {
          cargo = toolchain;
          rustc = toolchain;
        };

        rust = crossPkgs.rust;
        lib = pkgs.lib;

        rustTargetPlatform = rust.toRustTarget crossPkgs.stdenv.targetPlatform;
        rustTargetPlatformUpper = lib.toUpper (
          builtins.replaceStrings [ "-" ] [ "_" ] rustTargetPlatform
        );
        targetCc = "${crossPkgs.stdenv.cc}/bin/${crossPkgs.stdenv.cc.targetPrefix}cc";
      in
      {
        defaultPackage = naersk'.buildPackage
          {
            CARGO_BUILD_TARGET = "${crossPkgs.stdenv.targetPlatform.config}";
            "CC_${rustTargetPlatform}" = "${targetCc}";
            "CARGO_TARGET_${rustTargetPlatformUpper}_LINKER" = "${targetCc}";
            depsBuildBuild = [ crossPkgs.stdenv.cc ];
            src = ./.;
            strictDeps = true;
          };

        formatter = pkgs.nixpkgs-fmt;
      }
    );
}

While I've defined a custom crossPkgs for x86_64-unknown-linux-gnu, it might be possible to use pkgs.pkgsCross.gnu64 instead.

IMO, it seems like a bug that naersk doesn't handle this for me when I do a naersk' = crossPkgs.callPackage naersk {}.

EDIT: unfortunately, this doesn't quite work completely. As soon as I add some macos frameworks in depsBuildBuild to fix build failures (for example, adding pkgs.darwin.apple_sdk.frameworks.SystemConfiguration), macos specific flags start getting picked up by the target cc (ie: x86_64-unknown-linux-gnu-gcc)

       >   running "/nix/store/b2smmmsv1zhjb584di9562rlkdpd08hl-x86_64-unknown-linux-gnu-stage-final-gcc-wrapper-12.3.0/bin/x86_64-unknown-linux-gnu-cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-DNDEBUG" "-c" "-o/private/tmp/nix-build-storysort-deps-0.1.0.drv-0/dummy-src/target/x86_64-unknown-linux-gnu/release/build/ring-afc72bf908900812/out/aesni-x86_64-elf.o" "/nix/store/3zkffyaawmi382w6y4cnc7k14yzsc2v4-dependencies/ring-0.16.20-3053cf52e236a3ed746dfc745aa9cacf1b791d846bdaf412f60a8d7d6e17c8fc/pregenerated/aesni-x86_64-elf.S"
       >   x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-iframework'
       >   x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-iframework'
       >   x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-iframework'
       >   thread 'main' panicked at /nix/store/3zkffyaawmi382w6y4cnc7k14yzsc2v4-dependencies/ring-0.16.20-3053cf52e236a3ed746dfc745aa9cacf1b791d846bdaf412f60a8d7d6e17c8fc/build.rs:656:9:
       >   execution failed
       >   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
       > warning: build failed, waiting for other jobs to finish...
       > [naersk] cargo returned with exit code 101, exiting
       For full logs, run 'nix log /nix/store/5518bfkafzbwjwlr0b83bghb8qapj59s-storysort-deps-0.1.0.drv'.

Probably being picked up via the wrapper for the compiler incorrectly.