1.29.0: failure to build against libssl-dev 3.x

[fanquake]

Using master at af16084. Inside a Ubuntu Mantic container (libssl-dev 3.0.9).

apt update && apt upgrade -y
apt install gcc g++ git make patch libz-dev curl cmake python3 pkg-config libssl-dev -y
git clone https://github.com/thepowersgang/mrustc
cd mrustc
make RUSTCSRC
make -f minicargo.mk
make -C run_rustc
….
Error { kind: ToolExecError, message: "Command \"cc\" \"-O2\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-g\" \"-m64\" \"-Wall\" \"-Wextra\" \"-E\" \"/mrustc/output/cargo-build/build_openssl-sys-0_9_35/expando.c\" with args \"cc\" did not execute successfully (status code exit code: 1)." }

Failed to find OpenSSL development headers.

You can try fixing this setting the `OPENSSL_DIR` environment variable
pointing to your OpenSSL installation or installing OpenSSL headers package
specific to your distribution:

    # On Ubuntu
    sudo apt-get install libssl-dev
    # On Arch Linux
    sudo pacman -S openssl
    # On Fedora
    sudo dnf install openssl-devel

See rust-openssl README for more information:

    https://github.com/sfackler/rust-openssl#linux
', rustc-1.29.0-src/src/vendor/openssl-sys/build/main.rs:451:15
Process was terminated with signal 6
FAILING COMMAND:  /mrustc/output/cargo-build/build_openssl-sys-0_9_35_run

# cat /mrustc/output/cargo-build/build_openssl-sys-0_9_35.txt_failed.txt
<snipped>
HOST = Some("x86_64-linux-gnu")
TARGET = Some("x86_64-linux-gnu")
HOST = Some("x86_64-linux-gnu")
CFLAGS_x86_64-linux-gnu = None
CFLAGS_x86_64_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
DEBUG = Some("0")
running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-m64" "-Wall" "-Wextra" "-E" "/mrustc/output/cargo-build/build_openssl-sys-0_9_35/expando.c"
cargo:warning=/mrustc/output/cargo-build/build_openssl-sys-0_9_35/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
cargo:warning=    4 | #define VERSION2(n, v) RUST_VERSION_ ## n ## _ ## v
cargo:warning=      |                        ^~~~~~~~~~~~~
cargo:warning=/mrustc/output/cargo-build/build_openssl-sys-0_9_35/expando.c:5:23: note: in expansion of macro 'VERSION2'
cargo:warning=    5 | #define VERSION(n, v) VERSION2(n, v)
cargo:warning=      |                       ^~~~~~~~
cargo:warning=/mrustc/output/cargo-build/build_openssl-sys-0_9_35/expando.c:7:1: note: in expansion of macro 'VERSION'
cargo:warning=    7 | VERSION(OPENSSL, OPENSSL_VERSION_NUMBER)
cargo:warning=      | ^~~~~~~
exit code: 1

Note that Ubuntu Jammy onwards ships with a 3.x version of libssl dev, and Jammy is currently the version used in the CI here (ubuntu-latest), so it's not clear to me why the CI build are passing? I can't see any additional libssl installation/building 1.1 from source etc.

I've checked that builds work as expected inside a Ubuntu Focal container, which ships with libssl 1.1.

[thepowersgang]

It sounds like openssl-sys 0.9.35 is broken with that version of libssl, it doesn't surprise me given that 1.19 cargo no longer builds on my debian build environment due to SSL versions.

CI passes becasue it doesn't do a full build of cargo/rustc - it just does a smoke test of libstd

[fanquake]

@thepowersgang fair enough. Is this something you're interested in fixing? Looks like there are ways to work around the libssl incompatiblities (although I'm not completetly sure of the ramifications of doing so yet). Otherwise, does it make sense to document what is / isn't expected to work somewhere? I'm happy to work on either.

[thepowersgang]

Hmm... it could be fixable by patching openssl-sys to support newer versions, but then building the n+1 version won't work without similar patching.

I've started ignoring 1.19 when building on linux due to the openssl issues, so I guess it's time to do the same for 1.29.

[stikonas]

I'm actually hitting the same issue when building for 1.54... So it seems we would either need mrustc to support newer even newer rust or we need to build openssl 1.1 locally (perhaps outside the standard prefix).

(My version of openssl is 3.0.10)

[thepowersgang]

Solution! Enable the vendored-openssl feature, which seems to be present on cargo since 1.29

[stikonas]

Solution! Enable the vendored-openssl feature, which seems to be present on cargo since 1.29

That probably works once we have cargo built. Presumably, in the bootstrap step when we only have minicargo available, we need to first manually build openssl from rustc-*-src/vendor/openssl and set OPENSSL_DIR to point to it. Or am I misunderstanding how minicargo works.

[thepowersgang]

minicargo is perfectly capable of running build scripts, and thus it should be able to enable that feature and build.
I'd test, but all of my machines are currently working on a branch.

[thepowersgang]

Update - Ran a test while doing other tests, and unfortunately there's a bug in how minicargo handles features (it doesn't recursively activate features).
I'll work on fixing that once the job system is cleaned up

[thepowersgang]

Fixed the cargo bug in the above commit. 1.29 can build the vendored openssl, but hits link errors for some reason - likely something like libgit2/libssh/curl trying to use system SSL

[thepowersgang]

Looked like vendored openssl is (maybe) working for 1.39/1.54 - but 1.29 is very unhappy due to linking issues with libssh.

[stikonas]

Oh that's still very useful. 1.29 has very long bootstrap chain to current version anyway..

[stikonas]

Hmm, partial success, worked on x86_64 but on aarch64:

--- RUNNING openssl-sys v0.9.61 (script run) (75.4% 1r,5w,35b/167t)
> /home/andrius/git/mrustc/output-1.54.0/cargo-build/build_openssl-sys-0_9_61_H1_run > /home/andrius/git/mrustc/output-1.54.0/cargo-build/build_openssl-sys-0_9_61_H1.txt
 (75.4% 1r,5w,35b/167t): openssl-sys v0.9.61 (script run)
Ignoring `--emit llvm-ir` for compatability with rustc
thread 'main' panicked at 'don't know how to configure OpenSSL for aarch64-linux-gnu', :0:0
Process was terminated with signal 6
FAILING COMMAND: /home/andrius/git/mrustc/output-1.54.0/cargo-build/build_openssl-sys-0_9_61_H1_run
Env:  CARGO_FEATURE_VENDORED=1 CARGO_FEATURE_OPENSSL_SRC=1 OUT_DIR=/home/andrius/git/mrustc/output-1.54.0/cargo-build/build_openssl-sys-0_9_61_H1 CARGO_MANIFEST_DIR=/home/andrius/git/mrustc/rustc-1.54.0-src/vendor/openssl-sys CARGO_PKG_NAME=openssl-sys CARGO_PKG_VERSION=0.9.61 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=9 CARGO_PKG_VERSION_PATCH=61 TARGET=aarch64-linux-gnu HOST=aarch64-linux-gnu NUM_JOBS=1 OPT_LEVEL=2 DEBUG=0 PROFILE=release RUSTC=/home/andrius/git/mrustc/bin/mrustc MRUSTC_LIBDIR=/home/andrius/git/mrustc/output-1.54.0 CARGO_CFG_TARGET_VENDOR= CARGO_CFG_TARGET_VENDOR=gnu CARGO_CFG_TARGET_POINTER_WIDTH=64 CARGO_CFG_TARGET_OS=linux CARGO_CFG_TARGET_HAS_ATOMIC_LOAD_STORE=8 CARGO_CFG_TARGET_HAS_ATOMIC_LOAD_STORE=ptr CARGO_CFG_TARGET_HAS_ATOMIC_LOAD_STORE=64 CARGO_CFG_TARGET_HAS_ATOMIC_LOAD_STORE=32 CARGO_CFG_TARGET_HAS_ATOMIC_LOAD_STORE=16 CARGO_CFG_RUST_COMPILER=mrustc CARGO_CFG_TARGET_ENDIAN=little CARGO_CFG_TARGET_ARCH=aarch64 CARGO_CFG_TARGET_ENV=gnu CARGO_CFG_TARGET_HAS_ATOMIC=8 CARGO_CFG_TARGET_HAS_ATOMIC=cas CARGO_CFG_TARGET_HAS_ATOMIC=ptr CARGO_CFG_TARGET_HAS_ATOMIC=64 CARGO_CFG_TARGET_HAS_ATOMIC=32 CARGO_CFG_TARGET_HAS_ATOMIC=16 CARGO_CFG_TARGET_FAMILY=unix CARGO_CFG_TARGET_HAS_ATOMIC_EQUAL_ALIGNMENT=8 CARGO_CFG_TARGET_HAS_ATOMIC_EQUAL_ALIGNMENT=ptr CARGO_CFG_TARGET_HAS_ATOMIC_EQUAL_ALIGNMENT=64 CARGO_CFG_TARGET_HAS_ATOMIC_EQUAL_ALIGNMENT=32 CARGO_CFG_TARGET_HAS_ATOMIC_EQUAL_ALIGNMENT=16 CARGO_CFG_UNIX=1 CARGO_CFG_LINUX=1
Calling output-1.54.0/cargo-build/build_openssl-sys-0_9_61_H1_run failed (see output-1.54.0/cargo-build/build_openssl-sys-0_9_61_H1.txt_failed.txt for stdout)
 (76.0% 0r,5w,35b/167t):
BUILD FAILED
make: *** [minicargo.mk:263: output-1.54.0/cargo] Error 1

[thepowersgang]

Likely needs -unknown in the target name, had to do that to x86-64 too

[thepowersgang]

Hopefully fixed in the above

[stikonas]

Hopefully fixed in the above

I can now build cargo 1.54 on aarch64 again.

[thepowersgang]

Closing as the underlying issue isn't really fixable, and TBH - 1.29 is too old to really build for anything other than a test.