From e7adf3e83747c2915c671f2e560cde6f3d4a4905 Mon Sep 17 00:00:00 2001 From: Viktor Szakats Date: Sat, 28 Dec 2024 00:27:26 +0100 Subject: [PATCH] cmake: add pre-fill for Unix, enable in GHA/macos, verify pre-fills TL;DR: Save 10 minutes of CI time for GHA/macos jobs using pre-fills and add pre-fill verification for Apple and Windows. Also restores Xcode job and saves 1.5-10 minutes configuring iOS jobs. Pre-filling feature detection results can bring down the CMake configure step to ~5 seconds on most GHA runners, ~10 seconds in slow envs like Cygwin/MSYS2. The potential savings per job are: - 5-40 (average 19) seconds on GHA/macos (33 jobs) - ~10 seconds on GHA for iOS GNU Makefile (1 job) - 1.5-10 minutes on GHA for iOS Xcode generator (1 job) - 10 seconds on GHA/linux with native Ubuntu (12 jobs) - 40 seconds for Cygwin/MSYS2 (2 jobs) - 5-10 seconds for virtualized BSDs, native CPU (3 jobs) - ~60 seconds for virtualized BSDs, emulated CPU (1 job) On native Windows pre-filling has been in place for a long time and saving 8 minutes (VS2019-VS2015) to 1.5-2 minutes (VS2022), 3 minutes (VS2022 UWP), and 30-60 seconds (MinGW), per CI job. The downside is that detection results need to be manually collected and filtered to those that universally apply to all platforms that they are enabled on. Another downside is that by using a cache, we're not running the actual detections, and thus won't catch regressions in them. It means we must make sure that the cache is solid and matches with actual detections results. An upside is that it gives a rough overview of which features are available on which platforms. Another upside is pre-filled values do work for feature detections skipped for cross-builds, e.g. `HAVE_WRITABLE_ARGV`. This PR adds a pre-fill cache that supports all Unixes (except OmniOS) used in CI, and makes it usable with an internal option. It also enables it for GHA/macos CI jobs, where the maximum savings are. And also for the two iOS [1] and two Cygwin/MSYS2 jobs. The latters don't have pre-fill checks and we can drop them if they turn into a hassle. Saving: - 10 minutes of CI time per GHA/macos workflow run. [2] - ~80 seconds per GHA/windows workflow run with Cygwin/MSYS2. (offsetting the cost of pre-fill verifications) - 1.5-10 minutes per GHA/non-native runs with iOS jobs. [3] You can enable pre-fill locally with `-D_CURL_PREFILL=ON`. It's experimental, and if you experience a problem, file a PR or an Issue. This PR also adds a pre-fill checker for macOS and MinGW/MSVC Windows GHA jobs to catch if the cache diverges from real detections. It also adds this logic to AppVeyor, but doesn't enable it due to the perf penalty of 2 minutes mininum. The pre-fill checker works by configuring out-of-tree with and without pre-fill, then diffing their `lib/curl_config.h` outputs. Exceptions are 3 detection results exposed indirectly [4], and missing to expose 2, of which one is the C89 header `stddef.h`. While we assume the C99 `stdint.h` available outside iOS. We can expose them in the future, if necessary. The pre-fill checks cost in total: - ~20 seconds for macOS - ~40 seconds for MinGW on GHA - ~80 seconds for MSVC on GHA (UWP would be 2x this) An extra time saving potential is caching type sizes. They are well-known, and seldom change, esp. in CI. GHA/Windows jobs spend 8-17 seconds per job on these ~12 feature checks. ~5s on Cygwin/MSYS2. Couple of seconds on other platforms. (This PR doesn't make this optimization.) Another opportunity is doing the same for autotools, which typically spends more time in the configuration step than cmake. [1] Xcode job restored as a follow-up to be5f20202c1618788b3d8f6d255543638f48bd65 #16302 [2] GHA/macos cmake configure times in seconds: Job | Bef. | After | Gain :----------------------------------------------- | ----: | ----: | ----: CM clang GnuTLS !ldap krb5 | 21.2 | 4.5 | 16.7 CM clang LibreSSL !ldap heimdal c-ares +examples | 13.3 | 3.9 | 9.4 CM clang OpenSSL +static libssh +examples | 20.0 | 4.6 | 15.4 CM clang OpenSSL IDN clang-tidy~ (w/chkprefill) | 15.7 | 18.6 | -2.9 CM clang OpenSSL gsasl rtmp AppleIDN | 25.0 | 4.7 | 20.3 CM clang OpenSSL torture !FTP | 15.3 | 4.5 | 10.8 CM clang OpenSSL torture FTP | 25.0 | 5.9 | 19.1 CM clang SecureTransport debug | 18.0 | 3.8 | 14.2 CM clang macos-13 SecureTransport | 45.8 | 12.4 | 33.4 CM clang macos-14 SecureTransport | 15.8 | 4.6 | 11.2 CM clang macos-15 SecureTransport | 26.8 | 6.1 | 20.7 CM clang mbedTLS openldap brotli zstd | 15.1 | 6.5 | 8.6 CM clang wolfSSL !ldap brotli zstd | 27.0 | 4.4 | 22.6 CM gcc-12 GnuTLS !ldap krb5 | 39.1 | 8.7 | 30.4 CM gcc-12 LibreSSL !ldap heimdal c-ares +examples| 23.8 | 7.2 | 16.6 CM gcc-12 OpenSSL +static libssh +examples | 20.7 | 8.5 | 12.2 CM gcc-12 OpenSSL gsasl rtmp AppleIDN | 23.1 | 10.1 | 13.0 CM gcc-12 SecureTransport debug | 21.1 | 4.8 | 16.3 CM gcc-12 mbedTLS openldap brotli zstd | 21.4 | 5.8 | 15.6 CM gcc-12 wolfSSL !ldap brotli zstd | 21.1 | 6.9 | 14.2 CM gcc-14 macos-13 SecureTransport | 61.9 | 18.7 | 43.2 CM gcc-14 macos-14 SecureTransport | 30.5 | 6.4 | 24.1 CM gcc-14 macos-15 SecureTransport | 32.7 | 8.4 | 24.3 CM llvm@15 GnuTLS !ldap krb5 | 21.1 | 7.5 | 13.6 CM llvm@15 LibreSSL !ldap heimdal c-ares +exampl~| 24.6 | 6.8 | 17.8 CM llvm@15 OpenSSL +static libssh +examples | 19.0 | 6.4 | 12.6 CM llvm@15 OpenSSL gsasl rtmp AppleIDN | 19.0 | 8.2 | 10.8 CM llvm@15 SecureTransport debug | 18.0 | 5.4 | 12.6 CM llvm@15 macos-13 SecureTransport | 66.2 | 25.7 | 40.5 CM llvm@15 macos-14 SecureTransport | 31.9 | 6.1 | 25.8 CM llvm@15 mbedTLS openldap brotli zstd | 19.5 | 8.9 | 10.6 CM llvm@15 wolfSSL !ldap brotli zstd | 24.3 | 5.9 | 18.4 CM llvm@18 macos-15 SecureTransport | 33.8 | 6.4 | 27.4 Total | 856.8 | 257.3 | 599.5 Before: https://github.com/curl/curl/actions/runs/13311042735/job/37173478424 After: https://github.com/curl/curl/actions/runs/13313927119/job/37183206426?pr=15841 [3] iOS: Before: https://github.com/curl/curl/actions/runs/13326401704?pr=15841 After: https://github.com/curl/curl/actions/runs/13332177764?pr=15841 [4] detection results exposed indirectly in `curl_config.h`: - `HAVE_FILE_OFFSET_BITS` via `_FILE_OFFSET_BITS` - `HAVE_GETHOSTBYNAME_R_*_REENTRANT` via `NEED_REENTRANT` - `HAVE_SOCKADDR_IN6_SIN6_ADDR` via `USE_IPV6` Closes #15841 --- .github/workflows/macos.yml | 22 ++- .github/workflows/non-native.yml | 15 +- .github/workflows/windows.yml | 94 ++++++---- CMake/unix-cache.cmake | 313 +++++++++++++++++++++++++++++++ CMakeLists.txt | 21 ++- Makefile.am | 1 + appveyor.sh | 52 ++--- 7 files changed, 442 insertions(+), 76 deletions(-) create mode 100644 CMake/unix-cache.cmake diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml index dcdede5217ae..f53ce7d04b4f 100644 --- a/.github/workflows/macos.yml +++ b/.github/workflows/macos.yml @@ -135,6 +135,7 @@ jobs: install: llvm generate: -DOPENSSL_ROOT_DIR=$(brew --prefix openssl) -DUSE_APPLE_IDN=ON -DCURL_CLANG_TIDY=ON -DCLANG_TIDY=$(brew --prefix llvm)/bin/clang-tidy clang-tidy: true + chkprefill: _chkprefill - name: 'OpenSSL +static libssh +examples' install: libssh generate: -DOPENSSL_ROOT_DIR=$(brew --prefix openssl) -DBUILD_STATIC_LIBS=ON -DCURL_USE_LIBSSH2=OFF -DCURL_USE_LIBSSH=ON @@ -228,11 +229,19 @@ jobs: fi if [ -n '${{ matrix.build.generate }}' ]; then - [ -n '${{ matrix.build.macos-version-min }}' ] && options+=' -DCMAKE_OSX_DEPLOYMENT_TARGET=${{ matrix.build.macos-version-min }}' - cmake -B bld -G Ninja -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ - -DCMAKE_OSX_SYSROOT="${sysroot}" \ - -DCMAKE_C_COMPILER_TARGET="$(uname -m | sed 's/arm64/aarch64/')-apple-darwin$(uname -r)" \ - ${{ matrix.build.generate }} ${options} + for _chkprefill in '' ${{ matrix.build.chkprefill }}; do + options='' + [ -n '${{ matrix.build.macos-version-min }}' ] && options+=' -DCMAKE_OSX_DEPLOYMENT_TARGET=${{ matrix.build.macos-version-min }}' + [ "${_chkprefill}" = '_chkprefill' ] && options+=' -D_CURL_PREFILL=OFF' + cmake -B "bld${_chkprefill}" -G Ninja -D_CURL_PREFILL=ON \ + -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ + -DCMAKE_OSX_SYSROOT="${sysroot}" \ + -DCMAKE_C_COMPILER_TARGET="$(uname -m | sed 's/arm64/aarch64/')-apple-darwin$(uname -r)" \ + ${{ matrix.build.generate }} ${options} + done + if [ -d bld_chkprefill ]; then + diff -u bld/lib/curl_config.h bld_chkprefill/lib/curl_config.h + fi else export CFLAGS if [[ '${{ matrix.compiler }}' = 'llvm'* ]]; then @@ -443,7 +452,8 @@ jobs: [ '${{ matrix.config }}' = 'SecureTransport' ] && options+=' -DCURL_USE_SECTRANSP=ON' [ -n '${{ matrix.macos-version-min }}' ] && options+=' -DCMAKE_OSX_DEPLOYMENT_TARGET=${{ matrix.macos-version-min }}' # would pick up nghttp2, libidn2, and libssh2 - cmake -B bld -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ + cmake -B bld -D_CURL_PREFILL=ON \ + -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ -DCMAKE_OSX_SYSROOT="${sysroot}" \ -DCMAKE_C_COMPILER_TARGET="$(uname -m | sed 's/arm64/aarch64/')-apple-darwin$(uname -r)" \ -DCMAKE_IGNORE_PREFIX_PATH="$(brew --prefix)" \ diff --git a/.github/workflows/non-native.yml b/.github/workflows/non-native.yml index dcae3c326eb6..e620b970fe39 100644 --- a/.github/workflows/non-native.yml +++ b/.github/workflows/non-native.yml @@ -266,6 +266,17 @@ jobs: -DOPENSSL_CRYPTO_LIBRARY="$HOME/libressl/lib/libcrypto.a" -DCURL_USE_LIBPSL=OFF + - name: 'libressl' + install_steps: libressl + generator: Xcode + generate: >- + -DCMAKE_XCODE_ATTRIBUTE_CODE_SIGNING_ALLOWED=OFF + -DMACOSX_BUNDLE_GUI_IDENTIFIER=se.curl + -DOPENSSL_INCLUDE_DIR="$HOME/libressl/include" + -DOPENSSL_SSL_LIBRARY="$HOME/libressl/lib/libssl.a" + -DOPENSSL_CRYPTO_LIBRARY="$HOME/libressl/lib/libcrypto.a" + -DCURL_USE_LIBPSL=OFF + steps: - name: 'brew install' if: ${{ matrix.build.configure }} @@ -323,7 +334,9 @@ jobs: run: | if [ -n '${{ matrix.build.generate }}' ]; then # https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#cross-compiling-for-ios-tvos-visionos-or-watchos - cmake -B bld -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ + [ -n '${{ matrix.build.generator }}' ] && options='-G ${{ matrix.build.generator }}' + cmake -B bld -D_CURL_PREFILL=ON \ + -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON -DCURL_WERROR=ON \ -DCMAKE_SYSTEM_NAME=iOS \ -DUSE_APPLE_IDN=ON \ ${{ matrix.build.generate }} ${options} diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml index 29641393352e..d3fdec919531 100644 --- a/.github/workflows/windows.yml +++ b/.github/workflows/windows.yml @@ -85,7 +85,7 @@ jobs: run: | if [ '${{ matrix.build }}' = 'cmake' ]; then PATH="/usr/bin:$(cygpath "${SYSTEMROOT}")/System32" - cmake -B bld -G Ninja ${options} \ + cmake -B bld -G Ninja -D_CURL_PREFILL=ON ${options} \ -DCMAKE_UNITY_BUILD=ON -DCMAKE_UNITY_BUILD_BATCH_SIZE=30 -DCURL_TEST_BUNDLES=ON \ -DCURL_WERROR=ON \ ${{ matrix.config }} @@ -180,7 +180,7 @@ jobs: # FIXME: WebSockets test results ignored due to frequent failures on native Windows: - { build: 'cmake' , sys: 'mingw64', env: 'x86_64' , tflags: '' , config: '-DENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=OFF -DCURL_USE_SCHANNEL=ON -DENABLE_UNICODE=ON -DENABLE_ARES=ON', type: 'Debug', name: 'schannel c-ares U' } - { build: 'cmake' , sys: 'ucrt64' , env: 'ucrt-x86_64' , tflags: 'skiprun' , config: '-DENABLE_DEBUG=OFF -DBUILD_SHARED_LIBS=ON -DCURL_USE_SCHANNEL=ON -DENABLE_UNICODE=ON -DENABLE_CURLDEBUG=ON', type: 'Release', name: 'schannel R TrackMemory' } - - { build: 'cmake' , sys: 'clang64', env: 'clang-x86_64', tflags: 'skiprun' , config: '-DENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=OFF -DCURL_USE_OPENSSL=ON -DENABLE_UNICODE=OFF', type: 'Release', name: 'openssl' } + - { build: 'cmake' , sys: 'clang64', env: 'clang-x86_64', tflags: 'skiprun' , config: '-DENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=OFF -DCURL_USE_OPENSSL=ON -DENABLE_UNICODE=OFF', type: 'Release', name: 'openssl', chkprefill: '_chkprefill' } - { build: 'cmake' , sys: 'ucrt64' , env: 'ucrt-x86_64' , tflags: 'skiprun' , config: '-DENABLE_DEBUG=OFF -DBUILD_SHARED_LIBS=ON -DCURL_USE_SCHANNEL=ON', type: 'Release', test: 'uwp', name: 'schannel' } # { build: 'autotools', sys: 'ucrt64' , env: 'ucrt-x86_64' , tflags: 'skiprun' , config: '--without-debug --with-schannel --enable-shared', type: 'Release', test: 'uwp', name: 'schannel' } - { build: 'cmake' , sys: 'mingw64', env: 'x86_64' , tflags: 'skiprun' , config: '-DENABLE_DEBUG=ON -DBUILD_SHARED_LIBS=ON -DCURL_USE_SCHANNEL=ON -DENABLE_UNICODE=ON -DCMAKE_VERBOSE_MAKEFILE=ON', type: 'Debug', cflags: '-DCURL_SCHANNEL_DEV_DEBUG', name: 'schannel dev debug' } @@ -218,6 +218,7 @@ jobs: mingw-w64-${{ matrix.env }}-libssh2 mingw-w64-${{ matrix.env }}-libpsl mingw-w64-${{ matrix.env }}-c-ares + ${{ matrix.chkprefill == '_chkprefill' && format('mingw-w64-{0}-diffutils', matrix.env) || '' }} - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 with: @@ -241,20 +242,27 @@ jobs: fi fi if [ '${{ matrix.build }}' = 'cmake' ]; then - if [[ '${{ matrix.env }}' = 'clang'* ]]; then - options='-DCMAKE_C_COMPILER=clang' - else - options='-DCMAKE_C_COMPILER=gcc' + for _chkprefill in '' ${{ matrix.chkprefill }}; do + if [[ '${{ matrix.env }}' = 'clang'* ]]; then + options='-DCMAKE_C_COMPILER=clang' + else + options='-DCMAKE_C_COMPILER=gcc' + fi + [ '${{ matrix.sys }}' = 'msys' ] && options+=' -D_CURL_PREFILL=ON' + [ '${{ matrix.test }}' = 'uwp' ] && options+=' -DCMAKE_SYSTEM_NAME=WindowsStore -DCMAKE_SYSTEM_VERSION=10.0' + [ '${{ matrix.type }}' = 'Debug' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG=' + [ '${{ matrix.type }}' = 'Release' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE=' + [ "${_chkprefill}" = '_chkprefill' ] && options+=' -D_CURL_PREFILL=OFF' + cmake -B "bld${_chkprefill}" -G Ninja ${options} \ + -DCMAKE_C_FLAGS="${{ matrix.cflags }} ${CFLAGS_CMAKE} ${CPPFLAGS}" \ + -DCMAKE_BUILD_TYPE='${{ matrix.type }}' \ + -DCMAKE_UNITY_BUILD=ON -DCMAKE_UNITY_BUILD_BATCH_SIZE=30 -DCURL_TEST_BUNDLES=ON \ + -DCURL_WERROR=ON \ + ${{ matrix.config }} + done + if [ -d bld_chkprefill ]; then + diff -u bld/lib/curl_config.h bld_chkprefill/lib/curl_config.h fi - [ '${{ matrix.test }}' = 'uwp' ] && options+=' -DCMAKE_SYSTEM_NAME=WindowsStore -DCMAKE_SYSTEM_VERSION=10.0' - [ '${{ matrix.type }}' = 'Debug' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG=' - [ '${{ matrix.type }}' = 'Release' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE=' - cmake -B bld -G Ninja ${options} \ - -DCMAKE_C_FLAGS="${{ matrix.cflags }} ${CFLAGS_CMAKE} ${CPPFLAGS}" \ - -DCMAKE_BUILD_TYPE='${{ matrix.type }}' \ - -DCMAKE_UNITY_BUILD=ON -DCMAKE_UNITY_BUILD_BATCH_SIZE=30 -DCURL_TEST_BUNDLES=ON \ - -DCURL_WERROR=ON \ - ${{ matrix.config }} else export CFLAGS CPPFLAGS mkdir bld && cd bld && ../configure --enable-unity --enable-test-bundles --enable-warnings --enable-werror \ @@ -667,6 +675,7 @@ jobs: plat: 'windows' type: 'Debug' tflags: '~1516' + chkprefill: '_chkprefill' # WARNING: libssh uses hard-coded world-writable paths (/etc/..., ~/.ssh/) to # read its configuration from, making it vulnerable to attacks on # Windows. Do not use this component till there is a fix for these. @@ -718,31 +727,38 @@ jobs: timeout-minutes: 5 run: | [ -d /c/_gfw ] && PATH="/c/_gfw/usr/bin:$PATH" - if [ '${{ matrix.plat }}' = 'uwp' ]; then - options+=' -DCMAKE_SYSTEM_NAME=WindowsStore -DCMAKE_SYSTEM_VERSION=10.0' - cflags='-DWINAPI_FAMILY=WINAPI_FAMILY_PC_APP' - ldflags='-OPT:NOREF -OPT:NOICF -APPCONTAINER:NO' - vsglobals=';AppxPackage=false;WindowsAppContainer=false' + for _chkprefill in '' ${{ matrix.chkprefill }}; do + options='' + if [ '${{ matrix.plat }}' = 'uwp' ]; then + options+=' -DCMAKE_SYSTEM_NAME=WindowsStore -DCMAKE_SYSTEM_VERSION=10.0' + cflags='-DWINAPI_FAMILY=WINAPI_FAMILY_PC_APP' + ldflags='-OPT:NOREF -OPT:NOICF -APPCONTAINER:NO' + vsglobals=';AppxPackage=false;WindowsAppContainer=false' + fi + [ '${{ matrix.arch }}' = 'arm64' ] && options+=' -A ARM64' + [ '${{ matrix.arch }}' = 'x64' ] && options+=' -A x64' + [ '${{ matrix.arch }}' = 'x86' ] && options+=' -A Win32' + [ "${_chkprefill}" = '_chkprefill' ] && options+=' -D_CURL_PREFILL=OFF' + cmake -B "bld${_chkprefill}" ${options} \ + -DCMAKE_TOOLCHAIN_FILE="$VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake" \ + -DVCPKG_INSTALLED_DIR="$VCPKG_INSTALLATION_ROOT/installed" \ + -DVCPKG_TARGET_TRIPLET='${{ matrix.arch }}-${{ matrix.plat }}' \ + -DCMAKE_C_FLAGS="${cflags}" \ + -DCMAKE_EXE_LINKER_FLAGS="-INCREMENTAL:NO ${ldflags}" \ + -DCMAKE_SHARED_LINKER_FLAGS="-INCREMENTAL:NO ${ldflags}" \ + -DCMAKE_VS_GLOBALS="TrackFileAccess=false${vsglobals}" \ + -DCMAKE_BUILD_TYPE='${{ matrix.type }}' \ + -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON \ + -DCURL_WERROR=ON \ + -DBUILD_SHARED_LIBS=OFF \ + -DENABLE_DEBUG=ON \ + -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG= \ + -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE= \ + ${{ matrix.config }} + done + if [ -d bld_chkprefill ]; then + diff -u bld/lib/curl_config.h bld_chkprefill/lib/curl_config.h fi - [ '${{ matrix.arch }}' = 'arm64' ] && options+=' -A ARM64' - [ '${{ matrix.arch }}' = 'x64' ] && options+=' -A x64' - [ '${{ matrix.arch }}' = 'x86' ] && options+=' -A Win32' - cmake -B bld ${options} \ - -DCMAKE_TOOLCHAIN_FILE="$VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake" \ - -DVCPKG_INSTALLED_DIR="$VCPKG_INSTALLATION_ROOT/installed" \ - -DVCPKG_TARGET_TRIPLET='${{ matrix.arch }}-${{ matrix.plat }}' \ - -DCMAKE_C_FLAGS="${cflags}" \ - -DCMAKE_EXE_LINKER_FLAGS="-INCREMENTAL:NO ${ldflags}" \ - -DCMAKE_SHARED_LINKER_FLAGS="-INCREMENTAL:NO ${ldflags}" \ - -DCMAKE_VS_GLOBALS="TrackFileAccess=false${vsglobals}" \ - -DCMAKE_BUILD_TYPE='${{ matrix.type }}' \ - -DCMAKE_UNITY_BUILD=ON -DCURL_TEST_BUNDLES=ON \ - -DCURL_WERROR=ON \ - -DBUILD_SHARED_LIBS=OFF \ - -DENABLE_DEBUG=ON \ - -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG= \ - -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE= \ - ${{ matrix.config }} - name: 'configure log' if: ${{ !cancelled() }} diff --git a/CMake/unix-cache.cmake b/CMake/unix-cache.cmake new file mode 100644 index 000000000000..65bfe6a84b30 --- /dev/null +++ b/CMake/unix-cache.cmake @@ -0,0 +1,313 @@ +#*************************************************************************** +# _ _ ____ _ +# Project ___| | | | _ \| | +# / __| | | | |_) | | +# | (__| |_| | _ <| |___ +# \___|\___/|_| \_\_____| +# +# Copyright (C) Daniel Stenberg, , et al. +# +# This software is licensed as described in the file COPYING, which +# you should have received as part of this distribution. The terms +# are also available at https://curl.se/docs/copyright.html. +# +# You may opt to use, copy, modify, merge, publish, distribute and/or sell +# copies of the Software, and permit persons to whom the Software is +# furnished to do so, under the terms of the COPYING file. +# +# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY +# KIND, either express or implied. +# +# SPDX-License-Identifier: curl +# +########################################################################### +# Based on CI runs for Cygwin/MSYS2, Linux, macOS, FreeBSD, NetBSD, OpenBSD +if(NOT UNIX) + message(FATAL_ERROR "This file should be included on Unix platforms only") +endif() + +set(HAVE_ALARM 1) +if(ANDROID) + set(HAVE_ARC4RANDOM 1) +else() + set(HAVE_ARC4RANDOM 0) +endif() +set(HAVE_ARPA_INET_H 1) +set(HAVE_ATOMIC 1) +set(HAVE_BASENAME 1) +set(HAVE_BOOL_T 1) +if(NOT APPLE) + set(HAVE_CLOCK_GETTIME_MONOTONIC 1) + if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_CLOCK_GETTIME_MONOTONIC_RAW 1) + else() + set(HAVE_CLOCK_GETTIME_MONOTONIC_RAW 0) + endif() +endif() +set(HAVE_CLOSESOCKET 0) +set(HAVE_DECL_FSEEKO 1) +set(HAVE_DIRENT_H 1) +if(APPLE OR + CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "OpenBSD") + set(HAVE_EVENTFD 0) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux" OR + CMAKE_SYSTEM_NAME STREQUAL "FreeBSD" OR + CMAKE_SYSTEM_NAME STREQUAL "NetBSD") + set(HAVE_EVENTFD 1) +endif() +set(HAVE_FCNTL 1) +set(HAVE_FCNTL_H 1) +set(HAVE_FCNTL_O_NONBLOCK 1) +if(CMAKE_SYSTEM_NAME STREQUAL "iOS") + set(HAVE_FILE_OFFSET_BITS 0) +else() + set(HAVE_FILE_OFFSET_BITS 1) +endif() +set(HAVE_FNMATCH 1) +set(HAVE_FREEADDRINFO 1) +set(HAVE_FSEEKO 1) +if(APPLE) + set(HAVE_FSETXATTR 1) + set(HAVE_FSETXATTR_5 0) + set(HAVE_FSETXATTR_6 1) +elseif(CMAKE_SYSTEM_NAME STREQUAL "FreeBSD" OR + CMAKE_SYSTEM_NAME STREQUAL "OpenBSD") + set(HAVE_FSETXATTR 0) + set(HAVE_FSETXATTR_5 0) + set(HAVE_FSETXATTR_6 0) +elseif(CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux" OR + CMAKE_SYSTEM_NAME STREQUAL "NetBSD") + set(HAVE_FSETXATTR 1) + set(HAVE_FSETXATTR_5 1) + set(HAVE_FSETXATTR_6 0) +endif() +set(HAVE_FTRUNCATE 1) +set(HAVE_GETADDRINFO 1) +if(CMAKE_SYSTEM_NAME STREQUAL "OpenBSD") + set(HAVE_GETADDRINFO_THREADSAFE 0) +elseif(CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux" OR + CMAKE_SYSTEM_NAME STREQUAL "FreeBSD" OR + CMAKE_SYSTEM_NAME STREQUAL "NetBSD") + set(HAVE_GETADDRINFO_THREADSAFE 1) +endif() +set(HAVE_GETEUID 1) +if(APPLE OR + CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "NetBSD" OR + CMAKE_SYSTEM_NAME STREQUAL "OpenBSD") + set(HAVE_GETHOSTBYNAME_R 0) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux" OR + CMAKE_SYSTEM_NAME STREQUAL "FreeBSD") + set(HAVE_GETHOSTBYNAME_R 1) +endif() +set(HAVE_GETHOSTBYNAME_R_3 0) +set(HAVE_GETHOSTBYNAME_R_3_REENTRANT 0) +set(HAVE_GETHOSTBYNAME_R_5 0) +set(HAVE_GETHOSTBYNAME_R_5_REENTRANT 0) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_GETHOSTBYNAME_R_6 1) + set(HAVE_GETHOSTBYNAME_R_6_REENTRANT 1) +else() + set(HAVE_GETHOSTBYNAME_R_6 0) + set(HAVE_GETHOSTBYNAME_R_6_REENTRANT 0) +endif() +set(HAVE_GETHOSTNAME 1) +if(NOT ANDROID OR ANDROID_PLATFORM_LEVEL GREATER_EQUAL 24) + set(HAVE_GETIFADDRS 1) +else() + set(HAVE_GETIFADDRS 0) +endif() +if(APPLE OR + CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux" OR + CMAKE_SYSTEM_NAME STREQUAL "FreeBSD" OR + CMAKE_SYSTEM_NAME STREQUAL "OpenBSD") + set(HAVE_GETPASS_R 0) +elseif(CMAKE_SYSTEM_NAME STREQUAL "NetBSD") + set(HAVE_GETPASS_R 1) +endif() +set(HAVE_GETPEERNAME 1) +set(HAVE_GETPPID 1) +set(HAVE_GETPWUID 1) +set(HAVE_GETPWUID_R 1) +set(HAVE_GETRLIMIT 1) +set(HAVE_GETSOCKNAME 1) +set(HAVE_GETTIMEOFDAY 1) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_GLIBC_STRERROR_R 1) +else() + set(HAVE_GLIBC_STRERROR_R 0) +endif() +set(HAVE_GMTIME_R 1) +set(HAVE_IFADDRS_H 1) +set(HAVE_IF_NAMETOINDEX 1) +set(HAVE_INET_NTOP 1) +set(HAVE_INET_PTON 1) +set(HAVE_IOCTLSOCKET 0) +set(HAVE_IOCTLSOCKET_CAMEL 0) +set(HAVE_IOCTLSOCKET_CAMEL_FIONBIO 0) +set(HAVE_IOCTLSOCKET_FIONBIO 0) +set(HAVE_IOCTL_FIONBIO 1) +set(HAVE_IOCTL_SIOCGIFADDR 1) +if(CYGWIN) + set(HAVE_IO_H 1) +else() + set(HAVE_IO_H 0) +endif() +set(HAVE_LIBGEN_H 1) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_LINUX_TCP_H 1) +else() + set(HAVE_LINUX_TCP_H 0) +endif() +set(HAVE_LOCALE_H 1) +set(HAVE_LONGLONG 1) +if(APPLE) + set(HAVE_MACH_ABSOLUTE_TIME 1) +endif() +if(APPLE OR + CYGWIN) + set(HAVE_MEMRCHR 0) +else() + set(HAVE_MEMRCHR 1) +endif() +set(HAVE_MSG_NOSIGNAL 1) +set(HAVE_NETDB_H 1) +if(ANDROID) + set(HAVE_NETINET_IN6_H 1) +else() + set(HAVE_NETINET_IN6_H 0) +endif() +set(HAVE_NETINET_IN_H 1) +set(HAVE_NETINET_TCP_H 1) +set(HAVE_NETINET_UDP_H 1) +set(HAVE_NET_IF_H 1) +set(HAVE_OPENDIR 1) +set(HAVE_PIPE 1) +set(HAVE_POLL 1) +set(HAVE_POLL_H 1) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_POSIX_STRERROR_R 0) +else() + set(HAVE_POSIX_STRERROR_R 1) +endif() +set(HAVE_PWD_H 1) +set(HAVE_REALPATH 1) +set(HAVE_RECV 1) +set(HAVE_SA_FAMILY_T 1) +set(HAVE_SCHED_YIELD 1) +set(HAVE_SELECT 1) +set(HAVE_SEND 1) +if(APPLE OR + CYGWIN) + set(HAVE_SENDMMSG 0) +else() + set(HAVE_SENDMMSG 1) +endif() +set(HAVE_SENDMSG 1) +set(HAVE_SETLOCALE 1) +if(CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_SETMODE 0) +else() + set(HAVE_SETMODE 1) +endif() +set(HAVE_SETRLIMIT 1) +set(HAVE_SETSOCKOPT_SO_NONBLOCK 0) +set(HAVE_SIGACTION 1) +set(HAVE_SIGINTERRUPT 1) +set(HAVE_SIGNAL 1) +set(HAVE_SIGSETJMP 1) +set(HAVE_SNPRINTF 1) +if(CMAKE_SYSTEM_NAME STREQUAL "iOS") + set(HAVE_SOCKADDR_IN6_SIN6_ADDR 0) +else() + set(HAVE_SOCKADDR_IN6_SIN6_ADDR 1) +endif() +set(HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID 1) +set(HAVE_SOCKET 1) +set(HAVE_SOCKETPAIR 1) +set(HAVE_STDATOMIC_H 1) +set(HAVE_STDBOOL_H 1) +if(CMAKE_SYSTEM_NAME STREQUAL "iOS") + set(HAVE_STDDEF_H 0) + set(HAVE_STDINT_H 0) +else() + set(HAVE_STDDEF_H 1) + set(HAVE_STDINT_H 1) +endif() +set(HAVE_STRCASECMP 1) +set(HAVE_STRCMPI 0) +set(HAVE_STRDUP 1) +set(HAVE_STRERROR_R 1) +set(HAVE_STRICMP 0) +set(HAVE_STRINGS_H 1) +if(_CURL_OLD_LINUX) + set(HAVE_STROPTS_H 1) +else() + set(HAVE_STROPTS_H 0) # glibc 2.30 or newer. https://sourceware.org/legacy-ml/libc-alpha/2019-08/msg00029.html +endif() +set(HAVE_STRTOK_R 1) +set(HAVE_STRTOLL 1) +set(HAVE_STRUCT_SOCKADDR_STORAGE 1) +set(HAVE_STRUCT_TIMEVAL 1) +if(ANDROID OR CMAKE_SYSTEM_NAME STREQUAL "iOS") + set(HAVE_SUSECONDS_T 1) +endif() +if(APPLE OR + CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "OpenBSD") + set(HAVE_SYS_EVENTFD_H 0) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux" OR + CMAKE_SYSTEM_NAME STREQUAL "FreeBSD" OR + CMAKE_SYSTEM_NAME STREQUAL "NetBSD") + set(HAVE_SYS_EVENTFD_H 1) +endif() +if(CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_SYS_FILIO_H 0) +else() + set(HAVE_SYS_FILIO_H 1) +endif() +set(HAVE_SYS_IOCTL_H 1) +set(HAVE_SYS_PARAM_H 1) +set(HAVE_SYS_POLL_H 1) +set(HAVE_SYS_RESOURCE_H 1) +set(HAVE_SYS_SELECT_H 1) +set(HAVE_SYS_SOCKET_H 1) +if(CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_SYS_SOCKIO_H 0) +else() + set(HAVE_SYS_SOCKIO_H 1) +endif() +set(HAVE_SYS_STAT_H 1) +set(HAVE_SYS_TIME_H 1) +set(HAVE_SYS_TYPES_H 1) +set(HAVE_SYS_UN_H 1) +if(CYGWIN) + set(HAVE_SYS_UTIME_H 1) +else() + set(HAVE_SYS_UTIME_H 0) +endif() +set(HAVE_TERMIOS_H 1) +if(CYGWIN OR + CMAKE_SYSTEM_NAME STREQUAL "Linux") + set(HAVE_TERMIO_H 1) +else() + set(HAVE_TERMIO_H 0) +endif() +set(HAVE_TIME_T_UNSIGNED 0) +set(HAVE_UNISTD_H 1) +set(HAVE_UTIME 1) +set(HAVE_UTIMES 1) +set(HAVE_UTIME_H 1) +set(HAVE_WRITABLE_ARGV 1) +if(CYGWIN) + set(HAVE__SETMODE 1) +endif() +set(STDC_HEADERS 1) +set(USE_UNIX_SOCKETS 1) diff --git a/CMakeLists.txt b/CMakeLists.txt index d5f92ffffe11..ab0c77d3cef9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -539,16 +539,21 @@ include(CheckSymbolExists) include(CheckTypeSize) include(CheckCSourceCompiles) -option(_CURL_QUICK_DETECT "Fast-track known feature detection results (Windows, some Apple)" ON) -if(_CURL_QUICK_DETECT) +option(_CURL_PREFILL "Fast-track known feature detection results (Windows, some Apple)" "${WIN32}") +if(_CURL_PREFILL) if(WIN32) include("${CMAKE_CURRENT_SOURCE_DIR}/CMake/win32-cache.cmake") - elseif(APPLE) - set(HAVE_EVENTFD 0) - set(HAVE_GETPASS_R 0) - set(HAVE_WRITABLE_ARGV 1) - set(HAVE_SENDMMSG 0) - endif() + elseif(UNIX) + include("${CMAKE_CURRENT_SOURCE_DIR}/CMake/unix-cache.cmake") + message(STATUS "Pre-filling feature detection results for UNIX") + endif() +elseif(WIN32) + message(STATUS "Pre-filling feature detection results disabled.") +elseif(APPLE) + set(HAVE_EVENTFD 0) + set(HAVE_GETPASS_R 0) + set(HAVE_WRITABLE_ARGV 1) + set(HAVE_SENDMMSG 0) endif() if(AMIGA) diff --git a/Makefile.am b/Makefile.am index f69f0736105f..d63ef632cb9a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -60,6 +60,7 @@ CMAKE_DIST = \ CMake/OtherTests.cmake \ CMake/PickyWarnings.cmake \ CMake/Utilities.cmake \ + CMake/unix-cache.cmake \ CMake/win32-cache.cmake \ CMakeLists.txt \ tests/cmake/CMakeLists.txt \ diff --git a/appveyor.sh b/appveyor.sh index 8351e8312240..9b30f2379e08 100644 --- a/appveyor.sh +++ b/appveyor.sh @@ -43,28 +43,36 @@ fi openssl_root="$(cygpath "${openssl_root_win}")" if [ "${BUILD_SYSTEM}" = 'CMake' ]; then - options='' - [[ "${TARGET}" = *'ARM64'* ]] && SKIP_RUN='ARM64 architecture' - [ -n "${TOOLSET:-}" ] && options+=" -T ${TOOLSET}" - [ "${OPENSSL}" = 'ON' ] && options+=" -DOPENSSL_ROOT_DIR=${openssl_root_win}" - [ -n "${CURLDEBUG:-}" ] && options+=" -DENABLE_CURLDEBUG=${CURLDEBUG}" - [ "${PRJ_CFG}" = 'Debug' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG=' - [ "${PRJ_CFG}" = 'Release' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE=' - [[ "${PRJ_GEN}" = *'Visual Studio'* ]] && options+=' -DCMAKE_VS_GLOBALS=TrackFileAccess=false' - # shellcheck disable=SC2086 - cmake -B _bld "-G${PRJ_GEN}" ${TARGET} ${options} \ - -DCURL_USE_OPENSSL="${OPENSSL}" \ - -DCURL_USE_SCHANNEL="${SCHANNEL}" \ - -DHTTP_ONLY="${HTTP_ONLY}" \ - -DBUILD_SHARED_LIBS="${SHARED}" \ - -DCMAKE_UNITY_BUILD="${UNITY}" \ - -DCURL_TEST_BUNDLES=ON \ - -DCURL_WERROR=ON \ - -DENABLE_DEBUG="${DEBUG}" \ - -DENABLE_UNICODE="${ENABLE_UNICODE}" \ - -DCMAKE_INSTALL_PREFIX='C:/curl' \ - -DCMAKE_BUILD_TYPE="${PRJ_CFG}" \ - -DCURL_USE_LIBPSL=OFF + # Set env CHKPREFILL to the value '_chkprefill' to compare feature detection + # results with and without the pre-fill feature. They have to match. + for _chkprefill in '' ${CHKPREFILL:-}; do + options='' + [ "${_chkprefill}" = '_chkprefill' ] && options+=' -D_CURL_PREFILL=OFF' + [[ "${TARGET}" = *'ARM64'* ]] && SKIP_RUN='ARM64 architecture' + [ -n "${TOOLSET:-}" ] && options+=" -T ${TOOLSET}" + [ "${OPENSSL}" = 'ON' ] && options+=" -DOPENSSL_ROOT_DIR=${openssl_root_win}" + [ -n "${CURLDEBUG:-}" ] && options+=" -DENABLE_CURLDEBUG=${CURLDEBUG}" + [ "${PRJ_CFG}" = 'Debug' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG=' + [ "${PRJ_CFG}" = 'Release' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE=' + [[ "${PRJ_GEN}" = *'Visual Studio'* ]] && options+=' -DCMAKE_VS_GLOBALS=TrackFileAccess=false' + # shellcheck disable=SC2086 + cmake -B "_bld${_chkprefill}" "-G${PRJ_GEN}" ${TARGET} \ + -DCURL_USE_OPENSSL="${OPENSSL}" \ + -DCURL_USE_SCHANNEL="${SCHANNEL}" \ + -DHTTP_ONLY="${HTTP_ONLY}" \ + -DBUILD_SHARED_LIBS="${SHARED}" \ + -DCMAKE_UNITY_BUILD="${UNITY}" \ + -DCURL_TEST_BUNDLES=ON \ + -DCURL_WERROR=ON \ + -DENABLE_DEBUG="${DEBUG}" \ + -DENABLE_UNICODE="${ENABLE_UNICODE}" \ + -DCMAKE_INSTALL_PREFIX='C:/curl' \ + -DCMAKE_BUILD_TYPE="${PRJ_CFG}" \ + -DCURL_USE_LIBPSL=OFF ${options} + done + if [ -d _bld_chkprefill ]; then + diff -u _bld/lib/curl_config.h _bld_chkprefill/lib/curl_config.h + fi if false; then cat _bld/CMakeFiles/CMakeConfigureLog.yaml 2>/dev/null || true fi