Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makefile.in: build config.h before any object #192

Merged
merged 2 commits into from
Jul 12, 2024
Merged

Makefile.in: build config.h before any object #192

merged 2 commits into from
Jul 12, 2024

Conversation

trofi
Copy link
Contributor

@trofi trofi commented Nov 6, 2022

Object files lack dependency on config.h symlink (and usually depend only on ldns/config.h). As a result build occasionally fails in make --shuffe mode:

$ make --shuffle
...
./libtool --tag=CC --quiet --mode=compile gcc -I. -I.  \
  -DHAVE_CONFIG_H -DLDNS_TRUST_ANCHOR_FILE="\"...-dns-root-data-2019-01-11/root.key\"" \
  -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes \
  -Wwrite-strings -W -Wall -g -O2 -I...-openssl-3.0.7-dev/include -c examples/ldns-rrsig.c \
  -o examples/ldns-rrsig.o
examples/ldns-rrsig.c:10:10: fatal error: config.h: No such file or directory
   10 | #include "config.h"
      |          ^~~~~~~~~~

The change moves config.h symlink creation in setup-builddir from a dependency on final binary to dependency on all individual objects.

While at it regenerated dependencies with make depend.

before the change make --shuffle build was failing in about 50% cases.
After the change: make --shuffle succeeded 80 times in a row.

Object files lack dependency on `config.h` symlink (and usually depend
only on `ldns/config.h`). As a result build occasionally fails in
`make --shuffe mode`:

    $ make --shuffle
    ...
    ./libtool --tag=CC --quiet --mode=compile gcc -I. -I.  \
      -DHAVE_CONFIG_H -DLDNS_TRUST_ANCHOR_FILE="\"...-dns-root-data-2019-01-11/root.key\"" \
      -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes \
      -Wwrite-strings -W -Wall -g -O2 -I...-openssl-3.0.7-dev/include -c examples/ldns-rrsig.c \
      -o examples/ldns-rrsig.o
    examples/ldns-rrsig.c:10:10: fatal error: config.h: No such file or directory
       10 | #include "config.h"
          |          ^~~~~~~~~~

The change moves `config.h` symlink creation in `setup-builddir` from
a dependency on final binary to dependency on all individual objects.

While at it regenerated dependencies with `make depend`.

before the change `make --shuffle` build was failing in about 50% cases.
After the change: `make --shuffle` succeeded 80 times in a row.
Copy link
Member

@wtoorop wtoorop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that looks fine. And I learned something new: "make --shuffle"! Thanks!

@wtoorop wtoorop merged commit d7767a6 into NLnetLabs:develop Jul 12, 2024
4 of 8 checks passed
wtoorop added a commit that referenced this pull request Jul 12, 2024
This reverts commit d7767a6, reversing
changes made to 8d76771.
@trofi trofi deleted the fix-make-depends branch July 12, 2024 14:22
@trofi
Copy link
Contributor Author

trofi commented Jul 12, 2024

Oh, I just noticed you have reverted the change. I still see the build failures on the current masterdevelop:

ldns> ./examples/ldns-keyfetcher.c:10:10: fatal error: config.h: No such file or directory
ldns>    10 | #include "config.h"
ldns>       |          ^~~~~~~~~~
ldns> compilation terminated.
ldns> examples/ldns-keyfetcher.c:10:10: fatal error: config.h: No such file or directory
ldns>    10 | #include "config.h"
ldns>       |          ^~~~~~~~~~
ldns> compilation terminated.

@wtoorop
Copy link
Member

wtoorop commented Jul 14, 2024

Yes, I did reverse it and I was a bit in a hurry, so failed to tell you about it.
The problem basically was that setup-builddir doesn't have a target file to check if it is done. So if you make all object files depend on it, all object files will be built every time you run make. Even if they are already built.
If we can come up with a workaround, we can still include it for the release coming Friday.

@trofi
Copy link
Contributor Author

trofi commented Jul 19, 2024

Aha, the broken incremental rebuild makes sense. I remember it being the problem. But I had an impression that it's not a new behaviour and incremental rebuild is already happening on unmodified ldns. On current develop:

$ time { make | grep libtool | wc -l; }
51

real    0m7,475s
user    0m5,704s
sys     0m1,163s

$ time { make | grep libtool | wc -l; }
51

real    0m8,242s
user    0m5,957s
sys     0m1,279s

@wtoorop
Copy link
Member

wtoorop commented Jul 20, 2024

Wow, you have an impressively fast machine, but are you sure you did git pull? With me the second time the object files are not rebuild:

$ time { make | grep libtool | wc -l; }
96

real	0m28,250s
user	0m20,519s
sys	0m8,480s
$ time { make | grep libtool | wc -l; }
0

real	0m0,025s
user	0m0,010s
sys	0m0,019s

Anyway, I will do the release now, but don't worry, we'll put it in the next release if we can find a way to make make --shuffle work. And a next release doesn't need to take another 2 years.

@wtoorop
Copy link
Member

wtoorop commented Jul 20, 2024

Also, what options to configure did you use to get the 51 libtool invokes?

@trofi
Copy link
Contributor Author

trofi commented Jul 20, 2024

Oh, I'm using $ ./configure --disable-static --with-ssl=...-openssl-3.0.14-dev --with-trust-anchor=.../root.key --with-drill --disable-gost --with-examples. Looks like all rebuilds are related to examples.

Here is my full session:

git update
$ git clean -dfx

$ git pull --recurse
Fetching submodule contrib/DNS-LDNS
Fetching submodule test/tpkg
Already up to date.

$ git status
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean

$ git describe
1.8.4
$ libtoolize -ci && autoreconf -fi
$ libtoolize -ci && autoreconf -fi
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './config.guess'
libtoolize: copying file './config.sub'
libtoolize: copying file './install-sh'
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am.
configure.ac:789: warning: The preprocessor macro `STDC_HEADERS' is obsolete.
configure.ac:789:   Except in unusual embedded environments, you can safely include all
configure.ac:789:   ISO C90 headers unconditionally.
./configure
$ ./configure --disable-static --with-ssl=/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev --with-trust-anchor=/nix/store/g6zfrqxp8cybjrmrb8myk6lir5miv89q-dns-root-data-2023-11-27/root.key --with-drill --disable-gost --with-examples
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for wchar.h... yes
checking for minix/config.h... no
checking for vfork.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether _XOPEN_SOURCE should be defined... no
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /nix/store/cdhx572ij0pvn9ayd33lk8118137j6m8-gnused-4.9/bin/sed
checking for grep that handles long lines and -e... /nix/store/mdiqq9b2rvv7fajlmg3f7d6r6g78l547-gnugrep-3.11/bin/grep
checking for egrep... /nix/store/mdiqq9b2rvv7fajlmg3f7d6r6g78l547-gnugrep-3.11/bin/grep -E
checking for fgrep... /nix/store/mdiqq9b2rvv7fajlmg3f7d6r6g78l547-gnugrep-3.11/bin/grep -F
checking for ld used by gcc... ld
checking if the linker (ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... nm
checking the name lister (nm) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for ld option to reload object files... -r
checking for file... file
checking for objdump... objdump
checking how to recognize dependent libraries... (cached) pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking for gawk... gawk
checking command to parse nm output from gcc object... ok
checking for sysroot... no
checking for a working dd... /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/dd
checking how to truncate binary pipes... /nix/store/dhv5gh89him9a7ddr56cqg87zfkmjihp-coreutils-9.5/bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no
checking for gcc... (cached) gcc
checking whether the compiler supports GNU C... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to enable C11 features... (cached) none needed
checking gcc dependency flag... -MM
checking whether make sets $(MAKE)... yes
checking whether gcc supports -std=c99... yes
checking whether gcc supports -xc99... no
checking for an ANSI C-conforming const... yes
checking whether gcc supports -Wall... yes
checking whether gcc supports -W... yes
checking whether gcc supports -Wwrite-strings... yes
checking whether gcc supports -Wstrict-prototypes... yes
checking whether gcc supports -Wunused-function... yes
checking whether gcc supports -Wmissing-prototypes... no
checking whether gcc supports -fno-strict-aliasing... yes
checking for getopt.h... yes
checking for time.h... yes
checking for winsock2.h... no
checking for ws2tcpip.h... no
checking for TargetConditionals.h... no
checking whether gcc supports -Werror... yes
checking whether gcc supports -Wall... (cached) yes
checking whether gcc supports -std=c99... (cached) yes
checking whether gcc supports -xc99... (cached) no
checking for getopt.h... (cached) yes
checking for time.h... (cached) yes
checking whether we need -std=c99 -D__EXTENSIONS__ -D_BSD_SOURCE -D_DEFAULT_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -D_XOPEN_SOURCE_EXTENDED=1 -D_ALL_SOURCE as a flag for gcc... failed
checking whether we need -std=c99 -D__EXTENSIONS__ -D_BSD_SOURCE -D_DEFAULT_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -D_ALL_SOURCE as a flag for gcc... failed
checking whether we need -std=c99 as a flag for gcc... no
checking whether we need -D_BSD_SOURCE -D_DEFAULT_SOURCE as a flag for gcc... no
checking whether we need -D_GNU_SOURCE as a flag for gcc... yes
checking whether we need -D_GNU_SOURCE -D_FRSRESGID as a flag for gcc... no
checking whether we need -D_POSIX_C_SOURCE=200112 as a flag for gcc... no
checking whether we need -D__EXTENSIONS__ as a flag for gcc... failed
checking for inline... inline
checking for int8_t... yes
checking for int16_t... yes
checking for int32_t... yes
checking for int64_t... yes
checking for uint8_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint64_t... yes
checking for doxygen... no
checking for socket... yes
checking for inet_ntop... yes
checking for inet_pton... yes
checking for poll(2)... yes
checking for ar... /nix/store/kp2j7yn0wzwq5piy494r54dafrh83s6s-gcc-wrapper-13.3.0/bin/ar
checking for SSL... found in /nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev
checking for EVP_sha256 in -lcrypto... yes
checking for openssl/ssl.h... yes
checking for openssl/err.h... yes
checking for openssl/rand.h... yes
checking for LibreSSL... no
checking for openssl/ssl.h... (cached) yes
checking for openssl/evp.h... yes
checking for openssl/engine.h... yes
checking for openssl/conf.h... yes
checking for EVP_sha256... yes
checking for EVP_sha384... yes
checking for EVP_sha512... yes
checking for EVP_PKEY_keygen... yes
checking for ECDSA_SIG_get0... yes
checking for EVP_MD_CTX_new... yes
checking for DSA_SIG_set0... yes
checking for DSA_SIG_get0... yes
checking for EVP_dss1... no
checking for DSA_get0_pqg... yes
checking for DSA_get0_key... yes
checking for EVP_cleanup... no
checking for ENGINE_cleanup... no
checking for ENGINE_free... yes
checking for CRYPTO_cleanup_all_ex_data... no
checking for ERR_free_strings... no
checking for CONF_modules_unload... yes
checking for OPENSSL_init_ssl... no
checking for OPENSSL_init_crypto... yes
checking for ERR_load_crypto_strings... no
checking for CRYPTO_memcmp... yes
checking for EVP_PKEY_get_base_id... yes
checking for gcc options needed to detect all undeclared functions... none needed
checking whether EVP_PKEY_base_id is declared... yes
checking Checking for OpenSSL >= 3.0.0... yes
checking for SHA256 and SHA512... checking for SHA256_Init... yes
checking for ECDSA_sign... yes
checking for SHA384_Init... yes
checking whether NID_X9_62_prime256v1 is declared... yes
checking whether NID_secp384r1 is declared... yes
checking for DSA_SIG_new... yes
checking whether NID_ED25519 is declared... yes
checking whether NID_ED448 is declared... yes
checking for X509_check_ca... yes
checking for SSL_get0_dane... yes
checking whether byte ordering is bigendian... no
checking for egrep... (cached) /nix/store/mdiqq9b2rvv7fajlmg3f7d6r6g78l547-gnugrep-3.11/bin/grep -E
checking for _Bool... yes
checking for stdbool.h that conforms to C99 or later... yes
checking for getopt.h... (cached) yes
checking for stdarg.h... yes
checking for openssl/ssl.h... (cached) yes
checking for netinet/in.h... yes
checking for time.h... (cached) yes
checking for arpa/inet.h... yes
checking for netdb.h... yes
checking for sys/param.h... yes
checking for sys/mount.h... yes
checking for sys/socket.h... yes
checking for inttypes.h... (cached) yes
checking for sys/types.h... (cached) yes
checking for unistd.h... (cached) yes
checking size of time_t... 8
checking for pcap.h... no
checking for pcap_open_offline in -lpcap... no
configure: WARNING: Can't find pcap library (needed for ldns-dpa, will not build dpa now.)
checking for netinet/in_systm.h... yes
checking for net/if.h... yes
checking for netinet/ip.h... yes
checking for netinet/udp.h... yes
checking for netinet/igmp.h... yes
checking for netinet/if_ether.h... yes
checking for netinet/ip6.h... yes
checking for net/ethernet.h... yes
checking for netinet/ip_compat.h... no
checking for socklen_t... yes
checking for size_t... yes
checking for ssize_t... yes
checking for intptr_t... yes
checking for in_addr_t... yes
checking for in_port_t... yes
checking for struct sockaddr_storage.ss_family... yes
checking whether inet_pton is declared... yes
checking whether inet_ntop is declared... yes
checking for b64_pton... no
checking for b64_ntop... no
checking for calloc... yes
checking for timegm... yes
checking for gmtime_r... yes
checking for asctime_r... yes
checking for ctime_r... yes
checking for localtime_r... yes
checking for isblank... yes
checking for isascii... yes
checking for inet_aton... yes
checking for inet_pton... (cached) yes
checking for inet_ntop... (cached) yes
checking for snprintf... yes
checking for strlcpy... yes
checking for memmove... yes
checking for pid_t... yes
checking for fork... yes
checking for vfork... yes
checking for working fork... yes
checking for working vfork... (cached) yes
checking compile of fork... yes
checking for endprotoent... yes
checking for endservent... yes
checking for sleep... yes
checking for random... yes
checking for fcntl... yes
checking for strtoul... yes
checking for bzero... yes
checking for memset... yes
checking for b32_ntop... no
checking for b32_pton... no
checking for symlink... yes
checking for getaddrinfo... yes
checking for ioctlsocket... no
checking whether the C compiler (gcc) accepts the "format" attribute... yes
checking whether the C compiler (gcc) accepts the "unused" attribute... yes
configure: Default trust anchor: /nix/store/g6zfrqxp8cybjrmrb8myk6lir5miv89q-dns-root-data-2023-11-27/root.key
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libdns.doxygen
config.status: creating ldns/common.h
config.status: creating ldns/net.h
config.status: creating ldns/util.h
config.status: creating packaging/libldns.pc
config.status: creating packaging/ldns-config
config.status: creating ldns/config.h
config.status: executing libtool commands
copying header files
first make
$ make
<lots of output>
second make (with logs)
$ make
make
./libtool --tag=CC --quiet --mode=compile gcc -I. -I.  -DHAVE_CONFIG_H -DLDNS_TRUST_ANCHOR_FILE="\"/nix/store/g6zfrqxp8cybjrmrb8myk6lir5miv89q-dns-root-data-2023-11-27/root.key\"" -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE -I/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/include -c examples/ldns-chaos.c -o examples/ldns-chaos.o
./libtool --tag=CC --quiet --mode=link gcc -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE  -L/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/lib examples/ldns-chaos.lo compat/b64_pton.lo compat/b64_ntop.lo libldns.la -lssl  -lcrypto  -o examples/ldns-chaos ./libldns.la
./libtool --tag=CC --quiet --mode=compile gcc -I. -I.  -DHAVE_CONFIG_H -DLDNS_TRUST_ANCHOR_FILE="\"/nix/store/g6zfrqxp8cybjrmrb8myk6lir5miv89q-dns-root-data-2023-11-27/root.key\"" -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE -I/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/include -c examples/ldns-compare-zones.c -o examples/ldns-compare-zones.o
./libtool --tag=CC --quiet --mode=link gcc -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE  -L/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/lib examples/ldns-compare-zones.lo compat/b64_pton.lo compat/b64_ntop.lo libldns.la -lssl  -lcrypto  -o examples/ldns-compare-zones ./libldns.la
./libtool --tag=CC --quiet --mode=compile gcc -I. -I.  -DHAVE_CONFIG_H -DLDNS_TRUST_ANCHOR_FILE="\"/nix/store/g6zfrqxp8cybjrmrb8myk6lir5miv89q-dns-root-data-2023-11-27/root.key\"" -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE -I/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/include -c examples/ldnsd.c -o examples/ldnsd.o
./libtool --tag=CC --quiet --mode=link gcc -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE  -L/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/lib examples/ldnsd.lo compat/b64_pton.lo compat/b64_ntop.lo libldns.la -lssl  -lcrypto  -o examples/ldnsd ./libldns.la
...
./libtool --tag=CC --quiet --mode=compile gcc -I. -I.  -DHAVE_CONFIG_H -DLDNS_TRUST_ANCHOR_FILE="\"/nix/store/g6zfrqxp8cybjrmrb8myk6lir5miv89q-dns-root-data-2023-11-27/root.key\"" -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE -I/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/include -c examples/ldns-verify-zone.c -o examples/ldns-verify-zone.o
./libtool --tag=CC --quiet --mode=link gcc -DOPENSSL_API_COMPAT=10100 -fno-strict-aliasing -Wunused-function -Wstrict-prototypes -Wwrite-strings -W -Wall -g -O2 -D_GNU_SOURCE  -L/nix/store/dp6821c0s4bbmmws3lc6nicz09lnjlgl-openssl-3.0.14-dev/lib examples/ldns-verify-zone.lo compat/b64_pton.lo compat/b64_ntop.lo libldns.la -lssl  -lcrypto  -o examples/ldns-verify-zone ./libldns.la

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants