PDL::GIS::Proj can't load, Makefile has strange LD_RUN_PATH

[d-lamb]

I after installing Alien::proj from CPAN, PDL tries to build PDL::GIS::Proj, but fails make test and even a simple module load:

$ perl -Mblib -MPDL::GIS::Proj -e'1;'
Can't load '/Users/derek/Build/PDL/blib/arch/auto/PDL/GIS/Proj/Proj.bundle' for module PDL::GIS::Proj: dlopen(/Users/derek/Build/PDL/blib/arch/auto/PDL/GIS/Proj/Proj.bundle, 0x0001): Library not loaded: @rpath/libproj.25.dylib
  Referenced from: /Users/derek/Build/PDL/blib/arch/auto/PDL/GIS/Proj/Proj.bundle
  Reason: tried: '/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), '/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), '/usr/local/lib/libproj.25.dylib' (no such file), '/usr/lib/libproj.25.dylib' (no such file) at /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/DynaLoader.pm line 197.
 at -e line 0.
Compilation failed in require.
BEGIN failed--compilation aborted.

It is trying to find libproj.25.dylib in a directory with a colon in it:
/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib.
The correct location is:
/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib.

blib/arch/auto/PDL/GIS/Proj/Proj.bundle has that same string in it.
Libtmp/GIS/Proj/Makefile has the line

LD_RUN_PATH = /opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib

Looking at the Makefile.PL, I don't see anything suspicious. The two values pulled from Alien::proj are correct:

$ perl -MAlien::proj -E 'say Alien::proj->cflags; say Alien::proj->libs;'
-I/opt/local/include -I/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/include -I/opt/local/include 
-L/opt/local/lib -lsqlite3 -L/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib -lproj 

All the other Makefiles in the PDL tree have either empty or sensible LD_RUN_PATH values, such as /usr/local/lib or /opt/local/lib.
I'm using Perl 5.28.3, EUMM 7.64.

[mohawk2]

As a workaround, would it be workable to try instead with the Homebrew PROJ?

[mohawk2]

Also this might be related: Perl-Toolchain-Gang/ExtUtils-MakeMaker#402

[shawnlaffan]

LD_RUN_PATH should be a colon separated list of paths, should it not? If so then that line is correct and something else is treating it as a single string. Is it being quoted somewhere along the line?

[mohawk2]

That was my understanding too. @d-lamb How sure are you that the :-separated LD_RUN_PATH you see there is actually wrong?

[zmughal]

I don't currently have the tuits to look into this, but I mentioned in IRC that I have some notes on linking/rpath here and @mohawk2 suggested I post them: https://orbital-transfer.github.io/doc/development/os/macos/.

[shawnlaffan]

That was my understanding too. @d-lamb How sure are you that the :-separated LD_RUN_PATH you see there is actually wrong?

This part of the error looks like it is using it as a single path. (Broken into several lines for clarity)

Reason: tried: 
'/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), 
'/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), 
'/usr/local/lib/libproj.25.dylib' (no such file), 
'/usr/lib/libproj.25.dylib' (no such file) 
at /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/DynaLoader.pm line 197.

However, there are two lines for the first so maybe it is just not breaking up the array when it gives feedback.

If so then it could be a more generic issue with linking, which comes back to shawnlaffan/perl-alien-proj#7 .

@d-lamb - What do you see when you run otool on the file?

otool -L /Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib

[d-lamb]

That was my understanding too. @d-lamb How sure are you that the :-separated LD_RUN_PATH you see there is actually wrong?

This part of the error looks like it is using it as a single path. (Broken into several lines for clarity)

Reason: tried: 
'/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), 
'/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), 
'/usr/local/lib/libproj.25.dylib' (no such file), 
'/usr/lib/libproj.25.dylib' (no such file) 
at /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/DynaLoader.pm line 197.

That was my thinking, too. I'm not sure that colon-separated LD_RUN_PATH is wrong (it's just different than anything else I see in PDL--but so too is the proj installation!). But looking for a file with a colon in the name seemed wrong, and LD_RUN_PATH is to where I (accurately or not??) traced it back.

@d-lamb - What do you see when you run otool on the file?

otool -L /Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib

I see this:

$ otool -L /Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib         
/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib:
	@rpath/libproj.25.dylib (compatibility version 25.0.0, current version 25.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 331.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

[shawnlaffan]

That all looks OK to my untrained eye. I suspect @mohawk2 or @zmughal can shed more light.

The other thing to check is which version of Alien::proj you are using. The build system for shared builds was changed to CMake in 1.17 and this might have knock-on effects.

There have also been changes in proj related to rpaths in version 9.0.

What do you get for otool -l libproj.22.2.1.dylib? (Mainly in sections around text like cmd LC_RPATH).

[mohawk2]

Also, could we see output of perl -V, and if that's not in there, also perl -V:path_sep?

[mohawk2]

Also, could we see /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/DynaLoader.pm line 197 (+/- 3-ish lines of context) so I can see which bit of https://github.com/Perl/perl5/blob/blead/ext/DynaLoader/DynaLoader_pm.PL is likely to be relevant there?

[mohawk2]

I've added to latest git master some diagnostic output of maybe-relevant env-vars for GIS::Proj test before it tries to load its XS library. Derek, might be worth trying that and see if it sheds any light on your env-vars as relevant here?

[d-lamb]

OK, here's all the output everybody asked for, so far!

Both Alien::proj and Proj are the latest version, since I didn't have either installed before this week:

perl -MAlien::proj -E 'say Alien::proj->VERSION();'
1.20

I don't see cmd LC_RPATH in the output of otool -l

otool -l <snip>/auto/share/dist/Alien-proj/lib/libproj.25.dylib |grep -i 'cmd lc'
      cmd LC_SEGMENT_64
      cmd LC_SEGMENT_64
      cmd LC_SEGMENT_64
      cmd LC_SEGMENT_64
          cmd LC_ID_DYLIB
      cmd LC_DYLD_CHAINED_FIXUPS
      cmd LC_DYLD_EXPORTS_TRIE
     cmd LC_SYMTAB
            cmd LC_DYSYMTAB
     cmd LC_UUID
      cmd LC_BUILD_VERSION
      cmd LC_SOURCE_VERSION
          cmd LC_LOAD_DYLIB
          cmd LC_LOAD_DYLIB
          cmd LC_LOAD_DYLIB
      cmd LC_FUNCTION_STARTS
      cmd LC_DATA_IN_CODE

The only mention of rpath in that output is

Load command 4
          cmd LC_ID_DYLIB
      cmdsize 48
         name @rpath/libproj.25.dylib (offset 24)
   time stamp 1 Wed Dec 31 17:00:01 1969
      current version 25.9.0
compatibility version 25.0.0

perl -V                                                                                                                           
Summary of my perl5 (revision 5 version 28 subversion 3) configuration:
   
  Platform:
    osname=darwin
    osvers=21.3.0
    archname=darwin-thread-multi-2level
    uname='darwin simurgh 21.3.0 darwin kernel version 21.3.0: wed jan 5 21:37:58 pst 2022; root:xnu-8019.80.24~20release_x86_64 x86_64 '
    config_args='-des -Dprefix=/opt/local -Dscriptdir=/opt/local/bin -Dvendorprefix=/opt/local -Dusemultiplicity=y -Dusethreads -Duseshrplib -Dcc=/usr/bin/cc -Dman1ext=1pm -Dman3ext=3pm -Dinstallstyle=lib/perl5 -Dman1dir=/opt/local/share/man/man1p -Dman3dir=/opt/local/share/man/man3p -Dsitebin=/opt/local/libexec/perl5.28/sitebin -Dsiteman1dir=/opt/local/share/perl5.28/siteman/man1 -Dsiteman3dir=/opt/local/share/perl5.28/siteman/man3 -Dvendorbin=/opt/local/libexec/perl5.28 -Dvendorman1dir=/opt/local/share/perl5.28/man/man1 -Dvendorman3dir=/opt/local/share/perl5.28/man/man3 -Dpager=/usr/bin/less -sR -Dperlpath=/opt/local/bin/perl5.28 -Dstartperl=#!/opt/local/bin/perl5.28 -Acppflags=-I/opt/local/include  -Accflags=-pipe -Os   -Alddlflags=-L/opt/local/lib -Wl,-headerpad_max_install_names   -Aldflags=-L/opt/local/lib -Wl,-headerpad_max_install_names  '
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='/usr/bin/cc'
    ccflags ='-fno-common -DPERL_DARWIN  -pipe -Os   -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3'
    cppflags='-I/opt/local/include  -fno-common -DPERL_DARWIN  -pipe -Os   -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include'
    ccversion=''
    gccversion='Apple LLVM 13.0.0 (clang-1300.0.29.30)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='/usr/bin/cc'
    ldflags ='  -L/opt/local/lib -Wl,-headerpad_max_install_names   -fstack-protector-strong'
    libpth=/opt/local/lib /Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib /Library/Developer/CommandLineTools/usr/lib /usr/lib
    libs=-lgdbm
    perllibs=
    libc=
    so=dylib
    useshrplib=true
    libperl=libperl.dylib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='  -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names   -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
  Built under darwin
  Compiled at Mar  7 2022 13:19:10
  %ENV:
    PERL5LIB="/Users/derek/pdl/SWAMIS-EF:/Users/derek/local/lib/perl5"
    PERLBREW_HOME="/Users/derek/.perlbrew"
    PERLBREW_ROOT="/Users/derek/perl5/perlbrew"
    PERLBREW_SHELLRC_VERSION="0.84"
  @INC:
    /Users/derek/pdl/SWAMIS-EF
    /Users/derek/local/lib/perl5/darwin-thread-multi-2level
    /Users/derek/local/lib/perl5
    /opt/local/lib/perl5/site_perl/5.28/darwin-thread-multi-2level
    /opt/local/lib/perl5/site_perl/5.28
    /opt/local/lib/perl5/vendor_perl/5.28/darwin-thread-multi-2level
    /opt/local/lib/perl5/vendor_perl/5.28
    /opt/local/lib/perl5/5.28/darwin-thread-multi-2level
    /opt/local/lib/perl5/5.28
derek@simurgh:PDL$ perl -V|grep path
    config_args='-des -Dprefix=/opt/local -Dscriptdir=/opt/local/bin -Dvendorprefix=/opt/local -Dusemultiplicity=y -Dusethreads -Duseshrplib -Dcc=/usr/bin/cc -Dman1ext=1pm -Dman3ext=3pm -Dinstallstyle=lib/perl5 -Dman1dir=/opt/local/share/man/man1p -Dman3dir=/opt/local/share/man/man3p -Dsitebin=/opt/local/libexec/perl5.28/sitebin -Dsiteman1dir=/opt/local/share/perl5.28/siteman/man1 -Dsiteman3dir=/opt/local/share/perl5.28/siteman/man3 -Dvendorbin=/opt/local/libexec/perl5.28 -Dvendorman1dir=/opt/local/share/perl5.28/man/man1 -Dvendorman3dir=/opt/local/share/perl5.28/man/man3 -Dpager=/usr/bin/less -sR -Dperlpath=/opt/local/bin/perl5.28 -Dstartperl=#!/opt/local/bin/perl5.28 -Acppflags=-I/opt/local/include  -Accflags=-pipe -Os   -Alddlflags=-L/opt/local/lib -Wl,-headerpad_max_install_names   -Aldflags=-L/opt/local/lib -Wl,-headerpad_max_install_names  '

derek@simurgh:PDL$ perl -V:path_sep
path_sep=':';

$ head -n 207 /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/DynaLoader.pm | tail -n 20
    # Many dynamic extension loading problems will appear to come from
    # this section of code: XYZ failed at line 123 of DynaLoader.pm.
    # Often these errors are actually occurring in the initialisation
    # C code of the extension XS file. Perl reports the error as being
    # in this perl code simply because this was the last perl code
    # it executed.

    my $flags = $module->dl_load_flags;
###LINE 197 is next    
    my $libref = dl_load_file($file, $flags) or
	croak("Can't load '$file' for module $module: ".dl_error());

    push(@dl_librefs,$libref);  # record loaded object

    $boot_symbol_ref = dl_find_symbol($libref, $bootname) or
         croak("Can't find '$bootname' symbol in $file\n");

    push(@dl_modules, $module); # record loaded module

  boot:

Compiling and testing latest git master, with the env var output, gives this:

t/gis_proj.t .. # ENV LD_LIBRARY_PATH = undef
# ENV DYLD_LIBRARY_PATH = undef
# ENV LDFLAGS = undef
# ENV CFLAGS = undef
# ENV CXXFLAGS = undef
# ENV LD_RUN_PATH = undef
# ENV LC_RUN_PATH = undef
Can't load '/Users/derek/Build/PDL/Libtmp/GIS/Proj/../../../blib/arch/auto/PDL/GIS/Proj/Proj.bundle' for module PDL::GIS::Proj: dlopen(/Users/derek/Build/PDL/Libtmp/GIS/Proj/../../../blib/arch/auto/PDL/GIS/Proj/Proj.bundle, 0x0002): Library not loaded: @rpath/libproj.25.dylib
  Referenced from: /Users/derek/Build/PDL/blib/arch/auto/PDL/GIS/Proj/Proj.bundle
  Reason: tried: '/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), '/opt/local/lib:/Users/derek/local/lib/perl5/darwin-thread-multi-2level/auto/share/dist/Alien-proj/lib/libproj.25.dylib' (no such file), '/usr/local/lib/libproj.25.dylib' (no such file), '/usr/lib/libproj.25.dylib' (no such file) at /opt/local/lib/perl5/5.28/darwin-thread-multi-2level/DynaLoader.pm line 197.
 at t/gis_proj.t line 18.

[kiwiroy]

From what I see the : separated paths are in a single LC_RPATH entry in the blib/arch/auto/PDL/GIS/Proj/Proj.bundle.

$ otool -l blib/arch/auto/PDL/GIS/Proj/Proj.bundle | grep RPATH -A2
          cmd LC_RPATH
      cmdsize 112
         path /opt/local/lib:/Users/hrards/alien/local/lib/perl5/darwin-2level/auto/share/dist/Alien-proj/lib (offset 12)

These should be separate entries, achievable using install_name_tool

$ install_name_tool -delete_rpath /opt/local/lib:/Users/hrards/alien/local/lib/perl5/darwin-2level/auto/share/dist/Alien-proj/lib \
    blib/arch/auto/PDL/GIS/Proj/Proj.bundle
$ install_name_tool -add_rpath /Users/hrards/alien/local/lib/perl5/darwin-2level/auto/share/dist/Alien-proj/lib \
    blib/arch/auto/PDL/GIS/Proj/Proj.bundle
$ install_name_tool -add_rpath /opt/local/lib \
    blib/arch/auto/PDL/GIS/Proj/Proj.bundle

Resulting in

$ otool -l blib/arch/auto/PDL/GIS/Proj/Proj.bundle | grep RPATH -A2
          cmd LC_RPATH
      cmdsize 96
         path /Users/hrards/alien/local/lib/perl5/darwin-2level/auto/share/dist/Alien-proj/lib (offset 12)
--
          cmd LC_RPATH
      cmdsize 32
         path /opt/local/lib (offset 12)

From here touch Libtmp/Transform/Proj4/Proj4.pd && make && make test is successful.