Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SwiftPM fails to resolve system libraries that are present in the target SDK (--experimental-swift-sdk) #7409

Open
1 task done
t089 opened this issue Mar 19, 2024 · 0 comments
Open
1 task done

SwiftPM fails to resolve system libraries that are present in the target SDK (--experimental-swift-sdk) #7409

t089 opened this issue Mar 19, 2024 · 0 comments
Assignees
@MaxDesiatov @euanh
Labels
bug cross-compilation

Comments

@t089
Copy link

t089 commented Mar 19, 2024

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

  • Confirmed reproduction steps with SwiftPM CLI.

Description

Currently, when cross-compiling packages with system dependencies, SwiftPM will parse .pc files from the SDK but then take those paths as-is without considering the path of the SDK itself.

For example, glib-2.0 on Ubuntu Jammy defines the following .pc file:

prefix=/usr
libdir=${prefix}/lib/x86_64-linux-gnu
includedir=${prefix}/include

bindir=${prefix}/bin
glib_genmarshal=${bindir}/glib-genmarshal
gobject_query=${bindir}/gobject-query
glib_mkenums=${bindir}/glib-mkenums

Name: GLib
Description: C Utility Library
Version: 2.72.4
Requires.private: libpcre >=  8.31
Libs: -L${libdir} -lglib-2.0
Libs.private: -pthread -lm
Cflags: -I${includedir}/glib-2.0 -I${libdir}/glib-2.0/include

Building a SwiftPM package using a Swift SDK that contains this file will result in flags like -I /usr/lib/x86_64-linux-gnu/glib-2.0. This path is not correct, the expected behaviour is to use a path inside the SDK root:

-I $SDK_ROOT/usr/lib/x86_64-linux-gnu/glib-2.0

One possible solution could be to mirror the pkg-config behaviour and respect the PKG_CONFIG_SYSROOT_DIR env variable. When set, this variable should be prepended to all resolved paths from the pc file.

But maybe, by default, the SDK sys root should be prepended when using a Swift SDK.

Expected behavior

Building Swift packages that rely on system libraries present in a Swift SDK should be possible. SwiftPM should pass flags that reference the files inside the SDK.

Actual behavior

SwiftPM passes flags that reference files in the host filesystem and not inside the SDK and thus the build fails or is incorrect.

Steps to reproduce

  1. Clone the example repo: t089/swift-glib-test
  2. Generate a Swift SDK based on ubuntu-jammy with libglib2.0-dev installed (see this issue on Swift SDK Generator: Can't get a custom SDK working with glib-2.0 swift-sdk-generator#89)
  3. Try to build the example package using the generated SDK

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.90.11.1 Apple Swift version 5.10 (swiftlang-5.10.0.13 clang-1500.3.9.4)
Target: arm64-apple-macosx14.0
Darwin tobias-m1-pro.local 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000 arm64
@t089 t089 added the bug label Mar 19, 2024
@MaxDesiatov MaxDesiatov self-assigned this Mar 19, 2024
@euanh euanh mentioned this issue Apr 16, 2024
Merged
@euanh euanh self-assigned this Apr 16, 2024
MaxDesiatov pushed a commit that referenced this issue Apr 18, 2024
@euanh
pkgconfig: Apply PKG_CONFIG_SYSROOTDIR when generating paths (#7461)
1fa6298 
pkgconfig: Apply PKG_CONFIG_SYSROOTDIR when generating paths

### Motivation:

SwiftPM's pkg-config implementation sets the `pc_sysrootdir` variable,
but most `.pc` files do not use this variable directly. Instead, they
rely on
the pkg-config tool rewriting the generated paths to include the sysroot
prefix when necessary. SwiftPM does not do this, so it does not generate
the correct compiler flags to use libraries from a sysroot.


This problem was reported in issue
#7409

### Modifications:

There are two major pkg-config implementations which handle sysroot
differently:

  *  `pkg-config` (the original https://pkg-config.freedesktop.org
     implementation) prepends sysroot after variable expansion,
     when it creates the compiler flag lists

  *  `pkgconf` (the newer http://pkgconf.org implementation) prepends
     sysroot to variables when they are defined, so sysroot is included
     when they are expanded

`pkg-config`'s method skips single character compiler flags, such as
`-I`
and `-L`, and has special cases for longer options. It does not handle
spaces between the flags and their values properly, and prepends sysroot
multiple times in some cases, such as when the .pc file uses the
`sysroot_dir` variable directly or has been rewritten to hard-code the
sysroot prefix.

`pkgconf`'s method handles spaces correctly, although it also makes
extra checks to ensure that sysroot is not applied more than once.

In 2024 `pkg-config` is the more popular option according to Homebrew
installation statistics, but the major Linux distributions have
generally
switched to `pkgconf`.

We will use `pkgconf`'s method here as it seems more robust than
`pkg-config`'s, and `pkgconf`'s greater popularity on Linux means
libraries
developed there may depend on the specific way it handles `.pc` files.

### Result:

SwiftPM will now apply the sysroot prefix to compiler flags, such as 
include (`-I`) and library (`-L`) search paths.

This is a partial fix for
#7409.

The sysroot prefix is only applied when the `PKG_CONFIG_SYSROOT_DIR` 
environment variable is set. A future commit could apply an appropriate
sysroot automatically when the `--experimental-swift-sdk` flag is used.
euanh added a commit to euanh/swift-package-manager that referenced this issue Apr 19, 2024
@euanh
pkgconfig: Apply PKG_CONFIG_SYSROOTDIR when generating paths
493ab68 
SwiftPM's pkg-config implementation sets the `pc_sysrootdir` variable,
but most `.pc` files do not use this variable directly. Instead, they
rely on
the pkg-config tool rewriting the generated paths to include the sysroot
prefix when necessary. SwiftPM does not do this, so it does not generate
the correct compiler flags to use libraries from a sysroot.

This problem was reported in issue
apple#7409

There are two major pkg-config implementations which handle sysroot
differently:

  *  `pkg-config` (the original https://pkg-config.freedesktop.org
     implementation) prepends sysroot after variable expansion,
     when it creates the compiler flag lists

  *  `pkgconf` (the newer http://pkgconf.org implementation) prepends
     sysroot to variables when they are defined, so sysroot is included
     when they are expanded

`pkg-config`'s method skips single character compiler flags, such as
`-I`
and `-L`, and has special cases for longer options. It does not handle
spaces between the flags and their values properly, and prepends sysroot
multiple times in some cases, such as when the .pc file uses the
`sysroot_dir` variable directly or has been rewritten to hard-code the
sysroot prefix.

`pkgconf`'s method handles spaces correctly, although it also makes
extra checks to ensure that sysroot is not applied more than once.

In 2024 `pkg-config` is the more popular option according to Homebrew
installation statistics, but the major Linux distributions have
generally
switched to `pkgconf`.

We will use `pkgconf`'s method here as it seems more robust than
`pkg-config`'s, and `pkgconf`'s greater popularity on Linux means
libraries
developed there may depend on the specific way it handles `.pc` files.

SwiftPM will now apply the sysroot prefix to compiler flags, such as
include (`-I`) and library (`-L`) search paths.

This is a partial fix for
apple#7409.

The sysroot prefix is only applied when the `PKG_CONFIG_SYSROOT_DIR`
environment variable is set. A future commit could apply an appropriate
sysroot automatically when the `--experimental-swift-sdk` flag is used.
@euanh euanh mentioned this issue Apr 19, 2024
Merged
MaxDesiatov pushed a commit that referenced this issue Apr 19, 2024
@euanh
[6.0] pkgconfig: Apply PKG_CONFIG_SYSROOTDIR when generating paths (#…
25ad105 
…7472)

Cherry pick of #7461 

SwiftPM's pkg-config implementation sets the `pc_sysrootdir` variable,
but most `.pc` files do not use this variable directly. Instead, they
rely on
the pkg-config tool rewriting the generated paths to include the sysroot
prefix when necessary. SwiftPM does not do this, so it does not generate
the correct compiler flags to use libraries from a sysroot.

This problem was reported in issue
#7409

There are two major pkg-config implementations which handle sysroot
differently:

* `pkg-config` (the original https://pkg-config.freedesktop.org
implementation) prepends sysroot after variable expansion, when it
creates the compiler flag lists

* `pkgconf` (the newer http://pkgconf.org implementation) prepends
sysroot to variables when they are defined, so sysroot is included when
they are expanded

`pkg-config`'s method skips single character compiler flags, such as
`-I`
and `-L`, and has special cases for longer options. It does not handle
spaces between the flags and their values properly, and prepends sysroot
multiple times in some cases, such as when the .pc file uses the
`sysroot_dir` variable directly or has been rewritten to hard-code the
sysroot prefix.

`pkgconf`'s method handles spaces correctly, although it also makes
extra checks to ensure that sysroot is not applied more than once.

In 2024 `pkg-config` is the more popular option according to Homebrew
installation statistics, but the major Linux distributions have
generally
switched to `pkgconf`.

We will use `pkgconf`'s method here as it seems more robust than
`pkg-config`'s, and `pkgconf`'s greater popularity on Linux means
libraries
developed there may depend on the specific way it handles `.pc` files.

SwiftPM will now apply the sysroot prefix to compiler flags, such as
include (`-I`) and library (`-L`) search paths.

This is a partial fix for
#7409.

The sysroot prefix is only applied when the `PKG_CONFIG_SYSROOT_DIR`
environment variable is set. A future commit could apply an appropriate
sysroot automatically when the `--experimental-swift-sdk` flag is used.

**Scope**: Limited to packages relying on system libraries that utilize
pkg-config.
**Risk**: Low, changes are isolated and the scope is limited to a small
fraction of packages.
**Testing**: Automated with new test cases.
**Issues**: #7409
**Reviewer**: @MaxDesiatov.
euanh added a commit to euanh/swift-package-manager that referenced this issue Apr 26, 2024
@euanh
pkgconfig: Apply PKG_CONFIG_SYSROOTDIR when generating paths
7868d7e 
SwiftPM's pkg-config implementation sets the `pc_sysrootdir` variable,
but most `.pc` files do not use this variable directly. Instead, they
rely on
the pkg-config tool rewriting the generated paths to include the sysroot
prefix when necessary. SwiftPM does not do this, so it does not generate
the correct compiler flags to use libraries from a sysroot.

This problem was reported in issue
apple#7409

There are two major pkg-config implementations which handle sysroot
differently:

  *  `pkg-config` (the original https://pkg-config.freedesktop.org
     implementation) prepends sysroot after variable expansion,
     when it creates the compiler flag lists

  *  `pkgconf` (the newer http://pkgconf.org implementation) prepends
     sysroot to variables when they are defined, so sysroot is included
     when they are expanded

`pkg-config`'s method skips single character compiler flags, such as
`-I`
and `-L`, and has special cases for longer options. It does not handle
spaces between the flags and their values properly, and prepends sysroot
multiple times in some cases, such as when the .pc file uses the
`sysroot_dir` variable directly or has been rewritten to hard-code the
sysroot prefix.

`pkgconf`'s method handles spaces correctly, although it also makes
extra checks to ensure that sysroot is not applied more than once.

In 2024 `pkg-config` is the more popular option according to Homebrew
installation statistics, but the major Linux distributions have
generally
switched to `pkgconf`.

We will use `pkgconf`'s method here as it seems more robust than
`pkg-config`'s, and `pkgconf`'s greater popularity on Linux means
libraries
developed there may depend on the specific way it handles `.pc` files.

SwiftPM will now apply the sysroot prefix to compiler flags, such as
include (`-I`) and library (`-L`) search paths.

This is a partial fix for
apple#7409.

The sysroot prefix is only applied when the `PKG_CONFIG_SYSROOT_DIR`
environment variable is set. A future commit could apply an appropriate
sysroot automatically when the `--experimental-swift-sdk` flag is used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MaxDesiatov MaxDesiatov

@euanh euanh

Labels
bug cross-compilation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MaxDesiatov @euanh @t089