unsymlink-lib with paludis based systems

[hlandgarten]

Gentoo is moving to profiles 17.1 which required lib and lib64 is be real directories and to eliminate the lib symlink to lib64 and lib32. The unsymlink-lib program does not work properly on paludis based Gentoo systems. The issue appears to be in vardb, where paludis appear to have recorded the physical file location instead or the logical file locations according to Gentoo. Any idea of what to do?

[MageSlayer]

I guess the most obvious way to fix it is to rewrite vdb filenames.
That will need to fix Paludis of course.

[negril]

Any progress on this? Any way we could help?

[MageSlayer]

Nothing concrete has been done yet.
Anybody who has some free time might at least investigate the current situation and provide details to ease further movement.

[negril]

All the packages that have files that got installed into lib64 that should be in lib still own /lib or /usr/lib.
The only issue is the wrong recording in the vdb. paludis would install files into /lib & /usr/lib if they were not symlinks.

There are some messy ways to fix this, but I think fixing the vdb record temporarily via a useflag and patch might be the best idea. You can then rebuild everything that owns /lib or /usr/lib, remove the useflag and rebuild again.

[MageSlayer]

Could you be more specific?

According to https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are-now-stable.html /lib /usr/lib become separate directories from /lib64 and /usr/lib64.

Now, to check which directories are actually recorded in /var/db/pkg directories e.g. for sys-libs/glibc-2.29-r2 I look into /var/db/pkg/sys-libs/glibc-2.29-r2/CONTENTS and see following:

dir /lib64
obj /lib64/libBrokenLocale-2.29.so 64bbea1845e530414229a10a86ec571b 1572637589
obj /lib64/libpthread-2.29.so 9531df17a18d07c78d70093d8b240233 1572637595
obj /lib64/libmvec-2.29.so d6addcca627b914236da8a7cd15ac0b4 1572637595
obj /lib64/ld-2.29.so e6a3bb1a71eed41d104ee1e219eab34a 1572637597
obj /lib64/libc-2.29.so d0174568c3b4a65eead5fe73c3fc3be8 1572637597
obj /lib64/libm-2.29.so 26de7cdfc7e28f4fbb5a2e30ffadb353 1572637592
obj /lib64/libdl-2.29.so 7f2b35ecd7b9f09895c9e9b0e268832a 1572637593
obj /lib64/libmemusage.so 52d844f8bc38fab9d2f612bf452a000e 1572637593
sym /lib64/ld-linux-x86-64.so.2 -> ld-2.29.so 1572637598
obj /lib64/librt-2.29.so f0250a28916ce490ec8da1d514a0dfc6 1572637595
...

Same for /var/db/pkg/sys-libs/binutils-libs-2.30-r4/CONTENTS

dir /usr
dir /usr/lib64
obj /usr/lib64/libopcodes-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so c6fabefc7c2152cd9c0acc6b97140df0 1554360828
obj /usr/lib64/libiberty.a e5e778c4b8199010271bafb1dd1aba84 1554360827
sym /usr/lib64/libbfd.so -> libbfd-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so 1554360828
obj /usr/lib64/libbfd.la b25cd30588e38bbf412aa8f46fe26983 1554360827
obj /usr/lib64/libbfd.a e310bb2155e3f2b9e4a654b9350a45d0 1554360827
sym /usr/lib64/libopcodes.so -> libopcodes-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so 1554360828
obj /usr/lib64/libopcodes.la ae287a544b7e984e73cab99892db5acf 1554360827
obj /usr/lib64/libbfd-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so c87aa64a9aa9e06160062f514888318f 1554360828
obj /usr/lib64/libopcodes.a e3db255eabc8bdb97f73598fddfe51f2 1554360827
dir /usr/lib32
obj /usr/lib32/libiberty.a 1f67446ba0742c56115bdb68e056fbcc 1554360826
obj /usr/lib32/libbfd-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so be4a715b7dfd1cf301bab6447ba7734e 1554360828
obj /usr/lib32/libopcodes-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so 92b9c97f7aa825d1a09dcdda0b76722e 1554360828
sym /usr/lib32/libbfd.so -> libbfd-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so 1554360828
obj /usr/lib32/libbfd.la 51f12ad21c503865d4d47705002103db 1554360827
obj /usr/lib32/libbfd.a a01bda3b8dae52d47ce528974310494c 1554360827
sym /usr/lib32/libopcodes.so -> libopcodes-2.30.0.gentoo-sys-libs-binutils-libs-st-def.so 1554360828
obj /usr/lib32/libopcodes.la d2698d384d0c92bb066550b626db50e8 1554360827
obj /usr/lib32/libopcodes.a 082d10a17c9c537819ddbfa9d041744c 1554360827

So Paludis recorded paths actually dereferencing symlinks.
Thus my immediate reaction is that just replacing symlinks with real directories is enough and no reinstallation is necessary.

Am I missing something obvious here?

[negril]

The issue is, that you don't know if the package should have been in lib or lib64, because there are be packages that install in both. E.g a fixed /var/db/pkg/dev-lang/python-3.7.5-r1/CONTENTS would look like this:

dir /usr/lib64
sym /usr/lib64/libpython3.7m.so -> libpython3.7m.so.1.0 1575039160
obj /usr/lib64/libpython3.7m.so.1.0 0ff3fee05b130a6af8d2ea6d381ed491 1575039158
dir /usr/lib64/pkgconfig
obj /usr/lib64/pkgconfig/python-3.7.pc 698a4ea3fb9966fb9fb8e7f236e7a836 1575039138
sym /usr/lib64/pkgconfig/python-3.7m.pc -> python-3.7.pc 1575039161
obj /usr/lib64/libpython3.7m.a 33cccb915ad8bec8edb4f1d7effb4a9d 1575039138
...
dir /usr/lib
dir /usr/lib/python3.7
dir /usr/lib/python3.7/lib-dynload
obj /usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so d809ef12cc21a8f7d2c79146ae8b4b91 1575039158
obj /usr/lib/python3.7/lib-dynload/xxlimited.cpython-37m-x86_64-linux-gnu.so 5f864e565a68de4ef5370b3ff6d1da70 1575039158
obj /usr/lib/python3.7/lib-dynload/_uuid.cpython-37m-x86_64-linux-gnu.so 7143571d32b4b9dfb51e15cda242fda5 1575039158
obj /usr/lib/python3.7/lib-dynload/ossaudiodev.cpython-37m-x86_64-linux-gnu.so 28940e55656776438ff9794a6844d49f 1575039158

Before removing the dereferencing this would have looked like this:

dir /usr/lib64
sym /usr/lib64/libpython3.7m.so -> libpython3.7m.so.1.0 1575039160
obj /usr/lib64/libpython3.7m.so.1.0 0ff3fee05b130a6af8d2ea6d381ed491 1575039158
dir /usr/lib64/pkgconfig
obj /usr/lib64/pkgconfig/python-3.7.pc 698a4ea3fb9966fb9fb8e7f236e7a836 1575039138
sym /usr/lib64/pkgconfig/python-3.7m.pc -> python-3.7.pc 1575039161
obj /usr/lib64/libpython3.7m.a 33cccb915ad8bec8edb4f1d7effb4a9d 1575039138
...
dir /usr/lib
dir /usr/lib64/python3.7
dir /usr/lib64/python3.7/lib-dynload
obj /usr/lib64/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so d809ef12cc21a8f7d2c79146ae8b4b91 1575039158
obj /usr/lib64/python3.7/lib-dynload/xxlimited.cpython-37m-x86_64-linux-gnu.so 5f864e565a68de4ef5370b3ff6d1da70 1575039158
obj /usr/lib64/python3.7/lib-dynload/_uuid.cpython-37m-x86_64-linux-gnu.so 7143571d32b4b9dfb51e15cda242fda5 1575039158
obj /usr/lib64/python3.7/lib-dynload/ossaudiodev.cpython-37m-x86_64-linux-gnu.so 28940e55656776438ff9794a6844d49f 1575039158

How would you decide where it should have been installed? The only hint that something was awry is this line:
dir /usr/lib

I have a patch that I would like to test that fixes that. But that was hastily written and I would like to test it further if it breaks something.

[MageSlayer]

Ha-ha, Python. Again.

Yes, I'll gladly review the patch when it is ready.
Please also include unit test :)

[hlandgarten]

anything new on this issue. 17.0 profiles are deprecated and we need an easy way to make the change to 17.1

[MageSlayer]

Hi.
Nothing has been done yet.

[negril]

I've pushed the patch to negril@f7af7dd. The vdb tests should capture this as well.

I'm not quite sure we should merge it. It seems wiser to add the patch via use-flag.

I am using the following set to reinstall affected packages. There are a few false positives.

#!/bin/bash

declare -a lib_packages
packages=$(qlist -CIv)
dirs=("/lib" "/usr/lib")

for dir in "${dirs[@]}"; do
  for package in $(cave print-owners "${dir}" -f '%c/%p-%v\n'); do
   HASLIB=$(grep -E "^dir ${dir}$" /var/db/pkg/${package}/CONTENTS)
   [[ -z "${HASLIB}" ]] && continue
   HASLIBCONTENT=$(grep -E "^[^ ]* ${dir}\/"  /var/db/pkg/${package}/CONTENTS)
   [[ -z "${HASLIBCONTENT}" ]] && lib_packages+=("${package}")
  done
done

for package in $(printf '%s\n' "${lib_packages[@]}");do
  ATOM=$(echo $package | grep -oP '(.*)(?=-[0-9])')
  SLOT=$(cat /var/db/pkg/${package}/SLOT)
  echo "? ${ATOM}:${SLOT}"
done

[MageSlayer]

Thanks for the patch.

Could you confirm that https://www.gentoo.org/support/news-items/2019-06-05-amd64-17-1-profiles-are-now-stable.html migration plan also suits Paludis?
I mean that utility unsymlink-lib

[negril]

I would assume so. The output from unsymlink-lib --analyze seems fine apart from untracked files (from gcc-config, eselect, ccache etc) and empty directories but that is a due to a design decision in unsymlink-lib.

I will run the full migration once I can take my test machine down.

[negril]

I successfully ran the full migration and the system seems fine. Can't recompile everything since the gentoo tree has to many packages relying on EAPI 8 now.

[MageSlayer]

Thanks for info.
I guess it's time for us to start looking at EAPI 8 specification ;)