Several upstream bottles cause segfault. related to old patchelf.rb bugs.
#163826
Comments
|
This issue seems related: #136432 |
|
I think the root cause related to many homebrew Linux segmentation faults is the homebrew build bot. The build bot making the bottles is still somehow using an older |
|
Latest patchelf updates in brew: Most issues have been fixed with version 1.4.0 in Nov. 2022. Why do you think it's still using an old patchelf.rb version? In your config:
This looks old. Can you update first? |
It's not an issue with See above the part with
How do I do that? I ran |
|
|
Don't worry about the 3 months old update message: you are now using the API to get the latest version, so I guess no need to update the git repo on your computer to get the latest version. I think that message is confusing, I'll have a look into it later. I think I know what happened.
We will have to rebuild these bottles. I have triggered a Once this is done, you can |
|
Sounds great. There are several other bottles with crashes, such as golang. |
|
👋 Been quietly watching this and previous issues. In WSL Ubuntu this is pretty wide spread from packages built with go and those that are built with gcc. I believe I saw less issues when I tested in a Ubuntu VM. What could be an effective way forward for the project maintainers? I imagine as the underlying packages upgrade, they'll be rebuilt, so as users, should we just wait? Would the repo owners find a list of affected packages be helpful? I'd be happy to send along the packages I've had issues with and help UAT rebuilds. |
|
@redspot a new
Yes, if you have a list, we can at least rebuild all of these. Most packages get updates / rebuilds at least once per year, but some others will take time to get update. Let's fix what is known to be broken. |
|
Here's a list from my bash/zsh history, though I think I had an issue with every package that uses
|
|
see this comment for more potentially broken bottles #132852 (comment) I definitely had issues with these formulas:
but, there could have been others. It's not that I would like you to fix a specific formula. It's that you most likely have many broken bottles that need a rebuild. I guess you could come up with a way to find bottles that are:
|
|
Can just someone confirm that Note: we probably won't rebottle everything, I'll go through the lists above, for anything else users will have to open an issue or post here. |
|
Sure, I'll test out the new bottle on Monday. I could always solve the problem building from source. This ticket was just to give the project and other users a heads up. |
|
@iMichka I installed Output$ brew install ssdeep
...
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/manifests/2.14.1-1
#################################################################################################################################################################################################################################### 100.0%
==> Fetching ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/blobs/sha256:d62ce2005fd901e4a363b3f6592dee0388e9114d323a058499d16e8dba9c4d38
#################################################################################################################################################################################################################################### 100.0%
==> Pouring ssdeep--2.14.1.x86_64_linux.bottle.1.tar.gz
� /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 215.4KB
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -h
ssdeep version 2.14.1 by Jesse Kornblum and the ssdeep Project
For copyright information, see man page or README.TXT.
Usage: ssdeep [-m file] [-k file] [-dpgvrsblcxa] [-t val] [-h|-V] [FILES]
-m - Match FILES against known hashes in file
-k - Match signatures in FILES against signatures in file
-d - Directory mode, compare all files in a directory
-p - Pretty matching mode. Similar to -d but includes all matches
-g - Cluster matches together
-v - Verbose mode. Displays filename as its being processed
-r - Recursive mode
-s - Silent mode; all errors are suppressed
-b - Uses only the bare name of files; all path information omitted
-l - Uses relative paths for filenames
-c - Prints output in CSV format
-x - Compare FILES as signature files
-a - Display all matches, regardless of score
-t - Only displays matches above the given threshold
-h - Display this help message
-V - Display version number and exit
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1 |
Yes, it works from the bottle. |
|
I've launched a couple of rebottles for formulas that were listed above and where the rebottling won't erase any old bottle (for previous macOS versions). In those cases, there is no downside to rebottling. Then, we can confirm whether the issues are all fixed, and decide how to proceed next. I've launched: |
|
❌ @fxcoudert bottle request for gh failed. |
|
❌ @fxcoudert bottle request for k9s failed. |
|
@fxcoudert I've tried a few of these ( $ k6 --version # previously built from source
k6 v0.49.0 (go1.21.6, linux/amd64)
$ brew update && brew reinstall k6
Already up-to-date.
==> Downloading https://ghcr.io/v2/homebrew/core/k6/manifests/0.49.0
####################################################################################################################################################################################################################### 100.0%
==> Fetching k6
==> Downloading https://ghcr.io/v2/homebrew/core/k6/blobs/sha256:2e548198d61b1f5336646ff43bf7f2cd75ed69c288920c5d47d18f3219c40e65
####################################################################################################################################################################################################################### 100.0%
==> Reinstalling k6
==> Pouring k6--0.49.0.x86_64_linux.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
/home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
� /home/linuxbrew/.linuxbrew/Cellar/k6/0.49.0: 8 files, 37.6MB
==> Running `brew cleanup k6`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ k6 --version
zsh: segmentation fault k6 --version
$ brew update && brew reinstall -s k6
Already up-to-date.
Warning: building from source is not supported!
You're on your own. Failures are expected so don't create any issues, please!
==> Fetching k6
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/726ba7780161a49c1d6b4b202f854e13fb5dca66/Formula/k/k6.rb
####################################################################################################################################################################################################################### 100.0%
==> Downloading https://github.com/grafana/k6/archive/refs/tags/v0.49.0.tar.gz
Already downloaded: /home/pdelre/.cache/Homebrew/downloads/cb0f5adcdfa30d830e9f759aeb1266ce066200d6e60a225a1a5fb2bf7d8ca74c--k6-0.49.0.tar.gz
==> Reinstalling k6
==> go build -ldflags=-s -w
==> Caveats
zsh completions have been installed to:
/home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
� /home/linuxbrew/.linuxbrew/Cellar/k6/0.49.0: 8 files, 38MB, built in 1 minute 8 seconds
==> Running `brew cleanup k6`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ k6 --version
k6 v0.49.0 (go1.22.0, linux/amd64) |
|
I don't know enough about how go executables are built & linked to be able to answer that. go is being rebuilt as part of this PR #165116 so we'll know soon. |
|
@pdelre perhaps you could |
|
@redspot Yes, that's how I resolve the builds locally. |
|
the root cause is with the CGO_ENABLED for go build, we should turn it all for all affected formulae. see https://github.com/grafana/k6/blob/master/release%20notes/v0.26.0.md?plain=1#L127 (for k6 for example) |
|
❌ @fxcoudert bottle request for go-task failed. |
|
The problem is (or was) specific to the RHEL kernel which is why no maintainer was able to reproduce the issue. Ubuntu does work fine, as evidenced by it being the platform we use for CI. The issue also cannot be reproduced on our systems under Docker, even on a RHEL/CentOS image. This bug isn't new - it's been around for years. The I'll look into this but it might take a few days to get back to you as I need to set up a special dev environment just to test this.
Is this WSL1 or WSL2? Note that we do not support WSL1 because of bugs on WSL's side that won't be fixed as it's not updated much anymore.
This disables glibc entirely FWIW. Upstreams often use it for Alpine support but we don't support Alpine. |
that is true, I wonder if we should just disable for the linux build to ease the installation for the linux world. |
WSL2 Ubuntu. I am able to reproduce under my default Ubuntu distro and a freshly created one. I was able able to reproduce in a Ubuntu docker image (running WSL2 docker distro).
I'm outside my area of expertise here but if it was |
Yeah Docker just uses the host kernel so if it's under WSL2 then that makes sense. I'll try WSL2 as that's an easier setup for me and will make this significantly easier to fix if I can reproduce the issue there. |
|
I can't reproduce the issue under this setup: > wsl.exe --version
WSL version: 2.1.4.0
Kernel version: 5.15.146.1-2
WSLg version: 1.0.60
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3155$ brew config
HOMEBREW_VERSION: 4.2.11
ORIGIN: https://github.com/Homebrew/brew
HEAD: d7d4c8266210c024b93a450a7d357cec0b46a1bb
Last commit: 2 days ago
Core tap JSON: 05 Mar 23:13 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 3.1.4 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: dodeca-core 64-bit unknown_0x6_0xa5
Clang: N/A
Git: 2.34.1 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 5.15.146.1-microsoft-standard-WSL2 x86_64 GNU/Linux
OS: Ubuntu 22.04.4 LTS (jammy)
WSL: 2 (Microsoft Store)
Host glibc: 2.35
/usr/bin/gcc: 11.4.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A$ act --version
act version 0.2.60
$ gh version
gh version 2.45.0 (2024-03-04)
https://github.com/cli/cli/releases/tag/v2.45.0
$ curl -o test.zip -sL https://github.com/Homebrew/install/archive/refs/heads/master.zip
$ unzip test.zip
Archive: test.zip
aceed88a4a062e2b41dc40a7428c71309fce14c9
creating: install-master/
creating: install-master/.github/
inflating: install-master/.github/ISSUE_TEMPLATE.md
creating: install-master/.github/ISSUE_TEMPLATE/
inflating: install-master/.github/ISSUE_TEMPLATE/bug.md
inflating: install-master/.github/ISSUE_TEMPLATE/config.yml
inflating: install-master/.github/dependabot.yml
creating: install-master/.github/workflows/
inflating: install-master/.github/workflows/tests.yml
inflating: install-master/.github/workflows/triage-issues.yml
inflating: install-master/LICENSE.txt
inflating: install-master/README.md
inflating: install-master/install.sh
inflating: install-master/uninstall.sh
$ pkg-config --cflags --libs bzip2
-I/home/linuxbrew/.linuxbrew/opt/bzip2/include -L/home/linuxbrew/.linuxbrew/opt/bzip2/lib -lbz2
$ k6 --version
k6 v0.49.0 (go1.22.0, linux/amd64) |
|
Hmm...there is a major kernel (5.x vs 4.x) difference depending on how WSL2 is installed (more). I will eventually try and upgrade to the latest kernel but want to first check if the maintainers would like me to test any fixes on the 4.x kernel as IDK if it's still generally available. |
|
That'll indeed be the key difference then. Things work as expected on a modern kernel (we have CI on 6.5 currently, but we did test 5.15 before fine), so I guess we have some compatibility issues with 4.x that I can try look into. RHEL historically has probably one of the slowest lifecycles (6 years between RHEL 7 and 8), which is probably why it's always been seen as a "RHEL-specific" issue as most other distros switched to 5.x years ago. WSL did too in October 2020 (well prior to our move to a newer glibc - we were still using Ubuntu 16.04 at this point), but it seems you're stuck on a June/August(?) 2020 version. I'm guessing RHEL 9 has similarly improved things. I'll try set up a VM with an old kernel and see what I can do. |
|
I found these additional packages that segfault:
My work requires RHEL. |
if there was an update it would be posted here, that's how it works.
Take one failure, for example the |
|
Glad that I could help. By the way, I used the |
|
I can repro this issue on latest patchelf.rb (1.5.0) on my aarch64 centos 7.2 setup. brew install --build-bottle xz # This will build from source, and no patchelf will be called
./bin/xz # works
brew bottle xz # this will call the patchelf
./bin/xz # segmentation faultI manually patched def save_using_patchelf_rb(new_interpreter, new_rpath)
patcher = patchelf_patcher
# patcher.interpreter = new_interpreter if new_interpreter.present?
patch_interpreter_cmd = "patchelf --set-interpreter #{new_interpreter} #{patcher.in_file}"
puts patch_interpreter_cmd if new_interpreter.present?
system patch_interpreter_cmd if new_interpreter.present?
# patcher.rpath = new_rpath if new_rpath.present?
patch_rpath_cmd = "patchelf --set-rpath #{new_rpath} #{patcher.in_file}"
puts patch_rpath_cmd if new_rpath.present?
system patch_rpath_cmd if new_rpath.present?
# patcher.save(patchelf_compatible: true)
endThe reinstall xz and bottle it, the xz still works. |
|
Hi, can you try the following patch: diff --git a/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb b/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb
index 7937d14e6e..d7caf39e54 100644
--- a/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb
+++ b/Library/Homebrew/vendor/bundle/ruby/3.1.0/gems/patchelf-1.5.0/lib/patchelf/alt_saver.rb
@@ -561,11 +561,12 @@ module PatchELF
if needed_space > start_offset
needed_space += seg_num_bytes # new load segment is required
- needed_pages = Helper.alignup(needed_space - start_offset, page_size) / page_size
+ extra_bytes = needed_space - start_offset
+ needed_pages = Helper.alignup(extra_bytes, page_size) / page_size
Logger.debug "needed pages is #{needed_pages}"
raise PatchError, 'virtual address space underrun' if needed_pages * page_size > first_page
- shift_file(needed_pages, start_offset)
+ shift_file(needed_pages, start_offset, extra_bytes)
first_page -= needed_pages * page_size
start_offset += needed_pages * page_size
@@ -776,7 +777,7 @@ module PatchELF
end
# rubocop:enable Metrics/PerceivedComplexity
- def shift_file(extra_pages, start_offset)
+ def shift_file(extra_pages, start_offset, extra_bytes)
raise PatchError, "start_offset(#{start_offset}) < ehdr.num_bytes" if start_offset < ehdr.num_bytes
oldsz = @buffer.size
@@ -799,8 +800,8 @@ module PatchELF
p_offset: split_phdr.p_offset - split_shift - shift,
p_vaddr: split_phdr.p_vaddr - split_shift - shift,
p_paddr: split_phdr.p_paddr - split_shift - shift,
- p_filesz: split_shift + shift,
- p_memsz: split_shift + shift,
+ p_filesz: split_shift + extra_bytes,
+ p_memsz: split_shift + extra_bytes,
p_flags: ELFTools::Constants::PF_R | ELFTools::Constants::PF_W,
p_align: page_size
)It seems to fix things for me on a x86_64 CentOS 8 machine. |
|
@Bo98 That patch works on my WSL2 Ubuntu, Kernel 4. |
This works on my aarch64, CentOS 7, linux 4.19.91. |
|
@Bo98 Ok, I'll give that patch a try. First, I'll make sure everything is up-to-date: So, it seems like the patch is already applied upstream. Next, I'll try the steps suggested by @Shaloc : Let's check No segfault. Now, we try bottling: Also no segfault. And, finally let's check the virtual address maps from the ELF header: No segments match virtual address and, here's some kernel info: |
|
@Bo98 Thank you for your patch solution as it's continued to work through multiple upgrades (including It appears an upstream contribution must be made to https://github.com/david942j/patchelf.rb and that you were the author of the NixOS |
|
Yes, I will be getting things moving this weekend. There was a patch for a |
|
the |
|
I just migrated a Centos 8 Stream to Rocky Linux, and encountered the While debugging I started with the But I'm dropping this here in case it can help the debugging and/or reproducing efforts. docker run \ # ...or nerdctl
-ti \
--rm \
--platform amd64 \
-v $HOME/.docker/certs.d:/etc/pki/ca-trust/source/anchors \ # My env requires a trust CA, remove at will
rockylinux:8 bash -c '
update-ca-trust # My env requires a trust CA, remove at will
dnf -y install git procps sudo
dnf -y groupinstall "Development Tools"
echo "brewtest ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/brewtest
useradd -m brewtest
sudo su - brewtest -c /bin/bash -c "NONINTERACTIVE=1 $(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
(echo; echo "eval \"$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)\"") >> /home/brewtest/.bashrc
sudo su - brewtest -c /bin/bash -c "brew install gcc"
sudo su - brewtest -c /bin/bash -c "brew install gh"
sudo su - brewtest -c /bin/bash -c "gh"
'Now, in my server env the |
and others
brew gist-logs <formula>link ORbrew configANDbrew doctoroutputVerification
brew doctoroutput" saysYour system is ready to brew.and am still able to reproduce my issue.brew updateand am still able to reproduce my issue.brew doctorand that did not fix my problem.What were you trying to do (and why)?
I was attempting to use several formulae installed via
brew.There are several related issues:
#137991 related to old
patchelf.rbbug#132852 bottles crashing on Centos
#116841 bottles crashing on RHEL
What happened (include all command output)?
Note the segment that causes the problem,
0x0000000000401000Now, take a look at the on-disk segments:
Note the segment
0x00000000004012aeAlso, of interest, when run using
brew's interpreter, programs seem to work:Here is the results when built from source:
What did you expect to happen?
I expect the upstream pre-built bottles to not have lingering issues with
patchelf.rb.Now, the issues with
patchelf.rbhave been fixed, and maybe those Centos/RHEL related issues as well.It's that the bottles still have the problem.
Step-by-step reproduction instructions (by running
brewcommands)As mentioned in this comment: #132852 (comment)
Some formulas work:
Some formulas fail:
Note the kernel reporting the same segment,
0x0000000000401000, and readelf showing that segment. And, it strangely works when run withbrew's interpreter.when building from source:
It also seems reliable to detect the bad packages by looking for a segment that matches the regex
0x0000000000401....The text was updated successfully, but these errors were encountered: