AMI: Clean up messy apt handling - #2351
Draft
mmlb wants to merge 4 commits into
Draft
Conversation
Crispy1975
approved these changes
Aug 8, 2026
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
6 times, most recently
from
August 10, 2026 19:59
a722383 to
4eae171
Compare
Collaborator
Author
|
Reverting the PR/commit that caused this and converting this to draft. Will re-introduce everything with fixes in here after bugs are ironed out. |
mmlb
marked this pull request as draft
August 11, 2026 14:38
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
from
August 17, 2026 15:52
237d765 to
5e8c157
Compare
Collaborator
Author
PostgreSQL Package Dependency Analysis: PR #2351
SummaryNo packages had MAJOR version updates. Full Analysis ResultsPostgreSQL 15 Dependency ChangesExtracting PostgreSQL 15 dependencies...
Runtime Closure Size
Raw Dependency ClosurePostgreSQL 17 Dependency ChangesExtracting PostgreSQL 17 dependencies...
Runtime Closure Size
Raw Dependency Closure |
PostgreSQL Extension Dependency Analysis: PR #2351
SummaryNo extensions had dependencies with MAJOR version updates. Full Analysis ResultsPostgreSQL 15 Extension DependenciesPostgreSQL 17 Extension DependenciesOrioleDB 17 Extension Dependencies |
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
14 times, most recently
from
August 19, 2026 13:57
b35a08f to
2a24f46
Compare
Collaborator
Author
|
depends on #2381 |
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
4 times, most recently
from
August 24, 2026 20:43
ad2d779 to
324a5c9
Compare
Collaborator
Author
|
Not sure why but when CI runs on the latest commit arm64 machines fail to boot up into stage2 correctly and ssh is never able to connect. Going to push commits one at a time to find when it breaks. |
Collaborator
Author
|
pg15 amd64 has made it into stage2 in https://github.com/supabase/postgres/actions/runs/32775503892/job/97588403063?pr=2351 already so pushing next commit. |
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
from
August 24, 2026 21:31
767de2a to
0c3fa9f
Compare
Just like the AMIs, good to know here too.
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
from
August 25, 2026 14:53
0c3fa9f to
600f8ce
Compare
It turns out that most of the code deleted here was actually not doing
anything useful. For example, switch_mirror modifies
/etc/apt/sources.list in place but the file doesn't have any mirrors
configured there so it's really a no-op! Here's the contents from an
instance I just fired up[^1]:
ubuntu@ip-172-31-26-227:~$ tail -n+1 /etc/apt/sources.list /etc/apt/sources.list.d/*
==> /etc/apt/sources.list <==
# Ubuntu sources have moved to the /etc/apt/sources.list.d/ubuntu.sources
# file, which uses the deb822 format. Use deb822-formatted .sources files
# to manage package sources in the /etc/apt/sources.list.d/ directory.
# See the sources.list(5) manual page for details.
==> /etc/apt/sources.list.d/ubuntu.sources <==
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
--- 8< ---
Types: deb
URIs: http://us-east-2.ec2.archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Types: deb
URIs: http://us-east-2.ec2.archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
## Ubuntu security updates. Aside from URIs and Suites,
## this should mirror your choices in the previous section.
Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
We get fallback handling by apt itself by adding multiple mirrors in
URIs, apt tries first (the regional) and falls back to global ubuntu
repos if there's an issue. We also setup a temporary apt config that
forces the APT_OPTIONS for all apt calls so we don't need to do it in
every call site.
I ended up dropping the in-repo sources file since they are strictly
worse than what we get from AWS in the build. There's no change in
suites or components between old and new, just that we get ubuntu
upstream as a fallback. Well technically there's a slight difference
since we are basing off of cloud-init generated files and they can
theoretically change under us but I'll gamble that it'll be fine or
better off. Besides, one day we'll be on NixOS as the ultimate "make
sure we know everything in the instance" ;).
I also got rid of the `add-apt-repository --yes universe` because
universe is already enabled.
[^1]: AMI=ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-20260604
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
2 times, most recently
from
August 25, 2026 19:36
7257b4b to
dd89896
Compare
We are wasting a bunch of time and brain power keeping track of package installs, so lets minimize them while keeping the grouping/messages intact. I did move the package installs from setup_apparmor && setup_grub into update_install_packages because why not just install everything in one place and leave the _setup_ to somewhere else. I moved them out of because package installation should all be done early and ASAP IMO so it can fail fast if its going to fail. Also I moved ec2-hibinit-agent, ec2-instance-connect, hibagent here since this is only run for AMIs and the comment isn't currently accurate. Either through `export DEBIAN_FRONTEND=noninteractive`, `disable_services` or maybe it was treated as a bug and is now fixed in the package I'm not sure but I ran this out of AWS and it was fine.
mmlb
force-pushed
the
manny/psql-1450-minimize-apt-get-updatesupgrades-1-at-start-1-at-end
branch
from
August 25, 2026 19:54
dd89896 to
dbb8d87
Compare
|
Found 1 test failure on Blacksmith runners: Failure
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light.png)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Maintenance
What is the current behavior?
So much confusing code here when apt can do fallbacks better.
Non nix'd scripts aren't shellcheck clean.
Bunch of dead/unused code hanging around.
We run apt-get update/upgrade all over the ami/qemu image build stages.
What is the new behavior?
We now let apt handle fallbacks itself.
All the scripts use apt with the same settings.
I deleted unused files from scripts/ and moved those that were used into ebssurrogate/scripts (may be worth renaming to packer or images).
I enabled shellcheck in git-hooks and excluded currently failing files.