Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

various: tidy up S3 URLs #175197

Merged
merged 45 commits into from
Jun 19, 2024
Merged

various: tidy up S3 URLs #175197

merged 45 commits into from
Jun 19, 2024

Conversation

vszakats
Copy link
Contributor

@vszakats vszakats commented May 29, 2024

Important: Do not tick a checkbox if you haven’t performed its action. Honesty is indispensable for a smooth review process.

In the following questions <cask> is the token of the cask you're submitting.

After making any changes to a cask, existing or new, verify:

Additionally, if adding a new cask:

  • Named the cask according to the token reference.
  • Checked the cask was not already refused (add your cask's name to the end of the search field).
  • Checked the cask is submitted to the correct repo.
  • brew audit --cask --new <cask> worked successfully.
  • HOMEBREW_NO_INSTALL_FROM_API=1 brew install --cask <cask> worked successfully.
  • brew uninstall --cask <cask> worked successfully.

Prefer:

  • <prefix>.s3.amazonaws.com (modern) URL flavour over s3.amazonaws.com/<prefix>.
    (this is always supported by S3 for dotless, lowercase prefixes.)
  • region-agnostic URLs unless they do a permanent redirect to region-specific.
  • s3-<region>.amazonaws.com over s3.<region>.amazonaws.com.

All hand checked.

@vszakats
Copy link
Contributor Author

vszakats commented May 29, 2024

All failures seem unrelated to these updates:

  • unmodified homepage unreachable: arctype
    project discontinued: https://news.ycombinator.com/item?id=33114111
  • packages needing version update: datadog-agent, ripx, rstudio@daily
    of this, ripx advertises a new 7.1.1 version, but it's not available at the pre-existing location.
    Website requires a working email address to maybe reveal a download URL.
  • livecheck check telling version doesn't match ''?: data-integration
  • broken check for a working livecheck URL: izotope-product-portal
    (the URL works from here from a browser, checker says it's a 403)

@bevanjkay
Copy link
Member

bevanjkay commented May 29, 2024

I'm not sure how much it matters either way, but I would suggest that most of these instances were consistent with what is broadcast by the vendor, whether it be linked from their homepage, appcast or other medium.

I don't feel strongly whether we should stick to a standard, or mirror what is used upstream.

But if we are going to use a standard then this should be added to either brew audit or brew style rather than requiring manual fixes.

@SMillerDev
Copy link
Member

I think requiring changes here will complicate contributions. I can't imagine it'll be easy if you want to make a cask, copy an upstream download link and brew style tells you it's wrong.

@vszakats
Copy link
Contributor Author

The prefixed URLs are the ones recommened by Amazon (for a long time now). I've been doing similar updates for a few years way back when. These URLs also play better with corporate (or non-corporate) firewalls. Region-agnostic URLs most likely play better (faster) when downloading from a different region, and one of the points of S3.

I'm not sure how this complicates controbutions. There is no enforcement for this AFAIK.

@vszakats
Copy link
Contributor Author

vszakats commented May 29, 2024

I don't think it's realistic to expect me to push through an brew audit update for this. I haven't done it for a long time and don't have the guts or time to touch it. Furthermore region-agnostic URLs almost always need manual checks, because S3 can be configured to play bad with them, and in a few rare cases, they do.

I haven't touched S3 URLs in the last 5 years and 50-ish fallouts that accumulated throughout this time isn't that much to justify a brew audit effort IMO.

@vszakats
Copy link
Contributor Author

vszakats commented May 29, 2024

data-integration now fails because the homepage is apparently temporarly down.

version bumped datadog-agent, rstudio@daily.

will delete commits for the 2 broken casks.

@vszakats
Copy link
Contributor Author

The remaining issues are still unrelated to this PR, and they also seem to be detection issues:

  • data-integration: homepage is temporarly down, it was still up on the first CI run.
  • izotope-product-portal: livecheck URL is actually working, and the returned version is 1.4.8, which is already offered by the cask.

@vszakats vszakats force-pushed the s3-urls-1 branch 2 times, most recently from 634a397 to 73c83f4 Compare June 12, 2024 15:37
@vszakats
Copy link
Contributor Author

drop for pre-existing issues:
anythingllm
data-integration
itubedownloader
izotope-product-portal
polypad

Casks/m/movist-pro.rb Outdated Show resolved Hide resolved
@vszakats
Copy link
Contributor Author

vszakats commented Jun 15, 2024

Sigh, failure because one cask received a version update, detected by livecheck.

@bevanjkay bevanjkay added the awaiting maintainer feedback Issue needs response from a maintainer. label Jun 17, 2024
@krehel
Copy link
Member

krehel commented Jun 17, 2024

If no one has objections to this, I'm fine for ✅, but do want to solve some automated way of identifying these going forward.

@vszakats
Copy link
Contributor Author

vszakats commented Jun 17, 2024

A rough sketch for the multi-step URL transformation:

  1. s3-accelerate.s3.: strip "accelerate" (very rare)
  2. s3.([a-z0-9-]+).amazonaws.coms3-\1.amazonaws.com: prefer single-level
  3. s3(-[a-z0-9-]+)?.amazonaws.com/([a-z][a-z0-9_-]*)/\2.s3\1.amazonaws.com/: prefer prefix-style
  4. .s3(-[a-z0-9-]+).amazonaws.com/.s3.amazonaws.com/: go region-agnostic — if returning 200. Or 301, which works if following the redirect, but I didn't go for it.

Double-check -gov- regions. (elasticwolf uses it in this repo, and both download and homepage seem to be gone now.)

@razvanazamfirei
Copy link
Member

/rebase

@razvanazamfirei razvanazamfirei changed the title tidy up S3 URLs various: tidy up S3 URLs Jun 19, 2024
@razvanazamfirei
Copy link
Member

This looks great! 👍 Thanks so much for your contribution! Without contributions like yours, it'd be impossible to keep our project going.

@bevanjkay
Copy link
Member

/rebase

@razvanazamfirei razvanazamfirei merged commit b423485 into Homebrew:master Jun 19, 2024
96 checks passed
@vszakats vszakats deleted the s3-urls-1 branch June 19, 2024 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge-skip awaiting maintainer feedback Issue needs response from a maintainer.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants