Thunderbird recipe not pulling down latest version for past few months

[fseesink]

I started this over in another issue that had to do with code signature failures which seemed to start around Thunderbird v60 (#258)

as I thought that was the sole issue. But after following the instructions there for turning off the code signature verification in a recipe override, it became clear that this is, in fact, another issue, so am opening a new ticket here.

I've noticed for the past few months that the Thunderbird recipe
AutoPkg/RecipeRepos/com.github.autopkg.recipes/Mozilla/Thunderbird.munki.recipe

is no longer downloading the latest version of Thunderbird available. After turning off code signature verification, while it did download a newer version than what I had previously (v52.9.1), it only downloaded v60.0.

I notice that if you look in the file MozillaURLProvider.py, you see the URL pattern used is

MOZ_BASE_URL = "https://download.mozilla.org/?product=%s-%s&os=osx&lang=%s"

And if you fill that in and put that in your browser of choice, e.g.,

https://download.mozilla.org/?product=thunderbird-latest&os=osx&lang=en-US

what you download is, in fact, as of when I type this on Tuesday, 30 Oct 2018,

Thunderbird 60.0.dmg

And if you download that file, mount it, and look at the binary, you see it is v60.0 (which came out on 6 Aug 2018). Now that's odd, considering the latest version is v60.2.1 (released 2 Oct 2018), and if you visit this page:

https://www.thunderbird.net/en-US/thunderbird/all/

and then manually choose to download the macOS English (US) file, the link you get is

https://download.mozilla.org/?product=thunderbird-60.2.1-SSL&os=osx&lang=en-US

I don't recall the "-SSL" bit the last time I went under the hood to look into this, but that's been awhile, so maybe this isn't anything new. But it caught my eye.

Anyway, right now turning off the code signature verification does not, in fact, "work around the issue" as indicated in the other issue.

Later I dug into this further. I've always had a bookmark to
https://www.mozilla.org/en-US/thunderbird/

but I recently noticed it redirects to
https://www.thunderbird.net/en-US/ (the last part fits as I'm in the US)

and if I Google, the latter is the URL that comes up. Not sure when that happened. But wasn't looking, so maybe it's been that way a long time.

Is it possible the URL structure used in the Python script MozillaURLProvider.py is no longer being maintained at Mozilla.org, as in if you go to
https://download.mozilla.org/?product=thunderbird-latest&os=osx&lang=en-US
that thunderbird-latest, which I'm guessing is just an alias, isn't being updated to point at the latest? And if so, is this because someone isn't paying attention, or they consciously chose to no longer do that? And how would we find out, or who would we contact?

There appears to be a bit of a disconnect as now the official Thunderbird site is www.thunderbird.net, but once you click a download link on that site, you're still downloading from download.mozilla.org. So I'm wondering if those who used to maintain the site when it was on www.mozilla.org are no longer doing so (or can't?) even though they link to download files on that site.

I did check, and
https://download.mozilla.org/?product=firefox-latest&os=osx&lang=en-US
does take you to the latest version of Firefox. So this appears to be Thunderbird specific, so not applicable to all Mozilla projects.

For what it's worth, there's a bug ticket over on TB's BugZilla about the download URL not working:

https://bugzilla.mozilla.org/show_bug.cgi?id=1497574

Unfortunately, so far, the Thunderbird folks don't seem to see this as an issue, though if you dig down through that to the links referenced and email threads, clearly others have noticed this issue.

Note in the BugZilla ticket above that they refer to it as a resolved duplicate of bug
https://bugzilla.mozilla.org/show_bug.cgi?id=1481198

and in THAT ticket there is an entry which reads, in part,

FYI, the source of truth for Thunderbird versions(used by all thunderbird.net websites as well) is https://product-details.mozilla.org/1.0/thunderbird_versions.json -- this is automatically updated when a new version is marked shipped by release engineering.

We may yet keep the bouncer aliases though, that has certainly not been decided or this bug would be closed. :)

• Link: https://bugzilla.mozilla.org/show_bug.cgi?id=1481198#c12

And one of the last posts in the first BugZilla ticket I mention above talks about the same: how they have a URL that provides you with JSON data showing the latest versions of Thunderbird here:
https://product-details.mozilla.org/1.0/thunderbird_versions.json
Output:

{"LATEST_THUNDERBIRD_ALPHA_VERSION": "54.0a2", "LATEST_THUNDERBIRD_VERSION": "60.2.1", "LATEST_THUNDERBIRD_DEVEL_VERSION": "63.0b1", "LATEST_THUNDERBIRD_NIGHTLY_VERSION": "64.0a1"}

and that from that you can parse out the LATEST_THUNDERBIRD_VERSION and then construct a proper URL to get the latest version:
https://download.mozilla.org/?product=thunderbird-60.2.1-SSL&os=osx&lang=en-US

Would it make sense to adjust the MozillaURLProvider.py script to make use of this?
I played around a bit and found they do, in fact, have a similar URL with JSON output for Firefox:
https://product-details.mozilla.org/1.0/firefox_versions.json

Anyway, just thought I'd throw this out there.

[gregneagle]

I'm sure we'd consider a PR that used the json feeds. But it's difficult to know what the right solution is. A lot of this is guesswork -- we have almost zero visibility inside the processes that Mozilla uses to release their products.

We no longer support the use of Thunderbird where I work, so I'm less motivated to fix this quickly -- in the past if we waited a bit it just resolved itself. Perhaps that won't be the case this time. Always in motion, the future is.