During publishing, "Download Latest Image Info" step can fail with 429 too many requests

[lbussell]

docker-tools/eng/common/templates/jobs/publish.yml

Lines 140 to 147 in fc626a4

	- script: >
	curl -fSL
	--output $(imageInfoHostDir)/full-image-info-orig.json
	https://raw.githubusercontent.com/$(gitHubVersionsRepoInfo.org)/$(gitHubVersionsRepoInfo.repo)/refs/heads/$(gitHubVersionsRepoInfo.branch)/$(gitHubImageInfoVersionsPath)
	condition: and(succeeded(), eq(variables['publishImageInfo'], 'true'))
	displayName: Download Latest Image Info

This pipeline step is just a raw curl command which can fail sometimes causing our publish pipeline to be unreliable. In the example below it fails with a 403 unauthorized. Things to consider:

• At a minimum we should wrap this request in an exponential backoff retry
• Do we need to authenticate for this request? As our GitHub bot account?

Example pipeline where this is failing: https://dev.azure.com/dnceng/internal/_build/results?buildId=2746286&view=logs&j=739bb7ec-7296-5b40-7dff-ee3297bfdaa4&t=bf6a5b93-88b0-5310-04fe-dec22ee32d0e&s=380ab8c6-4139-55ff-15c4-fb48260f4dca

[lbussell]

Correction: the original error was 429 "too many requests".

[lbussell]

From what I can tell, raw.githubusercontent.com does not accept authentication [citation needed]

[lbussell]

It appears that the unauthenticated rate limit is 60 requests per IP per hour: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28

[lbussell]

Putting the image info behind a retry helped some, but we should look into a better workaround:

• Can we authenticate to raw.githubusercontent.com?
• Can we use the gh CLI to download the file (authenticated)?
• Can we clone the repo instead (authenticated)?

https://github.com/dotnet/dotnet-docker-internal/issues/8439 is the long-term solution.