Pushing docker image to GPR: manifest invalid: Only schema version 2 is supported #25315
-
Since about an hour I’m getting the following error when I try to push a docker image tp GPR:
Two hours ago everything work fine, the Dockerfile hasn’t changed, neither did the upstream image. Did Github change something an hour ago? |
Beta Was this translation helpful? Give feedback.
Replies: 19 comments
-
I also had things working, then they started failing. From here there did seem to be someone who found tagging with the git sha was problematic (too many characters). |
Beta Was this translation helpful? Give feedback.
-
Tried it with the following tags so I doubt it’s the tag langth: Also interesting detail is that I’m building the image I’m building this image on publicly on Github Actions and push it to Docker Hub. (Frequently, in this case the base image was less then a day old.) |
Beta Was this translation helpful? Give feedback.
-
I have the same issue pushing to the Github repository but via CircleCI, can you update the version of docker client your using it the Github action for pushing… ( This may be a case of just updating the version of the GitHub action you’re using for pubishing) I think its just because the old API has been deprecated and we are still pushing using older versions of the api ? I updated my docker image used to publish my image to Docker version 19.03.5, build 633a0ea838 with no success, build is still failing with the metadata error. |
Beta Was this translation helpful? Give feedback.
-
I tried your suggestion by trying to push outside of Github Actions. I’m currently on Docker version 19.03.2, build 6a30dfc on mac and I’m still getting the manifest invalid error. This may be caused by a different issue? |
Beta Was this translation helpful? Give feedback.
-
Sorry, I just got mine building with another version and had the same issue again… Are people using the github private repo ? I also don’t know if it’s worth noting that the docker manual says : > When pushing images, clients which support the new manifest format should first construct a manifest in > the new format. If uploading this manifest fails, presumably because the registry only supports the old > format, the client may fall back to uploading a manifest in the old format. This makes me think that it may be failing to upload using v2 then reverting to v1 where it then gets this error back from the server. |
Beta Was this translation helpful? Give feedback.
-
Yes I’m trying to push from a private repo to it’s private repo. This fallback to v1 could be it, there is a noticeable delay after the last layer is pushed before the error shows up. Nevermind that, for both success and failure the delay is 6 seconds. Success gives you the latest digest SHA And failure that error. |
Beta Was this translation helpful? Give feedback.
-
FYI, I sent a message detailing the issue github developer support, who have escalated the issue to the product team with the link to this community post. I’ll update if I hear back from them. Update, I figured out my issue: So nested in the logs ( journalctl ) is the actual issue:
Seems our account is over the storage allocation ( I’m not the certain what this limit is ) and therefore we are no longer able to upload our images. It is attempting to upload using schema 2 then failing and falling back, docker does not know how to handle the error from the github private repo and is therefor falling back in a reall confusing manner. So i am not certain if we can delete older packages to free up space or what the limit is. I used journalctl | grep dockerd (since I aways forget the flags) to get the full logs on Arch Linux, this will be different on OSX (https://docs.docker.com/docker-for-mac/troubleshoot/) or windows (https://docs.docker.com/v17.12/docker-for-windows/troubleshoot/). But Docker will log the reason it falled back to the version 1 schema in its logs. |
Beta Was this translation helpful? Give feedback.
-
So nested in the logs ( journalctl ) is the actual issue:
Seems our account is over the storage allocation ( I’m not the certain what this limit is ) and therefore we are no longer able to upload our images. It is attempting to upload using schema 2 then failing and falling back, docker does not know how to handle the error from the github private repo and is therefor falling back in a reall confusing manner. So i am not certain if we can delete older packages to free up space or what the limit is. I used journalctl | grep dockerd (since I aways forget the flags) to get the full logs on Arch Linux, this will be different on OSX (https://docs.docker.com/docker-for-mac/troubleshoot/) or windows (https://docs.docker.com/v17.12/docker-for-windows/troubleshoot/). But Docker will log the reason it falled back to the version 1 schema in its logs. |
Beta Was this translation helpful? Give feedback.
-
We are seeing the same error and we have fixed tag length. Started after a few days, it was working before. Hopefully some staff member can write here to give more information. |
Beta Was this translation helpful? Give feedback.
-
A related question: if the space is limited, can we automatically remove old images via some command/API? |
Beta Was this translation helpful? Give feedback.
-
Yes we can, I already have a work in progress action to remove package versions: https://github.com/WyriHaximus/github-action-delete-package Which is based on this bit of documentation: https://help.github.com/en/github/managing-packages-with-github-packages/deleting-a-package#deleting-a-version-of-a-private-package The thing is I’m trying to get the list of this packages with this GraphQL query but it’s giving me an unexpected not found error:
Glueing these two together and get them to work should make you able to delete old docker image tags. |
Beta Was this translation helpful? Give feedback.
-
UPDATE: When we increased our total quota for the registry the action started working. Also looking into that delete API, but we have a different approach because we don’t want to delete last n, we want to delete all images related to a tag too. Pretty simple to call the API once you get your head around it. I find compounding actions with |
Beta Was this translation helpful? Give feedback.
-
I might be stating things that people already know, But talking to github support they advised that the default storage for Github team is: GitHub Team
Also that the organisation storage can be checked by viewing: https://github.com/organizations/ORG/settings/billing The following works for me, to list versions of packages on a project.
|
Beta Was this translation helpful? Give feedback.
-
One thing you have to be careful with, we just discovered, is not deleting all the versions for a package; Github at that point will prevent you from publishing any version under that name and will return this error:
Still waiting for an answer from support to understand how to fix it. |
Beta Was this translation helpful? Give feedback.
-
@thepont Yeah that fixes it. Would be nice to get a more descriptive error message in the long run tbh. Thank you for fixing the GraphQL query as well, they will get me to something working this weekend 🎉. @aledalgrande That’s good to know! Wanted to do a latest tag for master builds and SHA or tag title for tagged releases, but sounds like the latest tag for master builds isn’t an option then :(. The reason I’m also doing an Action is to both give back and make it easier to compose actions between my projects. I’m intending to build an auto clean up of old image tags that aren’t relevant anymore but are still taking up space. (Also where I can I’m building Actions directly on alpine with a quick package install keeping that build time as low as possible.) @thepont @aledalgrande Thank you both for all the help on this. I’ve accepted @thepont’s post because it’s a nice concise summary, but want to let you know you both been very helpful 👍. |
Beta Was this translation helpful? Give feedback.
-
Any update from support on this error? |
Beta Was this translation helpful? Give feedback.
-
Did you ever get an answer? I am having the exact same problem now. |
Beta Was this translation helpful? Give feedback.
-
I’m experiencing the same issue now, were you able to find a solution? |
Beta Was this translation helpful? Give feedback.
-
How did any of you delete a docker package? |
Beta Was this translation helpful? Give feedback.
I might be stating things that people already know, But talking to github support they advised that the default storage for Github team is:
GitHub Team
Also that the organisation storage can be checked by viewing: https://github.com/organizations/ORG/settings/billing
The following works for me, to list versions of packages on a project.