Adjust docker image cleanup tolerances. #238
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jenkins and the cleanup script do their accounting slightly differently so there are times when the cleanup script runs and Jenkins still considers the disk to be too full. By bumping the cleanup minimum to 60G that gives us a pretty comfortable margin over the 50G minimum for Jenkins. I also bumped the percentage so we don't flap on and off as frequently. These changes affect all agents but I don't think any of them are harmed by the tuning of these parameters. As we work on the future agents we're going to delegate cleanup responsibility to docker prune commands, which we are already doing for the Jenkins host which previously did not have a configured cleanup script.
cottsay
approved these changes
Aug 18, 2020
|
The specific numbers of the different nodes should be taken into consideration here (non-ARM as well as ARM nodes) and based on their disk sizes the limits should be set. Atm for e.g. ARM nodes which have less than 500 GB disk space the cleanup script only kicks in at 50 GB (since 10% of less than 500 GB is less) but the configured threshold in Jenkins is also 50 GB which leaves no margin. So when the cleanup script kicks in it is already too late and Jenkins will take the node offline. |
dirk-thomas
reviewed
Aug 26, 2020
| @@ -132,7 +132,7 @@ | |||
|
|
|||
| # clean up containers and dangling images https://github.com/docker/docker/issues/928#issuecomment-58619854 | |||
| cron {'docker_cleanup_images': | |||
| command => "bash -c \"python3 -u /home/${agent_username}/cleanup_docker_images.py --minimum-free-percent 10 --minimum-free-space 50\"", | |||
| command => "bash -c \"python3 -u /home/${agent_username}/cleanup_docker_images.py --minimum-free-percent 30 --minimum-free-space 60\"", | |||
There was a problem hiding this comment.
I don't think 30% is necessary / reasonable. That would waste a lot of disk space rather than using it for caching.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jenkins and the cleanup script do their accounting slightly differently
so there are times when the cleanup script runs and Jenkins still
considers the disk to be too full.
By bumping the cleanup minimum to 60G that gives us a pretty comfortable
margin over the 50G minimum for Jenkins. I also bumped the percentage so
we don't flap on and off as frequently.
These changes affect all agents but I don't think any of them are harmed
by the tuning of these parameters.
As we work on the future agents we're going to delegate cleanup
responsibility to docker prune commands, which we are already doing for
the Jenkins host which previously did not have a configured cleanup
script.