Terminated pods remain visible in metrics

[Flasheh]

Hey

For a certain application that regularly spins up in our CICD flow, I've noticed that some instances keep hanging around in the metrics long after the pod gets terminated. In the sample metrics below you'll see that there are different values for latest_version indicating old data. Restarting version_checker does clean these up.

I'm running:
version-checker:
Container ID: docker://581de433d72c6199d5194af1eb3c5fb8deff331acb932567ab92368afb4ae64c
Image: quay.io/jetstack/version-checker:v0.2.1
Image ID: docker-pullable://quay.io/jetstack/version-checker@sha256:5f6f8ba0b671db2382f811b58fad64a2fd22ca1caaf087f9708c8e4b8309d7e0

Metrics:

version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.0.2",namespace="jenkins",pod="release-bump-9371-hbjm2-z1g91-6sd8x"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.0.3",namespace="jenkins",pod="release-bump-9379-z9lfh-xp6gf-j22qn"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9442-gc98d-9rc8c-hzjld"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9450-9jc31-fr5f5-kzxch"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9451-rp62c-kt9fj-4msmb"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.4",namespace="jenkins",pod="release-bump-9483-5q6pn-c9951-pvk7l"} 0
version_checker_is_latest_version{container="renovatebot",current_version="23.99.0-slim",image="docker.io/renovate/renovate",latest_version="24.2.5",namespace="jenkins",pod="release-bump-9493-bsnzm-2f3q9-0nczp"} 0

[davidcollom]

This sounds to be the likely cause of #76 - you can also provide the --image-cache-timeout to reduce the timeout/cleanup of terminated images (default is 30 minutes).

[Zefiii]

Hi

I keep seeing the same behavior, even after updating. So, I have updated version-checker to the new version. Waited for an hour (I did not modify the timout-cache option, so, in 30 min the old version checker pod should be deleted) and I keep seeing the old version-checker pod:

version_checker_is_latest_version{container="version-checker",container_type="container",current_version="v0.2.6@sha256:10f1a09e57412a874d8a63f3b832207ea63df2ee683d7a195a31a4b97c87b368",image="quay.io/jetstack/version-checker",latest_version="v0.3.0@sha256:9fc1feadfc6a0d8657bcfca0389364539630f28abb6ccb15ce54a2d76cb090bd"} 1
version_checker_is_latest_version{container="version-checker",container_type="container",current_version="v0.3.0@sha256:a7471e20b37c22bc7b82dd67ec426c3953cb3bc220858a66ccd30c3f71aece4a",image="quay.io/jetstack/version-checker",latest_version="v0.3.0@sha256:9fc1feadfc6a0d8657bcfca0389364539630f28abb6ccb15ce54a2d76cb090bd"} 0

I would expect that after 30 minutes the old version-checker with version v0.2.6 to be deleted. Is this the expected behavior?

Right now, as a workaround we are restarting the pod once every day, but it feels a little bit ugly.

[github-actions]

This issue has been automatically marked as stale because it has not had
any activity in the last 90 days. It will be closed if no further activity
occurs. Thank you for your contributions.