Artifact GC still acting up for failed Workflows #12845
Replies: 18 comments 32 replies
-
|
I haven't used or worked on Artifact GC, but can try to help
No, I'm reading that as the Artifact GC Pod failed and so the Workflow has not been cleaned up properly and so has that message on it. That's also why it didn't finish deleting (as it failed to clean up), if I'm understanding correctly, as you don't have
I think you are correct -- would you like to submit a PR to fix that?
The Artifact GC Pod log message is the same log message you get when you try to add |
Beta Was this translation helpful? Give feedback.
-
The docs would suggest it uses the same Artifact Repository configuration (or at least don't suggest otherwise). If that's all fine, this might be a bug or missing feature; the much of the Artifact GC code is here and I don't see anything obvious wrong with it. It seems to inherit the same Secret Volumes + Mounts as Workflow Pods. |
Beta Was this translation helpful? Give feedback.
-
done, see #12935 |
Beta Was this translation helpful? Give feedback.
-
I would, but I couldn't find the correct location yet.
That is correct. I might consider enabling this once this kind of behavior is an exception and doesn't happen for every failed workflow. Otherwise, my artifact repository fills up with orphaned artifacts.
Are you sure? I mean, look at point 6 again. In the logs are all necessary details for the artifact repository. The only place where this is can come from is the controller-level artifact repository configuration. If that component/container would be unable to get that config (somehow), it wouldn't be able to write those logs. Based on the fact, that the container can write those logs, I would assume, the config is accessible, but there is another problem hidden behind that generic error message. But I can't fix it if I don't know what it is.
Argo runs in the |
Beta Was this translation helpful? Give feedback.
-
That line is right here, and indeed the interpolation arguments are in the wrong order. It's the only line that pops up for "failed for strategy"
The Artifact GC Pod is neither the Controller nor the Workflow 😅 I want to make sure it's running in the correct namespace as well.
Oh I missed that, didn't realize that the first few log lines had the correct artifact repository while still getting an error. That is indeed odd 🤔
Fortunately, this is coming from the Executor, so there's not too much code that could produce this. Here's the code for |
Beta Was this translation helpful? Give feedback.
-
|
Since this is only happening for failed Workflows and based on the above, my guess is that some of the failed tasks may have a corrupted |
Beta Was this translation helpful? Give feedback.
-
Yes, I'm aware :) The
It should, yes. Otherwise even the successful workflows wouldn't be able to delete artifacts. But I was thinking about that as well. I still have that on my todo list to verify if there are any errors in the audit logs of MinIO while that happens. It's one of the very few ways to potentially make whatever-this-is visible.
I looked at the code and I don't see a reason, why this would suddenly fail? 🤔 Also: What is the
What could possibly make the GC work for one artifact but not for another? 🤔
What does that mean exactly? I can check, when I know what I am looking for. Thanks btw. for all the answers. Hopefully I can fix this some day. It's driving me crazy. Since I can easily implement a workaround for successful workflows (exits handler), failed workflows are the main(!) reason, why I need this feature. |
Beta Was this translation helpful? Give feedback.
-
Yes that's more or less what I meant by corrupted
My guess is that the Workflow Executor was terminated before it could properly save all the state, so only some got properly saved.
The |
Beta Was this translation helpful? Give feedback.
-
|
Just an idea: is it possible, that for failed workflows, somehow additional Kubernetes resources are being created/used that my workflow service account can't access, and that's why the artifact repository seems to be not configured? This is the role, used by the workflow service account. By any chance, is there some important permission missing? apiVersion: "rbac.authorization.k8s.io/v1"
kind: "Role"
metadata:
name: "workflow-argo"
rules:
# See https://argoproj.github.io/argo-workflows/workflow-rbac/
- apiGroups:
- "argoproj.io"
resources:
- "workflowtaskresults"
verbs:
- "create"
- "patch"
- apiGroups:
- "argoproj.io"
resources:
- "workflows"
verbs:
# Permissions to submit a workflow
- "list"
- "create"
# Permissions to resubmit, retry, resume, suspend a workflow
- "get"
- "update"
# See https://argoproj.github.io/argo-workflows/walk-through/artifacts/#service-accounts-and-annotations
- apiGroups:
- "argoproj.io"
resources:
- "workflowartifactgctasks"
verbs:
- "list"
- "watch"
# See https://argoproj.github.io/argo-workflows/walk-through/artifacts/#service-accounts-and-annotations
- apiGroups:
- "argoproj.io"
resources:
- "workflowartifactgctasks/status"
verbs:
- "patch"
|
Beta Was this translation helpful? Give feedback.
-
|
I think this is fine. |
Beta Was this translation helpful? Give feedback.
-
|
@static-moonlight can you confirm that the artifact still exists in the repository? Have you done a manual inspection of it? I assume the artifactgc finalizer doesn't get removed, right? |
Beta Was this translation helpful? Give feedback.
-
|
@static-moonlight I just experienced an artifactgc failure for one of my failed workflows. Will investigate and report back. |
Beta Was this translation helpful? Give feedback.
-
|
Hmm... It turns out that in my case, I was just being too impatient. GC was just taking longer than I expected. |
Beta Was this translation helpful? Give feedback.
-
|
@static-moonlight is your workflow controller configmap syntax correct? Notably:
https://argo-workflows.readthedocs.io/en/latest/workflow-controller-configmap/ |
Beta Was this translation helpful? Give feedback.
-
|
Since artifacts get uploaded properly and successful Workflows' artifacts get deleted properly, it must be correct. |
Beta Was this translation helpful? Give feedback.
-
|
@static-moonlight I've added a test specific to failed workflow artifact garbage collection. It seems to be working. #12904 |
Beta Was this translation helpful? Give feedback.
-
|
My guess is that there's a race where |
Beta Was this translation helpful? Give feedback.
-
It should. Otherwise, my editor would most likely show me syntax errors, and nothing would work in the cluster. I don't like the inline yaml config ( apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argo
# [...]
configMapGenerator:
- name: workflow-controller-configmap
behavior: replace
files:
- config=config.yaml
# [...]config.yaml: # [...]
artifactRepository:
s3:
bucket: argo-artifacts
endpoint: s3.storage:9999
insecure: true
accessKeySecret:
name: artifact-repository
key: USERNAME
secretKeySecret:
name: artifact-repository
key: PASSWORD
# [...]It works nice though for normal operation. Workflows are being executed. The artifact repository only contains artifacts for active workflows. Which means it works ok. Because of our latest stabilization, workflow errors are kind of rare now. I'm currently working on a dedicated setup to forcefully fail a few workflow runs to hopefully find the missing puzzle piece which is causing all this trouble ... |
Beta Was this translation helpful? Give feedback.
-
|
I have a suspicion that it has something to do with this: "Pod was active on the node longer than the specified deadline" I have collected roughly 200 defect input files (which will cause the workflow to fail) and threw them at the cluster. Most of those workflows failed, but not because they couldn't handle the input, but because they took too long an hit the configured "activeDeadlineSeconds" of the pod. And for all those failed workflows, I see the "Artifact garbage collection failed" message again. ... I'm doing more tests on this one. |
Beta Was this translation helpful? Give feedback.
-
|
It seems like it might indeed be the |
Beta Was this translation helpful? Give feedback.
-
|
Well, that doesn't seem to be it either. When I try to specifically fail a single workflow by setting a pods So far it seems that there is a random factor involved. Sometimes it works and sometimes it doesn't. In high-load situations I could produce lots of artifact-gc errors. During normal operation I almost never have them. I'm a little lost here. I can't seem to find the magic recipe to reliably (re)produce this behavior. But apparently it still happens sometimes. Any more ideas? |
Beta Was this translation helpful? Give feedback.
-
It may depend on the size of the artifact (i.e. how long it takes to upload) as well as how long it takes the step to produce the artifact (i.e. whether it does so before or after
Your Workflow Pods don't seem to have resource requests, so I wonder if they're getting starved potentially? And that may then cause them to hit the Also yea, high load is typically when you see race conditions like these get hit more often, that's why they're rare (and hard to diagnose, debug, reproduce, test, etc). |
Beta Was this translation helpful? Give feedback.
-
|
I did the same test again. This time I limited the number of parallel workflows with a semaphore. The result: green across the board. The workflows failed, as expected, and they were removed successfully, including artifact gc. This amplifies my suspicion that workload has something to do with the issue. I would even throw the theory of a potential race condition into the ring. All I can do now is to do incremental tests, increasing the number of parallel workflows each time, and look when it starts to break ... EDIT: The intended limit (for parallel workflows) is set to 5. I increased it to 10, 15 and now 20. With the limit of 20, things start to break. I have a couple left-overs from this test run ... with "Artifact garbage collection failed" errors again. I assume the number of artifact-gc-errors increases when I put even more load on the cluster. Meaning: Load is obviously a factor. Can someone confirm this? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for getting to that! On first glance, it only has a single I see that your I see you're using artifact logs as well, but I wouldn't think that's related either. The one with failed GC does have a The one with failed GC has two strategies in It would be helpful if we knew which artifacts weren't deleted -- then we could match those up to their nodes/steps. The original error message doesn't say which failed to delete, only which ones successfully deleted (potentially that error message could be improved). The original error message is also from different Workflows ofc. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Hmm there are also only 5 items in the Wait a minute, I think I did find a corrupted artifact: workflow-j7mjs-2762173675:
boundaryID: workflow-j7mjs
children:
- workflow-j7mjs-3284464227
displayName: create-variant-d
finishedAt: "2024-04-12T14:34:55Z"
hostNodeName: node-5.company.com
id: workflow-j7mjs-2762173675
inputs:
artifacts:
- name: RAW_DATA
path: /tmp/raw-data
s3:
key: workflow-j7mjs/workflow-j7mjs-download-raw-data-538700425/RAW_DATA.tgz
parameters:
- name: PRODUCT_NAME
value: path/to/input-file
- name: REGION
value: region-2
- name: MAP_TYPE
value: map-type-2
message: Pod was active on the node longer than the specified deadline
name: workflow-j7mjs.create-variant-d
outputs:
artifacts:
- name: FINAL_DATA
path: /tmp/final-data # <------- this is missing an s3.key??
- name: main-logs
s3:
key: workflow-j7mjs/workflow-j7mjs-create-data-variant-2762173675/main.log
exitCode: "143"
phase: Failed
progress: 0/1
resourcesDuration:
cpu: 18
memory: 182
startedAt: "2024-04-12T14:33:23Z"
templateName: create-data-variant
templateScope: local/
type: PodThat is one of the steps that failed due to Was that artifact ever uploaded? It might not have been if the Pod got killed due to an Also if this is the root cause, you should be able to cross-check that against other failed artifact GC Workflows and see if they're also missing an |
Beta Was this translation helpful? Give feedback.
-
|
That is a lot on information. And I hope I understand that correctly. Unfortunately, I took too long and by now I lost log and artifact data due to automatic cleanup (it's just a test environment, and orphaned data was never meant to be kept for very long here, I disabled it for now). However, lets say I would run that test again, and lets say I could link those gc failures to missing |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that would be definitive evidence of root cause. Something is missing writing the If this were due to an inevitable race and the artifact is never uploaded, we could safely ignore this error. Just a few scenarios off the top of my head, I haven't worked with artifacts too much actually (didn't really use them when I was just a user and one of few areas I haven't worked extensively on as a maintainer yet), so would probably want another contributor to find and fix that scenario. Garrett's fixed a bunch of races with artifacts and GC, hence why I cc'ed him here. |
Beta Was this translation helpful? Give feedback.
-
|
@juliev0 @shuangkun thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
I apologize, I haven't been able to go through this entire thread yet, but in the off chance it helps, I did notice a bug in the existing ArtifactGC deletion pod code while I was reviewing a separate PR - that person was going to fix it in theirs, but their PR never got merged unfortunately and it still needs fixing. I will ask my colleague (who wrote it) to fix it. The issue is that if this line returns an error, it isn't handled. |
Beta Was this translation helpful? Give feedback.
-
|
@juliev0 the gist is in this comment; one of the artifacts in |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, my comment is a separate thing because it does seem like what Anton found as root cause makes sense. The call to |
Beta Was this translation helpful? Give feedback.
-
|
BTW, for future, reference, did you ever do |
Beta Was this translation helpful? Give feedback.
-
|
I'll take a look and see if I can think of any race condition, etc, for why that wouldn't get saved in the node status |
Beta Was this translation helpful? Give feedback.
-
|
So, the |
Beta Was this translation helpful? Give feedback.
-
|
It seems like this line is where s3 should be set, so it must not be getting called, or it's getting called and it's failing. |
Beta Was this translation helpful? Give feedback.
-
|
It sounds like we don't know if the Artifact ever got written in the first place? Yes, I am not sure if there's logic in here to take that into account. I see the file gets looked for here. If that fails, an error is simply written, but then the flow continues and the Outputs are still reported here (with no "s3" ever written) @Garett-MacGowan @shuangkun did your tests of Failed Workflows ever involve artifacts that never got written at all? (if you know) |
Beta Was this translation helpful? Give feedback.
-
|
@juliev0 I believe all my tests checked that the artifacts were written. I assume we would need to set up a mock, which I am not experienced in with golang yet. For reference, here's a PR I have for testing failed workflows, which I haven't had time to address the comment on yet #12904 |
Beta Was this translation helpful? Give feedback.
-
|
This Draft PR is what I have in mind. I haven't tested it at all, and not sure if anyone wants to take it over... |
Beta Was this translation helpful? Give feedback.
-
|
This seems like a reasonable thing to be doing. I'm not the best person to test this considering my comment above WRT mocks. |
Beta Was this translation helpful? Give feedback.
-
|
You don’t have to take it! But it can be tested with e2e and not necessarily unit test I suppose, right? (theoretically may not even need to be a failed workflow) |
Beta Was this translation helpful? Give feedback.
-
I still have problems with the artifact GC with Argo Workflows 3.5.5.
artifact-repositoryThis is working fine, as long as there are no errors. I can see artifacts being created and automatically removed in that bucket (for successful workflows).
"Artifact garbage collection failed"
"ArtifactGCError: Artifact Garbage Collection failed for strategy [...], pod OnWorkflowDeletion exited with non-zero exit code: check pod logs for more information".
The way how this reads is that the GC failed, because the workflow failed?!
(BTW: I think this error message is messed up as well: "[...] failed for strategy {pod-name}, pod {strategy} [...]". I think the placeholders are wrong)
[...]-artgc-[...]container I found this:So, in conclusion: the artifact GC failed, for an unknown reason. The
[...]-artgc-[...]container doesn't log anything specific, but it tells me to configure an artifact storage, which I already did (see point 1), and doesn't log any errors whatsoever. And remember: the artifact repository works absolutely fine for successful workflows. Which means: the S3 service and bucket are accessible, the credentials are correct, the required S3 permissions are working as well.So what is the problem here? What am I missing? Why is the GC somehow not working for failed workflows?
Beta Was this translation helpful? Give feedback.
All reactions