ArgoCD Sync Stuck Due to PreSync Hooks and Continuous Dry Run (v2.12.3, K8s 1.30)

[Naor-Armo]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

ArgoCD is stuck in a continuous syncing state and does not transition to Progressing or Succeeded. The logs indicate that ArgoCD detects differences and applies resources, but it only performs a dry-run instead of making actual changes.

Additionally:

PreSync hooks (pre-install, pre-upgrade) seem to block progress.
serverSideDiff=false and serverSideApply=false, preventing ArgoCD from properly applying changes.
Skipping PreSync hooks manually using "Apply Only" in the UI works.

example for used hooks
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-30"
"helm.sh/hook-delete-policy": before-hook-creation

If I manually delete PreSync resources (Jobs, ConfigMaps, or SealedSecrets), ArgoCD recreates them and syncs successfully.

it's only happening in one (env) cluster we have 2 other(envs) clusters with different argocd ( the same versions and manifests) that runs the same charts and it works there as it should

Current Workarounds Attempted:
Skipping PreSync Hooks using "Apply Only" in the ArgoCD UI
this allows the sync to succeed.

Manually deleting PreSync resources (Jobs, ConfigMaps, SealedSecrets)

After deletion, ArgoCD recreates them and syncs successfully.

Trying to ignore Helm hooks in ignoreDifferences
Used the following but it did not fully resolve the issue:

ignoreDifferences:

• group: batch
  kind: Job
  jsonPointers:
  • /metadata/annotations/helm.sh~1hook
• group: v1
  kind: ConfigMap
  jsonPointers:
  • /metadata/annotations/helm.sh~1hook
• group: ""
  kind: ServiceAccount
  jsonPointers:
  • /metadata/annotations/helm.sh~1hook

Still, ArgoCD remains in a sync loop.
Forcing a full sync with --force

argocd app sync kubescape --force

Did not resolve the issue.

Expected Behavior:
ArgoCD should progress past the PreSync stage after running the hooks.
Sync should transition to Progressing and Succeeded instead of remaining in Running.
The dry-run behavior should not block the actual resource reconciliation.

ArgoCD & Kubernetes Environment Details:
ArgoCD Version: v2.12.3
Kubernetes Version: 1.30
Sync Policy: Automated (prune: true, selfHeal: true)

Logs:

time="2025-03-04T09:39:12Z" level=info msg="Applying resource Service/kubevuln in cluster: https://172.20.0.1:443, namespace: kubescape" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
...
time="2025-03-04T09:39:12Z" level=info msg="Updating operation state. phase: Running -> Running, message: 'waiting for deletion of /ServiceAccount/service-discovery and 2 more hooks'" application=argocd/kubescape syncId=03397-xzIyK
time="2025-03-04T09:39:14Z" level=info msg="sync/terminate complete" application=argocd/kubescape duration=2.202488055s syncId=03397-xzIyK
time="2025-03-04T09:39:14Z" level=info msg="No operation updates necessary to 'argocd/kubescape'. Skipping patch" app-namespace=argocd app-qualified-name=argocd/kubescape application=kubescape project=cluster-apps

Impact:
ArgoCD is unable to complete sync cycles.
This blocks critical application updates.
Manual intervention is required every time.

Some questions :
Why is ArgoCD not progressing past PreSync hooks even after applying them?
Why is ArgoCD doing a dry-run (serverSideDiff=false and serverSideApply=false) instead of applying changes?
Is there a way to configure ArgoCD to always delete and recreate PreSync hooks properly?
Is there a recommended approach to prevent ArgoCD from getting stuck in this state when using pre-install and pre-upgrade hooks?

Would appreciate guidance from the ArgoCD team on resolving this issue.

[andrii-korotkov-verkada]

Maybe a dumb question, but why is hook weight negative (i.e. -30)? How does it work?

[Naor-Armo]

so adding to here that we use the same argo version in 2 different envs and it works. only one env have this issue and from some reasion all the presync resources that argo tries to delete and recreate have finelizers and that's why it cant delete

and that's why I have this issue

finalizers:                                                                                                                                                                      
 - foregroundDeletion

regarding the hook weight from what I know : Helm executes all hooks of the same type in order from lowest to highest weight,negative weights run first, while higher (positive) weights run later and If no weight is set, the default is 0.

Would appreciate guidance on resolving this issue.

[andrii-korotkov-verkada]

I'll bring this up with other contributors. Meanwhile, can you try upgrading to 2.14 to see if that helps, please?

[andrii-korotkov-verkada]

Didn't have time today to discuss at the contributors meeting https://docs.google.com/document/d/1xkoFkVviB70YBzSEa4bDnu-rUZ1sIFtwKKG1Uw8XsY8/edit?tab=t.0. Feel free to join next time and put this topic for discussion.

[Naor-Armo]

I have 3 cluster with the same version only one have this issue and this is production I cant upgrade it right now

This solution works for now, but I want to know the meaning of using it. Can it accidentally delete PVCs or any other issues that I need to know about?

syncPolicy:
    syncOptions:
      - PrunePropagationPolicy=background 

I'll bring this up with other contributors. Meanwhile, can you try upgrading to 2.14 to see if that helps, please?