What services to debug to identify the problem causing "InvalidSpecError: Application referencing project ... which does not exists"?

[DS-KrzJon]

Hi all,

Background:
I am currently using ArgoCD version v2.8.4+c279299, hosted on a GKE Cluster in Europe. The issue arises when syncing applications intended for deployment in a Shanghai cluster on AliCloud. My ArgoCD instance is deployed via Helm chart, and we use nginx for ingress. Other applications in the same ArgoCD instance do not exhibit this issue, leading me to suspect that the problem may be related to internal timeouts or readiness probe configurations.

Issue Description:
The specific error I encounter is an "InvalidSpecError" indicating that the application is referencing a project which does not exist. This error occurs sporadically during the synchronization of large projects. The project in question, d-cn-shanghai, does indeed exist as confirmed by multiple checks, yet the error persists intermittently.

Troubleshooting Steps Taken:

Cache Refreshing: Executed commands to refresh ArgoCD’s cache to ensure that the internal state is consistent with the Kubernetes cluster state.
Logs Checking: Reviewed logs from the following services to look for relevant error messages or clues:

• argocd-repo-server
• argocd-server
• argocd-applicationset-controller

Configuration and Permissions Review: Ensured that all configurations, including ConfigMaps and Secrets, are correctly set and that RBAC settings are appropriate.
ApplicationSet and Application Controller Investigation: Since we use ApplicationSets, I investigated if there's an issue with how these sets are generating or managing applications.
Cluster and Namespace Health Checks: Conducted general health checks on the cluster and the specific namespace to rule out broader Kubernetes issues.

Speculations:
Given that other applications are syncing without issues, it suggests a potential misconfiguration or a specific condition affecting only the Shanghai cluster deployments. The intermittent nature of the error might indicate a problem related to Kubernetes resource limits, network latency, or other environmental conditions unique to the Shanghai cluster.

Request for Insights:
I am looking for guidance on additional debugging steps I might have overlooked, particularly any ArgoCD or Kubernetes services that should be monitored more closely. Insights into similar issues or recommended configurations to handle such cross-regional deployments would be highly appreciated.

BR, Kris