ArgoCD rpc error from default installation

[rzvn2600]

Hi All,

I am tryinf to deploy an application ito a fresh argocd installation and it does not work. I cannot create an app from the UI/CLI or through kubectl apply. Everything fails with the following error:

➜  argo_test_yaml argocd app create --name blue-green --repo https://github.com/argoproj/argocd-example-apps --dest-server https://kubernetes.default.svc --dest-namespace default --path blue-green && argocd app sync blue-green

FATA[0020] rpc error: code = InvalidArgument desc = application spec for blue-green is invalid: InvalidSpecError: repository not accessible: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.20.86.254:8081: i/o timeout" 

Pod stats:

➜  terraform kubectl get pods -n argocd
NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          2d19h
argocd-applicationset-controller-58f69d4b8f-tdm84   1/1     Running   0          2d19h
argocd-dex-server-ff489bd4-vmn6h                    1/1     Running   0          2d19h
argocd-notifications-controller-567f4c469-699jp     1/1     Running   0          2d19h
argocd-redis-55d64cd8bf-6kp9n                       1/1     Running   0          3h52m
argocd-repo-server-86d9878d56-cmw8r                 1/1     Running   0          102m
argocd-server-54bc687b4b-jn4zj                      1/1     Running   0          101m

kubectl logs:

argocd-server

time="2022-07-18T10:34:49Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"exp\":1658226650,\"iat\":1658140250,\"iss\":\"argocd\",\"jti\":\"cc016145-c46d-4fc3-b82a-95bc5270bac1\",\"nbf\":1658140250,\"sub\":\"admin\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2022-07-18T10:34:49Z" span.kind=server system=grpc
time="2022-07-18T10:35:09Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for blue-green is invalid: InvalidSpecError: repository not accessible: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.20.86.254:8081: i/o timeout\"" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2022-07-18T10:34:49Z" grpc.time_ms=20024.885 span.kind=server system=grpc
time="2022-07-18T10:35:22Z" level=info msg="Alloc=12679 TotalAlloc=91128 Sys=28497 NumGC=53 Goroutines=76"

argocd-repo-server different app deployed:

time="2022-07-18T10:38:20Z" level=error msg="finished unary call with code Unknown" error="unexpected client error: unexpected requesting \"https://xxx.xxx.com/xx.xx/argocd/info/refs?service=git-upload-pack\" status code: 301" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2022-07-18T10:38:20Z" grpc.time_ms=175.553 span.kind=server system=grpc`

CLI logs from app deployed through kubectl:

➜  terraform argocd app logs guestbook --loglevel debug
FATA[0000] stream read failed: rpc error: code = Unknown desc = error getting app resource tree: error getting cached app state: ComparisonError: rpc error: code = Unknown desc = unexpected client error: unexpected requesting "ttps://xxx.xxx.com/xx.xx/argocd/info/refs?service=git-upload-pack" status code: 301 
➜  terraform 

app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://gitlab.lukapo.com/razvan.gherlea/argocd
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook

But from the POD level I can easily clone the repo without any credentials. no networking issue:

argocd@argocd-server-fd9ccd79b-dxrd6:~$ git clone https://xxx.xxx.com/xx.xx/argocd.git
Cloning into 'argocd'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 7 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (7/7), done.
argocd@argocd-server-fd9ccd79b-dxrd6:~$ ls -ltr
total 0
drwxr-xr-x 4 argocd argocd 61 Jul 15 09:00 argocd
argocd@argocd-server-fd9ccd79b-dxrd6:~$ ls -ltr *
total 8
drwxr-xr-x 2 argocd argocd   33 Jul 15 09:00 argocd-appprojects
-rw-r--r-- 1 argocd argocd 6233 Jul 15 09:00 README.md
argocd@argocd-server-fd9ccd79b-dxrd6:~$ 
exit