kubectl get gitrepositories reports error: 'key is unknown'

[aamoroso]

Hi again :)

I've managed to progress quite far but am hitting a snag with Flux.

(NB: I am using a private GitHub repository.)

When I run task kubernetes:resources I get the following error when it runs kubectl get gitrepositories -A

task: [kubernetes:resources] kubectl get gitrepositories -A
NAMESPACE     NAME              URL                                                     AGE   READY   STATUS
flux-system   home-kubernetes   ssh://[email protected]:443/aamoroso/kubecluster.git   68m   False   failed to checkout and determine revision: unable to clone 'ssh://[email protected]:443/aamoroso/kubecluster.git': ssh: handshake failed: knownhosts: key is unknown

I have placed the public key into github as a read/write Deploy Key. The key appears to be there in secrets:

kubectl get secrets -A                                                                                                                                                               ─╯

NAMESPACE     NAME                                         TYPE                            DATA   AGE
flux-system   cluster-secrets                              Opaque                          4      74m
flux-system   github-deploy-key                            Opaque                          2      74m
flux-system   sops-age                                     Opaque                          1      74m
kube-system   bootstrap-token-bkl3uo                       bootstrap.kubernetes.io/token   4      86m
kube-system   sh.helm.release.v1.cilium.v1                 helm.sh/release.v1              1      86m
kube-system   sh.helm.release.v1.kubelet-csr-approver.v1   helm.sh/release.v1

Looking at the secrets the identity appears correct:

$ kubectl get secret -n flux-system  github-deploy-key -o jsonpath="{.data.identity}" | base64 --decode
REDACTED_BUT_CORRECT

But the known_hosts key has 3 values...

$ kubectl get secret -n flux-system  github-deploy-key -o jsonpath="{.data.known_hosts}" | base64 --decode  
                                                                           
github.com ssh-ed25519 REDACTED
github.com ecdsa-sha2-nistp256 REDACTED
github.com ssh-rsa REDACTED

github does appear to authenticate with that key:

ssh [email protected] -p 443 -i github-deploy.key                                                                                                                                   
Hi aamoroso/kubecluster! You've successfully authenticated, but GitHub does not provide shell access.
Connection to ssh.github.com closed.

Is there anything obvious I am doing wrong?

Thanks again!

Cheers,
Adrian

[onedr0p]

Something doesn't seem to be lining up here, are you sure the flux gitrepo resource has the right config, did you try reconciling it again?

[aamoroso]

Thanks for replying :)

I'm not really sure of much at this point :)

Is this the resource you mean?

kubectl describe -n flux-system gitrepositories                                                                                                                                      ─╯
Name:         home-kubernetes
Namespace:    flux-system
Labels:       <none>
Annotations:  <none>
API Version:  source.toolkit.fluxcd.io/v1
Kind:         GitRepository
Metadata:
  Creation Timestamp:  2024-04-20T03:25:05Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        1
  Resource Version:  42846
  UID:               20eb1085-0414-4c36-9b34-966661fefc2f
Spec:
  Ignore:  # exclude all
/*
# include kubernetes directory
!/kubernetes

  Interval:  30m
  Ref:
    Branch:  main
  Secret Ref:
    Name:   github-deploy-key
  Timeout:  60s
  URL:      ssh://[email protected]:443/aamoroso/kubecluster.git
Status:
  Conditions:
    Last Transition Time:  2024-04-20T06:36:39Z
    Message:               building artifact
    Observed Generation:   1
    Reason:                ProgressingWithRetry
    Status:                True
    Type:                  Reconciling
    Last Transition Time:  2024-04-20T06:36:39Z
    Message:               failed to checkout and determine revision: unable to clone 'ssh://[email protected]:443/aamoroso/kubecluster.git': ssh: handshake failed: knownhosts: key is unknown
    Observed Generation:   1
    Reason:                GitOperationFailed
    Status:                False
    Type:                  Ready
    Last Transition Time:  2024-04-20T03:25:29Z
    Message:               failed to checkout and determine revision: unable to clone 'ssh://[email protected]:443/aamoroso/kubecluster.git': ssh: handshake failed: knownhosts: key is unknown
    Observed Generation:   1
    Reason:                GitOperationFailed
    Status:                True
    Type:                  FetchFailed
  Observed Generation:     -1
Events:
  Type     Reason              Age                   From               Message
  ----     ------              ----                  ----               -------
  Warning  GitOperationFailed  64s (x23 over 3h12m)  source-controller  failed to checkout and determine revision: unable to clone 'ssh://[email protected]:443/aamoroso/kubecluster.git': ssh: handshake failed: knownhosts: key is unknown

[onedr0p]

I think you are using the wrong URL in the Flux GitRepo resource, can you try:

ssh://[email protected]/aamoroso/kubecluster

That is how I have it here.

[aamoroso]

Thanks. I will try that!

Sorry to be a pain, but... Every time Ive tried to change the git repo url Ive ended up having to nuke the cluster and start from scratch - which probably shows my utter ignorance. Is there a better way to just set the url?

[onedr0p]

Change it in your Flux GitRepository file and commit it to your repo. Then run kubectl -n flux-system edit gitrepository home-kubernetes and update it there, save it and see if it works.

[aamoroso]

That worked! Thank you so so much! <3