clush ssh issues on AWS Graviton instances

[tomdeakin]

I'm having trouble doing some low level admin on the compute nodes.

[citc@mgmt ~]$ clush -w @compute uname -r
artistic-fox-m6g-xlarge-0001: citc@artistic-fox-m6g-xlarge-0001: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
clush: artistic-fox-m6g-xlarge-0001: exited with exit code 255

Regular ssh also doesn't work:

[citc@mgmt ~]$ ssh citc@artistic-fox-m6g-xlarge-0001                                                                                                                                                        
citc@artistic-fox-m6g-xlarge-0001: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). 

The verbose output is quite long:

[citc@mgmt ~]$ ssh -v citc@artistic-fox-m6g-xlarge-0001                                                                                                                                                     
OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS  28 May 2019                                                                                                                                                             
debug1: Reading configuration data /etc/ssh/ssh_config                                                                                                                                                      
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf                                                                                                                                     
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config                                                                                                                            
debug1: configuration requests final Match pass                                                                                                                                                             
debug1: re-parsing configuration                                                                                                                                                                            
debug1: Reading configuration data /etc/ssh/ssh_config                                                                                                                                                      
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf                                                                                                                                     
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config                                                                                                                            
debug1: Connecting to artistic-fox-m6g-xlarge-0001 [10.0.106.94] port 22.                                                                                                                                   
debug1: Connection established.                                                                                                                                                                             
debug1: identity file /home/citc/.ssh/id_rsa type -1
debug1: identity file /home/citc/.ssh/id_rsa-cert type -1
debug1: identity file /home/citc/.ssh/id_dsa type -1
debug1: identity file /home/citc/.ssh/id_dsa-cert type -1
debug1: identity file /home/citc/.ssh/id_ecdsa type -1
debug1: identity file /home/citc/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/citc/.ssh/id_ed25519 type -1
debug1: identity file /home/citc/.ssh/id_ed25519-cert type -1
debug1: identity file /home/citc/.ssh/id_xmss type -1
debug1: identity file /home/citc/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to artistic-fox-m6g-xlarge-0001:22 as 'citc'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp521
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp521 SHA256:oPPUTE/WRw+1XMvTNaaaP98DdNivY3SzfDQpQ0KttJI
debug1: Host 'artistic-fox-m6g-xlarge-0001' is known and matches the ECDSA host key.
debug1: Found key in /home/citc/.ssh/known_hosts:1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/citc/.ssh/id_rsa 
debug1: Will attempt key: /home/citc/.ssh/id_dsa 
debug1: Will attempt key: /home/citc/.ssh/id_ecdsa  
debug1: Will attempt key: /home/citc/.ssh/id_ed25519 
debug1: Will attempt key: /home/citc/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic 
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug1: Next authentication method: publickey
debug1: Trying private key: /home/citc/.ssh/id_rsa
debug1: Trying private key: /home/citc/.ssh/id_dsa
debug1: Trying private key: /home/citc/.ssh/id_ecdsa
debug1: Trying private key: /home/citc/.ssh/id_ed25519
debug1: Trying private key: /home/citc/.ssh/id_xmss 
debug1: No more authentication methods to try.
citc@artistic-fox-m6g-xlarge-0001: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

[milliams]

The clush support has not been used much recently and so it's possible that there's a bug in there indeed. In general it was most useful back in the day when the cluster was static as these commands would be the main way of running admin.

Regardless, SSH as citc should work. The first thing to check is /home/citc/.ssh/authorized_keys on the management node as that files is copied into the image which runs on the compute nodes. Make sure that whatever keys the citc user has are in that file and rerun packer.

Personally, when I connect to a cluster I use SSH agent forwarding which lets me use my local keys to authenticate with the compute nodes so perhaps that could help here too if you use an SSH agent.