ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

[bmnizar]

RKE version: rke version v1.2.12

**Docker version: (docker version,docker info preferred)**20.10.8

Operating system and kernel: (cat /etc/os-release, uname -r preferred) Centos 7

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) Vmware

cluster.yml file:

nodes:
- address: 192.168.198.135
user: xp
role:
- controlplane
- etcd
- worker
ssh_key_path: /home/xp/.ssh/id_rsa

Steps to Reproduce:
STEP1:ssh-keygen
STEP2: rke --debug up

Results:

DEBU[0000] Checking if cluster version [1.20.10-rancher1-1] needs to have kube-api audit log enabled
DEBU[0000] Cluster version [1.20.10-rancher1-1] needs to have kube-api audit log enabled
DEBU[0000] Enabling kube-api audit log for cluster version [v1.20.10-rancher1-1]
DEBU[0000] Host: 192.168.198.135 has role: controlplane
DEBU[0000] Host: 192.168.198.135 has role: etcd
DEBU[0000] Host: 192.168.198.135 has role: worker
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [192.168.198.135]
DEBU[0000] Connecting to Docker API for host [192.168.198.135]
DEBU[0010] FIXME: Got an status-code for which error does not match any expected type!!!: -1 module=api status_code=-1
WARN[0010] Failed to set up SSH tunneling for host [192.168.198.135]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [192.168.198.135:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0010] Removing host [192.168.198.135] from node lists
FATA[0010] Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [192.168.198.135]

[superseb]

Did you copy the SSH keys to the host? Did you successfully test using SSH on the command line (If so, how), what type of SSH key is generated? Is it passphrase protected or not?

[WillNilges]

Howdy,
My co-worker and I had the same problem for two separate installations:
Single-node cluster. SSH key installed in a user account
Triple-node cluster. One master, two workers. SSH key installed in a user account

Nothing I could think of worked to fix this issue:

• SSH directory file permissions
• docker permissions
• eks config
  • ssh key at absolute path
  • ssh key at relative path
  • internal address field not filled out
  • address not in quotes

The thing that finally worked in both cases was to use an ed25519 SSH key, NOT an rsa key.

[superseb]

What OS were you using? Some don't accept RSA by default anymore.

[WillNilges]

We're using Fedora 34. Using the ssh -i command itself worked fine.

[superseb]

@WillNilges What is in the log of sshd on the server when RKE attempts to login and fails? If it works manually, it sounds like a Golang library (version) issue or something.

[WillNilges]

https://www.reddit.com/r/Fedora/comments/jhxbdh/no_ssh_public_key_auth_after_upgrade_to_fedora_33/

Figured it out. It's a Fedora issue, not an RKE issue. The sshd service was giving my co-worker these logs:

Sep 29 14:48:59 rke-master sshd[1242159]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 29 14:48:59 rke-master sshd[1242159]: Connection closed by authenticating user ezerai 10.10.1.254 port 55553 [preauth]
Sep 29 14:48:59 rke-master sshd[1242161]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 29 14:48:59 rke-master sshd[1242161]: Connection closed by authenticating user ezerai 10.10.1.254 port 55556 [preauth]
Sep 29 14:48:59 rke-master sshd[1242163]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 29 14:48:59 rke-master sshd[1242163]: Connection closed by authenticating user ezerai 10.10.1.254 port 55559 [preauth]
Sep 29 14:50:36 rke-master sshd[1243327]: Accepted password for ezerai from 10.10.1.254 port 50273 ssh2
Sep 29 14:50:36 rke-master sshd[1243327]: pam_unix(sshd:session): session opened for user ezerai(uid=1000) by (uid=0)
Sep 29 14:50:36 rke-master sshd[1243408]: Accepted password for ezerai from 10.10.1.254 port 50456 ssh2
Sep 29 14:50:36 rke-master sshd[1243408]: pam_unix(sshd:session): session opened for user ezerai(uid=1000) by (uid=0)
Sep 29 14:53:01 rke-master sshd[1245369]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 29 14:53:01 rke-master sshd[1245369]: Connection closed by authenticating user ezerai 10.10.1.254 port 54073 [preauth]
Sep 29 14:53:01 rke-master sshd[1245371]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 29 14:53:01 rke-master sshd[1245371]: Connection closed by authenticating user ezerai 10.10.1.254 port 56509 [preauth]
Sep 29 14:53:02 rke-master sshd[1245373]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 29 14:53:02 rke-master sshd[1245373]: Connection closed by authenticating user ezerai 10.10.1.254 port 56512 [preauth]

He's currently in the process of replacing his keys. Though, because he's on Windows, there's some wackyness with the way that keys are being handled. He's got an ed25519 key, and that's the one his rke config is pointed to, but for some reason, as you can see, it's defaulting to the RSA key still. I told him to yeet the key out of his SSH directory so there's no way it'll be used. Will keep you posted.

[bmnizar]

So what's the solution?
I have tried with ed25519 in vain with this config:
nodes:

• address: 192.168.198.136
  user:root
  role:
• controlplane
• etcd
• worker
  ssh_key_path: /home/xp/.ssh/id_ed25519

This is blocking for us to continue with Rancher.

[bmnizar]

@WillNilges What was your workaround to make it work?

[WillNilges]

@WillNilges What was your workaround to make it work?

Can you post your complete ignition config? I think there are several places where keys are mentioned, and you have to be sure that they're all correct.

Also post logs from your rke up attempt and your nodes' sshd service, if you have them. I'm pretty sure all we had to do was confirm we were using the new keys, and that they were correctly installed in the node user accounts.

I see that you're trying to set up with root. That could be problematic. I used a normal user account, but if you must, check your sshd configs to make sure that authentication with the root account is allowed.

[stale]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[abh23]

As @WillNilges said, changing to ed25519 method helped to get past this error.

[EtachGu]

I change RSA to ed25519 method helped to get past this error

OpenSSh version on host which run RKE is
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSh version on Node host is
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022