Connection reset by server after SSH2_MSG_KEXINIT sent

[asterikx]

"OpenSSH for Windows" version
v1.0.0.0-Beta

Server OperatingSystem
Windows 10 Pro

Client OperatingSystem
MacOS High Sierra 10.13.2

What is failing
The connection is reset by the server after SSH2_MSG_KEXINIT sent by the client. I needed RSA support, that's why I switched from the built-in version (which was working) to v1.0.0.0-Beta. I followed the release notes and installation instructions.

Expected output
An established connection/SSH shell.

Actual output
Client-side:

<LOCAL MACHINE>:~ <LOCAL USER>$ ssh <SERVER USER>@<SERVER DOMAIN> -p 40544 -vvv
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/<LOCAL USER>/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <SERVER DOMAIN> port 40544.
debug1: Connection established.
debug1: identity file /Users/<LOCAL USER>/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to <SERVER DOMAIN>:40544 as '<SERVER USER>'
debug3: put_host_port: [<SERVER DOMAIN>]:40544
debug3: hostkeys_foreach: reading file "/Users/<LOCAL USER>/.ssh/known_hosts"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
Connection reset by <SERVER PUBLIC IP> port 40544
<LOCAL MACHINE>:~ <LOCAL USER>$ 

Server-side:

C:\Program Files\OpenSSH>sshd.exe -ddd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 195
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 195
debug3: __PROGRAMDATA__\\ssh/sshd_config:25 setting LogLevel DEBUG3
debug3: __PROGRAMDATA__\\ssh/sshd_config:39 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: __PROGRAMDATA__\\ssh/sshd_config:53 setting PasswordAuthentication yes
debug3: __PROGRAMDATA__\\ssh/sshd_config:78 setting Subsystem sftp      sftp-server.exe
debug1: sshd version OpenSSH_7.6, LibreSSL 2.5.3
debug1: private host key #0: ssh-rsa SHA256:K7c/7Bdh/fcE/Ldb0I0REXxJ8LWbLfMLDYgBJGboVLE
debug1: private host key #1: ssh-dss SHA256:CSRd/ulPsEqftkWTEXhkzh2mRM/AerSBwrOVHqD+1Zk
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:SYsWLu8BRugZDIC4YxVY2bPkn4drfd8CS9pNQoX2ebc
debug1: private host key #3: ssh-ed25519 SHA256:VSh1cYltJmCYeS6Xg2i6LFK6PCYzCbvMoJSfeQ0yAHA
debug1: rexec_argv[0]='sshd.exe'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 195
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
Connection from <CLIENT PUBLIC IP> port 49687 on <SERVER LOCAL IP> port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug2: fd 5 setting O_NONBLOCK
debug3: LsaLogonUser succeeded
debug3: spawning "C:\\Program Files\\OpenSSH\\sshd.exe" "-ddd" "-y"
debug3: send_rexec_state: entering fd = 4 config len 195
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug3: ssh_msg_send: type 0
debug3: ssh_msg_send: type 0
debug3: preauth child monitor started
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup

[bagajjal]

Are you running Sshd debug mode as system? If not please see the release notes

[asterikx]

Yes, I did run it as system

C:\Program Files\OpenSSH>whoami
nt authority\system

[manojampalam]

@asterikx
Thanks for reporting this. Apparently, when psexec is used with -i switch, it is causing the stderr redirection issues with sshd that's causing the unprivileged worker to terminate abruptly.

As a workaround, skip -i switch for now (I've updated wiki and trouble shooting pages accordingly)
psexec -s cmd

Please regnerate sshd -d output within the above SYSTEM console.

[asterikx]

Thanks @manojampalam. This indeed gives me more information on what went wrong, but still I can't figure out the causing issue.

Server-side output:

C:\Program Files\OpenSSH>sshd.exe -ddd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 168
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 168
debug3: __PROGRAMDATA__\\ssh/sshd_config:25 setting LogLevel DEBUG3
debug3: __PROGRAMDATA__\\ssh/sshd_config:39 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: __PROGRAMDATA__\\ssh/sshd_config:77 setting Subsystem sftp sftp-server.exe
debug1: sshd version OpenSSH_7.6, LibreSSL 2.5.3
debug1: private host key #0: ssh-rsa SHA256:K7c/7Bdh/fcE/Ldb0I0REXxJ8LWbLfMLDYgBJGboVLE
debug1: private host key #1: ssh-dss SHA256:CSRd/ulPsEqftkWTEXhkzh2mRM/AerSBwrOVHqD+1Zk
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:SYsWLu8BRugZDIC4YxVY2bPkn4drfd8CS9pNQoX2ebc
debug1: private host key #3: ssh-ed25519 SHA256:VSh1cYltJmCYeS6Xg2i6LFK6PCYzCbvMoJSfeQ0yAHA
debug1: rexec_argv[0]='sshd.exe'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 168
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
Connection from <CLIENT PUBLIC IP> port 49833 on <SERVER LOCAL IP> port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug2: fd 5 setting O_NONBLOCK
unable to generate token for user sshd
unable to generate token on 2nd attempt for user sshd
unable to get security token for user sshd
posix_spawn failed
debug3: send_rexec_state: entering fd = 4 config len 168
debug3: ssh_msg_send: type 0
debug3: write ERROR from cb(2):232, io:000001CDDCA18640
ssh_msg_send: write
send_rexec_state: ssh_msg_send failed
debug1: do_cleanup

[manojampalam]

In this case, sshd (privilege separation account) token couldn't be generated.

I would double check that you are running as SYSTEM.

If yes, please check Troubleshooting wiki and get debug3 traces from sshd service

[asterikx]

I was, checked it by running whoami just beofre `sshd.exe -ddd``

But anyways, I solved my issues, by removing C: \Program Files\OpenSSH and reinstalling. This time I extracted the archive via the Powershell (as admin). (Previously I did it via the GUI and as the currently logged in user. I adjusted the permissions to only include SYSTEM and the Administrators group (FullControl) but maybe they were corrupted nevertheless)

For future reference, here all the commands I run for removing the current installation and reinstalling (assuming the downloaded archive is located at D:\Downlaods):

powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\uninstall-sshd.ps1'
del 'C:\Program Files\OpenSSH\'
Expand-Archive D:\Downloads\OpenSSH-Win64.zip -DestinationPath 'C:\Program Files\OpenSSH'
copy 'C:\Program Files\OpenSSH\OpenSSH-Win64\*' 'C:\Program Files\OpenSSH\'
del 'C:\Program Files\OpenSSH\OpenSSH-Win64\'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\install-sshd.ps1'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\install-sshd.ps1'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\FixHostFilePermissions.ps1'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\FixUserFilePermissions.ps1'
Set-Service sshd -StartupType Automatic
Set-Service ssh-agent -StartupType Automatic

After that the permissions on the C: \Program Files\OpenSSH folder were:

icacls 'C:\Program Files\OpenSSH\'
      . NT SERVICE\TrustedInstaller:(I)(F)
        NT SERVICE\TrustedInstaller:(I)(CI)(IO)(F)
        NT AUTHORITY\SYSTEM:(I)(F)
        NT AUTHORITY\SYSTEM:(I)(OI)(CI)(IO)(F)
        BUILTIN\Administrators:(I)(F)
        BUILTIN\Administrators:(I)(OI)(CI)(IO)(F)
        BUILTIN\Users:(I)(RX)
        BUILTIN\Users:(I)(OI)(CI)(IO)(GR,GE)
        CREATOR OWNER:(I)(OI)(CI)(IO)(F)
        APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(RX)
        APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(OI)(CI)(IO)(GR,GE)
        APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(RX)
        APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(OI)(CI)(IO)(GR,GE)

@manojampalam Maybe update the installation instructions? Previously I removed all permissions except for NT AUTHORITY\SYSTEM and BUILTIN\Administrators which were set to FullControl. I assume it's there where I messed it up.

[ronindesign]

Just wasted 2 hours on this.

[tecfu]

Same problem. Emphasis on this issue in the installation docs recommended.

[manojampalam]

Sorry about that. I've added a reference to this thread in the troubleshooting page. https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps

[clintoncampbell]

Having just lost almost 6 hours over two days on this issue (and having overlooked the small note on the troubleshooting page), I'm wondering if there is more to be done than highlight in the Troubleshooting-Steps. Ideally this problem could be detected and fixed automatically via the Install script or FixHostFilePermissions.ps1 script?

Likewise, some indication of the problem in debug-level logging would be helpful. In my case, the DEBUG3 logs showed no indication of anything unusual when sshd was run as a service but connections would immediately reset. Likewise, the problem wouldn't manifest when running sshd directly in debug mode.

The crucial difference? My original script expanded the archive and moved the "OpenSSH-Win64" directory that had been at the root of the archive into "Program Files". This results in entirely different permissions than when I expand the archive into a destination directory in "Program Files" and then move the contents of the "OpenSSH-Win64" subdirectory up to the parent.

[asheroto]

Thank you!!!!!!! I should have ran sshd.exe -ddd a long time ago.

Told me the problem right there.

[RT-Tap]

So I'm having this issue ... except a bit different.
When sshd is run as user I can login as that user however if I try to run as system via either Start-Service sshd or psexec -s sshd.exe -dddd I get an error code 255 (sshd.exe exited on COMPUTER with error code 255.) from sshd and this is the error on client side when I run as SYSTEM using Start-Service sshd or psexec -s sshd.exe -dddd

debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: recv - from CB ERROR:10054, io:0000017BF832C3C0
Connection reset by 127.0.0.1 port 22

Any ideas? I've been banging my head for more than 6 hours and I have no idea what's going on. I tried what asterikx mentioned ( #1027 (comment) ) but no luck.

EDIT:
Got it to work - it was a permissions error. For anyone that has a similar issue, I copied the permissions that you get on C:\windows\System32\OpenSSH when installing using the official method proposed by Microsoft .
icacls 'C:\Users\...\portable\OpenSSH'

C:\Users\...\portable\OpenSSH
                                        NT SERVICE\TrustedInstaller:(OI)(CI)(F)
                                        NT AUTHORITY\SYSTEM:(M)
                                        NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
                                        BUILTIN\Administrators:(M)
                                        BUILTIN\Administrators:(OI)(CI)(IO)(F)
                                        BUILTIN\Users:(OI)(CI)(RX)
                                        CREATOR OWNER:(OI)(CI)(IO)(F)
                                        APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(OI)(CI)(RX)

Essentially what this means is the folder needs to be owned by NT Service\TrustedInstaller and the permissions in their respective order translate to :

Full control- 		this folder and subfolders
Modify-			this folder only
Full control- 		subfolders and files only
Modify- 		this folder only
Full control- 		subfolders and files only
Read and execute- 	this folder, subfolders and files
Full control- 		subfolders and files only
Full Control- 		this folder, subfolders and files

[Orionsgate]

@RT-Tap you're a star! This saved me hours of investigation.

[danielscatigno-ncpc]

In my case, I had to disable in the sshd_config
#Match Group administrators

AuthorizedKeysFile PROGRAMDATA/ssh/administrators_authorized_keys

It was throwing an error that I discovered using sshd -ddd

[jeremyevans]

I recently ran into this issue with Windows 2019/2022 domain servers that were firewalled off from the domain controllers and found that you need to enable the dynamic windows ports (49152-65535) between the domain servers and the domain controllers or you get the same failure described in this issue.

[geometrikal]

In my case, I had to disable in the sshd_config #Match Group administrators

AuthorizedKeysFile PROGRAMDATA/ssh/administrators_authorized_keys

It was throwing an error that I discovered using sshd -ddd

This worked for me on standard Windows 11 Pro install

[Aymane11]

@asterikx 's solution (#1027 (comment)) worked for me, I ran Get-Acl C:\windows\System32\OpenSSH | Set-Acl "C:\Program Files\OpenSSH" on Powershell, to copy permissions from the OpenSSH on Sys32 to the one I installed, I also commented Match Group administrators in C:\ProgramData\ssh\sshd_config

# Match Group administrators
#        AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys`

[icyveins7]

Just chiming in; this is still an issue in 2024 and I also wasted many days til I found this thread. Probably should put this up on the README in a FAQ list.

In my case (Win Server 2016), I similarly had an issue when unpacking to a non C:\Program Files destination at first (originally unpacking to a place in 'D:'). This resulted in the 'Trusted Installers' permissions never getting set (not even by the two 'Fix...Permissions.ps1'). Giving up and unpacking directly into C:\Program Files\OpenSSH and then checking icacls finally got me those permissions, and everything worked.