Can't connect through jumpbox

[Stephen9s]

Some users may leverage ~/.ssh/config to set up ProxyJump and ProxyCommands to get to their endpoints. Currently, it appears that VSCode Remote only supports a direct connection.

[tralston]

I'm able to connect to a JumpHost endpoint. Mine are configured in my config file. The only thing I had to do was copy my SSH key to the remote endpoint (and jump hosts). For example, if my config was:
local -> jump1 -> jump2 -> server
Then I'd have to run "ssh-copy-id" for jump1, then for jump2, then for server. After that, VSCode connected just fine.

[Stephen9s]

I am able to do the same, but what I found was that I could do the following:

local -> server = OK
local -> jump1 -> server = able to authenticate, but unable to connect.

I do know that when VSC connects to the remote server, it attempts to install or run a service. My suspicion is that by jumping through from X to Y to Z, the service ends up installing on Y and not X. Perhaps maybe there’s a better way to pivot using JumpHost as you said rather than ProxyCommand.

I’ll give ProxyJump a shot.

[tralston]

@TheStoof for what it's worth, here's an excerpt from my config file (sanitized):

host jump1-alias
  HostName jump1
  User myUser
  Port 2202

host server-alias
  HostName server
  User myUser
  Port 2202
  ProxyCommand ssh -W %h:%p jump1-alias

[tralston]

@TheStoof I just attempted the same connection, but from a Windows 10 machine with WSL, and it didn't work (same problem as #34). It did work though from my Mac.

[msalvaris]

Yea, tried various options and nothing succesful to far. I was able to connect once by creating a tunnel in WSL and pointing the Hostname and Port to the tunnel but wasn't able to repeat.

[pd93]

Ran into this issue while trialing Remote SSH as an alternative to local development. Connecting from a Windows 10 machine to a remote box works, but only when connecting directly. We cannot connect to any hosts via a jumpbox. In other words:

• local -> jumpbox works
• local -> jumpbox -> remote doesn't work

Here's my (sanitized) config:

Host jumpbox
    HostName    12.34.56.78
    Port        22
    User        user

Host remote
    Hostname    172.16.0.1
    Port        22
    User        user
    ProxyJump   jumpbox

There is no way for us to SSH into a remote machine without tunneling through a jumpbox, so this functionality it absolutely necessary for us to use Remote - SSH.

[LeuisKen]

I'm trying to solve this issue by writing a new RemoteAuthorityResolver, which the Remote SSH and Remote Container based on. But there is no document about this API...

[suanrong]

Anyone solve this issue?

[msalvaris]

@suanrong

Anyone solve this issue?

I was able to overcome the issue by using this method
#117

I create the proxy in wsl and simply make sure that my ssh config points to localhost and the correct port.

[pd93]

I decided to try SSHing (with the same config as above) to my remote boxes via Powershell today. This should use the same version of SSH as the Remote - SSH extension (rather than the WSL version of SSH).

Unsurprisingly, I got exactly the same result, but a better error message

• local -> jumpbox works
• local -> jumpbox -> remote fails with the error:

CreateProcessW failed error:2
posix_spawn: No such file or directory

A quick search of the error found PowerShell/Win32-OpenSSH#1185.
Not 100% sure if related, but that issue was fixed in Jan which implies that if that issue is the problem, this would be fixed in 1903. Unfortunately, we're still on 1809 at work, so I can't test.

A bit more searching also found PowerShell/Win32-OpenSSH#1172. This issue seems more relevant than 1185, but is still open 😢

[axiqia]

I have copyed my SSH key to the remote endpoint and jump hosts as @tralston suggested. But I still met the issue like @pd93 . I added more information.

My config:

Host jumpbox
    HostName example.edu.cn
    User U
    Port xxx
    IdentityFile xxx.id_rsa


Host remote
    HostName remote
    User U
    Port xx
    ProxyCommand ssh -q -W %h:%p jumpbox

On Git Bash(windows)

$ ssh -V                                                                         
OpenSSH_7.9p1, OpenSSL 1.1.1a  20 Nov 2018

• local -> jumpbox works

ssh jumpbox

• local -> jumpbox -> remote works

ssh remote

On Powershell:

ssh -V
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5

• local -> jumpbox works

ssh jumpbox

• local -> jumpbox -> remote fails with the error:

ssh remote
CreateProcessW failed error:2
posix_spawn: No such file or directory

On VS Code:

• local -> jumpbox works
• local -> jumpbox -> remote fails with the error:

Picking SSH host
Selected remote
Confirming that remote is a valid reachable host
Running type "C:\Users\xxx\AppData\Local\Temp\vscode-linux-multi-line-command-remote.sh" | ssh  "remote" bash to confirm the host platform
> 
> CreateProcessW failed error:2
> posix_spawn: No such file or directory
> 
"uname" terminal command done
remote: unreachable or not Linux x86_64. (posix_spawn: No such file or directory)

[johnymachine]

Maybe this could help:

ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jumphost netcat -w 120 %h %p

https://serverfault.com/questions/956613/windows-10-ssh-proxycommand-posix-spawn-no-such-file-or-directory

It worked as expected on Windows 10 1809 but there is probably bug in 1903 and this worked for me.

[davesdig]

Seeing same problem here when trying to connect to Centos 7 via a Centos 7 jump box from a mac. Tried several forms of ProxyCommand and ProxyJump. Can connect thru proxy to target from command line so the configuration should be good. Keys are set up on both jump box and target box.

Code Version: 1.35.0
Extension Version: 0.42.0

[Higher-Stark]

I encountered CreateProcessW failed error:2 posix_spawn: No such file or directory on my Windows 10 1903. The ProxyCommand in my config is ProxyCommand ssh jumphost -W %h:%p.
Then I change the command to ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jumphost -W %h:%p, and it just works.

@johnymachine I tried yours, but I got bash: netcat: command not found.
I am not going to look into the error, I'd appreciate it if you let me know the why this happens.

OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Windows 10 - 1903

[johnymachine]

I encountered CreateProcessW failed error:2 posix_spawn: No such file or directory on my Windows 10 1903. The ProxyCommand in my config is ProxyCommand ssh jumphost -W %h:%p.
Then I change the command to ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jumphost -W %h:%p, and it just works.

@johnymachine I tried yours, but I got bash: netcat: command not found.
I am not going to look into the error, I'd appreciate it if you let me know the why this happens.

OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Windows 10 - 1903

Sorry my bad, I probably copied bad command.

My config is exactly this and it works:
ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p -q user@ipaddr

[axiqia]

ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p -q user@ipaddr
@johnymachine Thank you. That works for me on my windows platform.

[asmitde]

@johnymachine Thanks for the solution! Do you know how to how to run the proxy ssh process in the background? There's a blank ssh process window that stays open as long as the connection is active.

[johnymachine]

Well 😀

Here Is my question on stackoverflow:

https://stackoverflow.com/questions/56477585/can-i-somehow-suppress-empty-window-with-ssh-exe-when-using-remote-development-e

No luck so far...

[pd93]

Thought I'd post a little update to this issue for those who run into it.

The root cause of this issue is a bug with OpenSSH for windows. However, this has now been patched in the latest release (v8.0.0.0) (v8.1.0.0).

Updates:

• Unfortunately, this likely won't get built into Windows until 1909 comes around.
• Seems like this didn't get into 1909. Waiting on news for when this patch might be available in a Windows release.
• We can expect to see the latest version of OpenSSH (v8.1) included in Windows 10 for the July 2020 release (see comment). However, in the mean time, you can manually patch it yourself!
• This comment has now been further clarified. This fix was not included in the Windows 2004 update, but will be released this fall.
• MSFT has created a pinned issue explaining why OpenSSH was not included in the 20H2 release. It now looks to be released in a servicing update. Please see the linked issue for further info.

Manual Fix

• Download the v8.1.0.0 release .zip.
• Extract the .zip file
• Copy and replace the extracted ssh.exe binary into your C:\Windows\System32\OpenSSH folder.
• Done! You can now use "Remote Development - SSH" with ProxyJump in the expected way.

If you get a permissions error then this may be because there is system protection in place on the existing ssh.exe binary. This can be removed by setting yourself as the owner of the file (requires admin) and granting yourself "full control". To do this, follow these steps:

• Right-click on the existing file C:/Windows/System32/OpenSSH/ssh.exe and select Properties.
• Open the Security tab and click on Advanced.
• Click the button to change the owner and change it from TrustedOwner to your Windows username.
• Type your username and click the Check Names button (This should autofill with your full username/email).
• Apply the settings and then close the properties window and reopen it.
• Navigate back to the Security tab and click on Edit.
• Select the Users group and give the group Full Control.
• Click Apply and exit.
• Now go back and try patching the file again.

[JustinGrote]

If I turn off "Use Local Server" it gets further and installs the server and starts it, and I can see it running as node on the remote system, however, it just hangs here and never connects. If I turn off "Use Local Server" it doesn't get past the remote port parsing again.

vscode-server already installed. Skipping download...
vscode-server with 3943d614597f9f7c384846613d417e97ea6d5ab4 is already running.
61359013d8ae: start
agentPort==2728==
osReleaseId==windows==
osVersion==6.3.9600==
arch==x64==
platform==windows==
61359013d8ae: end
Install script is 680, watching sshd parent 3396

[roblourens]

@JustinGrote I honestly don't know about your scenario - could you open a new issue with the full log, and some more details about what you're doing?

[JustinGrote]

@roblourens I just found my problem is #25, I'll continue it there. Doesn't appear to have anything to do with the port forwarding specifically, just coincidence.

[bagajjal]

@JustinGrote , @roblourens - If it's a OpenSSH issue then please create an issue here

[bagajjal]

Please provide the detailed repro steps isolating the vscode environment (Just using ssh client and sshd server).

[JustinGrote]

@bagajjal My Particular Issue apparently unrelated to this issue is opened as #2475. According to @roblourens it's been fixed downstream and will be in next release, so further discussion of my issue should go there :)

[thohe973]

I got it working by also sending my ssh key to the proxy.

Host random_name
  HostName <host name>
  Port 22
  User <host user>
  IdentityFile <private ssh key>
  ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -i <private ssh key> -W %h:%p -q <proxy user>@<proxy host>

[veekayr]

Below command is working

ProxyCommand C:\\Windows\\System32\\OpenSSH\\ssh.exe -i <ssh-key> -q -W %h:%p jumpbox

[holoword]

if ProxyCommand C:\\Windows\\System32\\OpenSSH\\ssh.exe -i <ssh-key> -q -W %h:%p jumpbox doesn't work, Just use another ssh client. Just like git bin bash ssh client
set Remote.ssh.path in settings as like this
C:\Program Files\Git\usr\bin\ssh.exe or any other ssh client you have

[Querela]

For me worked the following: [on Win10, w/ w/o WSL]

• starting with the same ssh configs described in https://code.visualstudio.com/blogs/2019/10/03/remote-ssh-tips-and-tricks#_proxycommand
• just replaced ssh with ssh.exe
• specifying identity files, ssh config files, netcat etc. on ProxyCommand did not work for me -- but I did not test an absolute ssh path, just added .exe

---

As for my ssh-key login:
For both hosts, I needed to specify my id_rsa_vscode file. That I needed to do regardless of proxying or not.

Host *
   IdentityFile C:\Users\<username>\.ssh\id_rsa_vscode

[...]

And I needed to copy my keys with ssh-copy-id to the remote hosts. I also needed to specify -f because I used different id_rsa files for normal ssh login in WSL and for VSCode, and without -f it would not copy the second VSCode ssh key. (That was neccessary because VSCode could not work with ssh-agent or password protected ssh identity files - as far as I know - without always typing the remote ssh passwords on each (re-)connect.)

[eromoe]

I patched windows 10 before.
After upgrading to Windows 10 2004(20H1) .

C:\Users\xxxx>ssh -vvv dc42
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug1: Reading configuration data C:\\Users\\xxxx/.ssh/config
debug1: C:\\Users\\xxxx/.ssh/config line 1: Applying options for *
debug1: C:\\Users\\xxxx/.ssh/config line 152: Applying options for dc42
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug1: Executing proxy command: exec ssh dc_entry -W dc42:22
debug3: spawning "C:\\Windows\\System32\\OpenSSH\\ssh dc_entry -W dc42:22"
CreateProcessW failed error:2
posix_spawn: No such file or directory

Is this a new but or not ?

[pd93]

@eromoe Unfortunately, that's still version 7.7 which does not contain the fix. You'll need to manually update to v8.0+ or use one of the other workarounds mentioned above.

It has been confirmed in another issue that v8.1 will be released in an update later this year, but is not included in Windows 10 2004 (20H1):

v8.1 goes into the next release (not into win10 version 2004) available during fall time.

[anji993]

@eromoe @pd93 Did you solve the problem after upgrading to Windows 10 2004 by patching it again. I meet the same problem after I upgrade to Windows 10 2004. But patching manually updating to v8.0+ does not solve this problem.

[eromoe]

@anji993 Patching does sovle my problem , may be something wrong with previllage ?

[susguzman]

For me worked the following: [on Win10, w/ w/o WSL]

• starting with the same ssh configs described in https://code.visualstudio.com/blogs/2019/10/03/remote-ssh-tips-and-tricks#_proxycommand
• just replaced ssh with ssh.exe
• specifying identity files, ssh config files, netcat etc. on ProxyCommand did not work for me -- but I did not test an absolute ssh path, just added .exe

As for my ssh-key login:
For both hosts, I needed to specify my id_rsa_vscode file. That I needed to do regardless of proxying or not.

Host *
   IdentityFile C:\Users\<username>\.ssh\id_rsa_vscode

[...]

And I needed to copy my keys with ssh-copy-id to the remote hosts. I also needed to specify -f because I used different id_rsa files for normal ssh login in WSL and for VSCode, and without -f it would not copy the second VSCode ssh key. (That was neccessary because VSCode could not work with ssh-agent or password protected ssh identity files - as far as I know - without always typing the remote ssh passwords on each (re-)connect.)

It worked for me :)
Thank you

[nicolaipre]

FWIW: For anyone wanting password authentication only, this config works on Linux for jump-hosts:

Host <alias>
    HostName <hostname/ip-addr>
    Port 22
    User <user>
    ProxyCommand ssh -W %h:%p -q <user>@<jump-host-ip>

[tysonite]

Have anyone seen #3857 that with jump server too?

[pd93]

A quick update for anyone following this issue. MSFT currently have a pinned issue explaining why OpenSSH was not included in the 20H2 release. I've updated my workaround comment above to reflect this.

[plevold]

The manual fix by @pd93 worked like a charm for me, but a Windows update seems to have overwritten the updated ssh.exe. Rather than potentially having to replace the executable after future updates I simply extracted the zip to a different location and added it to the systems PATH. My steps:

• Download OpenSSH v8.1.0.0 .zip
• Extract to a suitable location, e.g. C:\bin\OpenSSH-Win64
• Open Edit the system environmental variables (found in the control panel or start menu)
• Edit the Path variable and add C:\bin\OpenSSH-Win64
• Make sure that the new entry is above the entry %SYSTEMROOT%\System32\OpenSSH\
• Restart VSCode

Note that I had to edit PATH for the whole system, not my user. In the latter case the entry will be added after the system version giving it precedence.

Beware that sometime in the future this solution could cause problems if the system OpenSSH version is updated to a more recent version than the manually installed one...

[matijagrcic]

Manual Fix

• Download the v8.1.0.0 release .zip.
• Extract the .zip file
• Copy and replace the extracted ssh.exe binary into your C:\Windows\System32\OpenSSH folder.
• Done! You can now use "Remote Development - SSH" with ProxyJump in the expected way.

If you get a permissions error then this may be because there is system protection in place on the existing ssh.exe binary. This can be removed by setting yourself as the owner of the file (requires admin) and granting yourself "full control". To do this, follow these steps:

• Right-click on the existing file C:/Windows/System32/OpenSSH/ssh.exe and select Properties.
• Open the Security tab and click on Advanced.
• Click the button to change the owner and change it from TrustedOwner to your Windows username.
• Type your username and click the Check Names button (This should autofill with your full username/email).
• Apply the settings and then close the properties window and reopen it.
• Navigate back to the Security tab and click on Edit.
• Select the Users group and give the group Full Control.
• Click Apply and exit.
• Now go back and try patching the file again.

Worked without any issues. I've used MobaXterm for a long time to first setup the tunnel and then just connect to the destination, but now it's not needed anymore with ProxyJump working correctly.

[AdorableYoyo]

For me worked the following: [on Win10, w/ w/o WSL]

• starting with the same ssh configs described in https://code.visualstudio.com/blogs/2019/10/03/remote-ssh-tips-and-tricks#_proxycommand
• just replaced ssh with ssh.exe
• specifying identity files, ssh config files, netcat etc. on ProxyCommand did not work for me -- but I did not test an absolute ssh path, just added .exe

As for my ssh-key login:
For both hosts, I needed to specify my id_rsa_vscode file. That I needed to do regardless of proxying or not.

Host *
   IdentityFile C:\Users\<username>\.ssh\id_rsa_vscode

[...]

And I needed to copy my keys with ssh-copy-id to the remote hosts. I also needed to specify -f because I used different id_rsa files for normal ssh login in WSL and for VSCode, and without -f it would not copy the second VSCode ssh key. (That was neccessary because VSCode could not work with ssh-agent or password protected ssh identity files - as far as I know - without always typing the remote ssh passwords on each (re-)connect.)

life saving!!! I had to log in to appreciate your share! Got mine solved! thanks!

[pd93]

@roblourens I believe this can be closed now. This was never a VSCode issue and it should now be resolved anyway as a newer (and working) version of OpenSSH is now included in the latest version of Windows.

Anyone on older versions of Windows (and therefore OpenSSH) can follow the manual steps I outlined earlier in this issue in order to update.

[roblourens]

Thanks for pointing that out @pd93