Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Server SSH Remoting Fails if Server has Win32_OpenSSH 7.9 or newer installed #2475

Closed
JustinGrote opened this issue Mar 4, 2020 · 38 comments
Assignees
Labels
author-verification-requested Issues potentially verifiable by issue author bug Issue identified by VS Code Team member as probable bug ssh Issue in vscode-remote SSH upstream Issue identified as 'upstream' component related (exists outside of VS Code Remote) verified Verification succeeded windows Issue on Windows
Milestone

Comments

@JustinGrote
Copy link

  • VSCode Version: Insiders
  • Local OS Version: Windows Insider
  • Remote OS Version: Windows Server 2016
  • Remote Extension/Connection Type: SSH/Docker/WSL: SSH

Steps to Reproduce:

  1. Install Windows OpenSSH 8.0 or 8.1 on remote host server
  2. Attempt remote connection
  3. Connection fails with:
[17:55:01.210] stderr> OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2
[17:55:02.159] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ubsZl1Q9SjOlzifmMqGd/w8m95rTpUtsSeRL2FJgMh4
[17:55:03.703] stderr> Authenticated to 0.tcp.ngrok.io ([3.19.114.185]:19084).
[17:55:04.411] stderr> shell request failed on channel 2
[17:55:04.423] > local-server> ssh child died, shutting down
[17:55:04.433] Local server exit: 0
[17:55:04.434] Received install output: OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ubsZl1Q9SjOlzifmMqGd/w8m95rTpUtsSeRL2FJgMh4
Authenticated to 0.tcp.ngrok.io ([3.19.114.185]:19084).
shell request failed on channel 2

[17:55:04.434] Stopped parsing output early. Remaining text: OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ubsZl1Q9SjOlzifmMqGd/w8m95rTpUtsSeRL2FJgMh4Authenticated to 0.tcp.ngrok.io ([3.19.114.185]:19084).shell request failed on channel 2
[17:55:04.435] Failed to parse remote port from server output
[17:55:04.436] Resolver error: 
[17:55:04.438] ------
  1. Install 7.7.2.0p1-Beta OpenSSH version on server
  2. Reconnect -- Server Now Works

It is noted the regression is probably in 7.9 here:
#25 (comment)

And whatever code is used to parse out the remote port from the server output needs to detect this newer format in Windows OpenSSH and parse that newer format instead.

Does this issue occur when you try this locally?: N/A
Does this issue occur when you try this locally and all extensions are disabled?: N/A

Related Info: #25
@bagajjal @manojampalam @TylerLeonhardt @roblourens

@roblourens
Copy link
Member

That's right - the fix will be in the next major windows update. No need to ping the whole team. I have assumed that most people are not installing newer versions of OpenSSH manually so I have not tried to work around it. I'm curious why you installed v8 at first instead of installing the default build through windows?

@roblourens roblourens added the info-needed Issue requires more information from poster label Mar 4, 2020
@JustinGrote
Copy link
Author

JustinGrote commented Mar 4, 2020

@roblourens Just because those were additional that Tyler flagged for visibility in the previous issue, that's all.

As to the version, we use the Kerberos GSSAPI functionality extensively which requires 7.9, and win32_openssh is easy to deploy with chocolatey :)

Is the problem a downstream issue in windows?

Is there a reason just an update to the extension can't fix the issue?
EDIT: Nevermind you answered that in your reply, obviously there are higher priorities. Not a big deal, we can downgrade to the boxes we do remote development on, it's easy enough with chocolatey.

Thanks for your hard work.

@roblourens
Copy link
Member

Thanks for trying it, and for your tweet! I will leave this issue open for now to judge whether this ends up being a common scenario. The upstream issue is PowerShell/Win32-OpenSSH#1545 (fixed internally but not in a release yet)

@roblourens roblourens self-assigned this Mar 4, 2020
@roblourens roblourens added bug Issue identified by VS Code Team member as probable bug ssh Issue in vscode-remote SSH upstream Issue identified as 'upstream' component related (exists outside of VS Code Remote) windows Issue on Windows and removed info-needed Issue requires more information from poster labels Mar 4, 2020
@JustinGrote
Copy link
Author

Thanks! Once that gets released I'll try it and if so the remediation will be to just deploy the latest openssh version out-of-band from an OS release if you need 7.9+ features.

@roblourens
Copy link
Member

For anyone else finding this issue, a workaround is to set the following

"remote.SSH.useLocalServer": false,
"remote.SSH.windowsRemotes": ["<your remote's hostname>"]

@JustinGrote
Copy link
Author

@roblourens At least for me when disabling useLocalServer (already had server in windowsRemotes), it completely hangs here and never continues. This is true for both less and greater than 7.9

vscode-server already installed. Skipping download...
vscode-server with efd7548df65de8e775015b67863190e6bd4934fc is already running.
47e446dbd685: start
agentPort==26378==
osReleaseId==windows==
osVersion==6.3.9600==
arch==x64==
platform==windows==
47e446dbd685: end
Install script is 3640, watching sshd parent 9400

@roblourens
Copy link
Member

Can you share the log? That looks like output from the connection in the terminal which should never terminate.

@JustinGrote
Copy link
Author

@roblourens which log specifically? Do you have a bug report guide?

@roblourens
Copy link
Member

The Remote-SSH output channel - I think you had the output from there above

image

@roblourens
Copy link
Member

And basically what I'm saying is, it's ok if that terminal doesn't quit but vscode should connect (you should be able to open files and so on, and the progress spinner in the lower left corner should go away)

@JustinGrote
Copy link
Author

Got it, thanks. I have a pressing item I need to address but I'll do some full testing and report back.

@JustinGrote
Copy link
Author

JustinGrote commented Mar 13, 2020

7.7p1 on Server
8.1 on VSCode (Insider) client Windows 10 Insider Slow Ring
UseLocalServer=true works fine

UseLocalServer=false has this for SSH log and it just hangs at the prompt:
image

EDIT: Should also note am accessing this server via a port-forwarded ngrok tunnel

[16:29:27.181] Log Level: 1
[16:29:27.191] remote-ssh-nightly@2020.3.13860
[16:29:27.191] win32 x64
[16:29:27.195] SSH Resolver called for "ssh-remote+probe", attempt 1
[16:29:27.195] SSH Resolver called for host: probe
[16:29:27.195] Setting up SSH remote "probe"
[16:29:27.223] Using commit id "efd7548df65de8e775015b67863190e6bd4934fc" and quality "insider" for server
[16:29:27.225] Install and start server if needed
[16:29:27.234] Checking ssh with "ssh -V"
[16:29:27.321] > OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2
[16:29:27.322] Remote command length: 7464/8192 characters
[16:29:27.323] Running script with connection command: ssh -T -o ClearAllForwardings=true probe powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -NonInteractive -Command "$e=[Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes([Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('CmVjaG8gImYwZTI5MDQ4Yjg4NjogcnVubmluZyIKJFByb2dyZXNzUHJlZmVyZW5jZSA9ICdTaWxlbnRseUNvbnRpbnVlJwokY29tbWl0SWQgPSAnZWZkNzU0OGRmNjVkZThlNzc1MDE1YjY3ODYzMTkwZTZiZDQ5MzRmYycKCiR2c2NvZGVBcmNoID0gaWYgKCgkZW52OlBST0NFU1NPUl9BUkNISVRFQ1RVUkUgLWVxICdBTUQ2NCcpIC1vciAoJGVudjpQUk9DRVNTT1JfQVJDSElURUNUVVJFIC1lcSAnSUE2NCcpKSB7ICd4NjQnIH0gZWxzZSB7ICdpYTMyJyB9Cgokc2VydmVyUm9vdCA9IChKb2luLVBhdGggKFJlc29sdmUtUGF0aCB+KSAnLnZzY29kZS1zZXJ2ZXItaW5zaWRlcnMnKQokZW52OlZTQ09ERV9BR0VOVF9GT0xERVI9JHNlcnZlclJvb3QKJGxvZ2ZpbGUgPSAiJHNlcnZlclJvb3RcLiRjb21taXRJZC5sb2ciCiRzZXJ2ZXJEaXIgPSAiJHNlcnZlclJvb3RcYmluXCRjb21taXRJZCIKJHF1YWxpdHkgPSAnaW5zaWRlcicKJHRlbGVtZXRyeSA9ICIiCiRleHRlbnNpb25zID0gIiIKJHdlYlBhcnQgPSAiIgokc2VydmVyTmFtZSA9ICJzZXJ2ZXItd2luMzItJHZzY29kZUFyY2giICsgJHdlYlBhcnQKCmZ1bmN0aW9uIGdldFNzaGRQYXJlbnRQaWQgewokY3VycmVudFBJRCA9ICRQSUQKd2hpbGUgKCRUcnVlKSB7CiRwYXJlbnRQSUQgPSAoR2V0LUNpbUluc3RhbmNlIHdpbjMyX3Byb2Nlc3MgfCA/IHByb2Nlc3NpZCAtZXEgJGN1cnJlbnRQSUQpLnBhcmVudHByb2Nlc3NpZAppZiAoISRwYXJlbnRQSUQpIHsKZWNobyAiQ291bGQgbm90IGZpbmQgYW4gc3NoZCBwYXJlbnQgb2YgdGhpcyBwcm9jZXNzIgpleGl0IDAKfQoKaWYgKChncHMgLUlkICRwYXJlbnRQSUQpLk5hbWUgLWVxICdzc2hkJykgewpyZXR1cm4gJHBhcmVudFBJRAp9CgokY3VycmVudFBJRCA9ICRwYXJlbnRQSUQKfQp9CgpmdW5jdGlvbiBleGl0SWZOZWVkZWQgewppZiAoJGxhdW5jaGVkU2VydmVyUGlkKSB7CmlmICghKGdwcyAtSWQgJGxhdW5jaGVkU2VydmVyUGlkKSkgewplY2hvICJUaGUgbGF1bmNoZWQgc2VydmVyIGRpZWQsIGV4aXRpbmciCmV4aXQgMAp9Cn0gZWxzZSB7CmlmICghKGdwcyAtSWQgJHNzaGRQSUQpKSB7CmVjaG8gIlRoZSBzc2hkIHBhcmVudCBkaWVkLCBleGl0aW5nIgpleGl0IDAKfQp9Cn0KCmZ1bmN0aW9uIERvd25sb2FkU2VydmVyIHsKZWNobyAiRG93bmxvYWRpbmcgVlMgQ29kZSBTZXJ2ZXIiCmVjaG8gJ2YwZTI5MDQ4Yjg4NiUlMSUlJwokc3BsYXQgPSBAewpVcmk9Imh0dHBzOi8vdXBkYXRlLmNvZGUudmlzdWFsc3R1ZGlvLmNvbS9jb21taXQ6JGNvbW1pdElkLyRzZXJ2ZXJOYW1lLyRxdWFsaXR5IgpUaW1lb3V0U2VjPTIwCk91dEZpbGU9IiRzZXJ2ZXJOYW1lLnppcCIKVXNlQmFzaWNQYXJzaW5nPSRUcnVlCn0KCltOZXQuU2VydmljZVBvaW50TWFuYWdlcl06OlNlY3VyaXR5UHJvdG9jb2wgPSBbTmV0LlNlcnZpY2VQb2ludE1hbmFnZXJdOjpTZWN1cml0eVByb3RvY29sIC1ib3IgW05ldC5TZWN1cml0eVByb3RvY29sVHlwZV06OlRsczEyCkludm9rZS1SZXN0TWV0aG9kIEBzcGxhdAoKJHJhbmRvbURpck5hbWUgPSBbU3lzdGVtLklPLlBhdGhdOjpHZXRSYW5kb21GaWxlTmFtZSgpCiR0bXBEZXN0ID0gIiRlbnY6VEVNUFwkcmFuZG9tRGlyTmFtZSIKZWNobyAiRXhwYW5kaW5nIHNlcnZlciBhdCAkdG1wRGVzdCIKZWNobyAnZjBlMjkwNDhiODg2JSUyJSUnCkV4cGFuZC1BcmNoaXZlICIkc2VydmVyTmFtZS56aXAiIC1EZXN0aW5hdGlvblBhdGggIiR0bXBEZXN0IgpNb3ZlLUl0ZW0gIiR0bXBEZXN0XHZzY29kZS0kc2VydmVyTmFtZVwqIiAtRGVzdGluYXRpb24gLgp9CgpmdW5jdGlvbiBEb0NsaWVudERvd25sb2FkIHsKZWNobyAiVHJpZ2dlciBjbGllbnQgc2VydmVyIGRvd25sb2FkIgplY2hvIGYwZTI5MDQ4Yjg4Njp0cmlnZ2VyX3NlcnZlcl9kb3dubG9hZAplY2hvIHBsYXRmb3JtPT13aW5kb3dzPT0KZWNobyB2c2NvZGVBcmNoPT0kdnNjb2RlQXJjaD09CmVjaG8gZGVzdEZvbGRlcj09JHNlcnZlckRpcj09CmVjaG8gZjBlMjkwNDhiODg2OnRyaWdnZXJfc2VydmVyX2Rvd25sb2FkX2VuZAoKZWNobyAiV2FpdGluZyBmb3IgY2xpZW50IHRvIHRyYW5zZmVyIHNlcnZlciBhcmNoaXZlLi4uIgplY2hvICJXYWl0aW5nIGZvciAkc2VydmVyRGlyXHZzY29kZS1zY3AtZG9uZS5mbGFnIGFuZCB2c2NvZGUtc2VydmVyLnppcCB0byBleGlzdCIKCndoaWxlKCRUcnVlKSB7CmlmKFRlc3QtUGF0aCAiJHNlcnZlckRpclx2c2NvZGUtc2NwLWRvbmUuZmxhZyIpIHsKaWYoIShUZXN0LVBhdGggIiRzZXJ2ZXJEaXJcdnNjb2RlLXNlcnZlci56aXAiKSkgewplY2hvICJGb3VuZCBmbGFnIGJ1dCBub3Qgc2VydmVyIHRhciAtIHNlcnZlciB0cmFuc2ZlciBmYWlsZWQiCmVjaG8gImYwZTI5MDQ4Yjg4NiMjMzEjIyIKZXhpdCAwCn0KCmVjaG8gIkZvdW5kIGZsYWcgYW5kIHNlcnZlciBvbiBob3N0IgpkZWwgJHNlcnZlckRpclx2c2NvZGUtc2NwLWRvbmUuZmxhZwpicmVhawp9IGVsc2UgewpTdGFydC1TbGVlcCAtU2Vjb25kcyAzCmV4aXRJZk5lZWRlZAp9Cn0KfQoKJHNzaGRQSUQgPSBnZXRTc2hkUGFyZW50UGlkCgppZighKFRlc3QtUGF0aCAkc2VydmVyRGlyKSkgewp0cnkgewokbnVsbCA9IE5ldy1JdGVtIC1JdGVtVHlwZSBEaXJlY3RvcnkgJHNlcnZlckRpciAtRm9yY2UgLUVycm9yQWN0aW9uIFNpbGVudGx5Q29udGludWUKfSBjYXRjaCB7CmVjaG8gIkNvdWxkIG5vdCBjcmVhdGUgdnNjb2RlLXNlcnZlciBkaXJlY3RvcnkuIC0gJCgkXy5Ub1N0cmluZygpKSIKcmV0dXJuCn0KCmlmKCEoVGVzdC1QYXRoICRzZXJ2ZXJEaXIpKSB7CmVjaG8gIkNvdWxkIG5vdCBjcmVhdGUgdnNjb2RlLXNlcnZlciBkaXJlY3RvcnkuIgpyZXR1cm4KfQp9CgpjZCAkc2VydmVyRGlyCgokbG9ja0ZpbGVQYXRoID0gKEpvaW4tUGF0aCAiJHNlcnZlckRpciIgInZzY29kZS1yZW1vdGUtbG9jay4kY29tbWl0SWQiKQp0cnkgewokbnVsbCA9IE5ldy1JdGVtICRsb2NrRmlsZVBhdGggLUl0ZW1UeXBlIEZpbGUgLUVycm9yQWN0aW9uIFNpbGVudGx5Q29udGludWUKfSBjYXRjaCB7CmVjaG8gIkNvdWxkIG5vdCBjcmVhdGUgdnNjb2RlLXNlcnZlciBsb2NrIGZpbGUuIC0gJCgkXy5Ub1N0cmluZygpKSIKcmV0dXJuCn0KCnRyeSB7CmVjaG8gIkFjcXVpcmluZyBsb2NrIG9uICRsb2NrRmlsZVBhdGgiCgokZmlsZSA9IFtTeXN0ZW0uaW8uRmlsZV06Ok9wZW4oJGxvY2tGaWxlUGF0aCwgJ09wZW4nLCAnUmVhZCcsICdOb25lJykKfSBjYXRjaCB7CmVjaG8gIkluc3RhbGxhdGlvbiBhbHJlYWR5IGluIHByb2dyZXNzLi4uIC0gJCgkXy5Ub1N0cmluZygpKSIKZWNobyAiZjBlMjkwNDhiODg2IyMyNCMjIgpyZXR1cm4KfQoKdHJ5IHsKZWNobyAiTG9va2luZyBmb3IgZXhpc3Rpbmcgc2VydmVyIGluICRzZXJ2ZXJEaXIiCmlmKFRlc3QtUGF0aCAiJHNlcnZlckRpclxzZXJ2ZXIuY21kIikgewplY2hvICJ2c2NvZGUtc2VydmVyIGFscmVhZHkgaW5zdGFsbGVkLiBTa2lwcGluZyBkb3dubG9hZC4uLiIKfSBlbHNlIHsKdHJ5IHsKRG93bmxvYWRTZXJ2ZXIKfSBjYXRjaCB7CmVjaG8gIkZhaWxlZCB0byBkb3dubG9hZCAmIGV4dHJhY3QgdnNjb2RlLXNlcnZlci4gLSAkKCRfLlRvU3RyaW5nKCkpIgpEb0NsaWVudERvd25sb2FkCgpyZXR1cm4KfQoKaWYoIShUZXN0LVBhdGggIiRzZXJ2ZXJEaXJcc2VydmVyLmNtZCIpKSB7CmVjaG8gIkZhaWxlZCB0byBkb3dubG9hZCAmIGV4dHJhY3QgdnNjb2RlLXNlcnZlci4iCmVjaG8gImYwZTI5MDQ4Yjg4NiMjMjUjIyIKcmV0dXJuCn0KfQoKaWYgKCRleHRlbnNpb25zIC1uZSAiIikgewplY2hvICJJbnN0YWxsaW5nIGV4dGVuc2lvbnMuLi4iCiYgIiRzZXJ2ZXJEaXJcc2VydmVyLmNtZCIgJHRlbGVtZXRyeSAgIyA/Pwp9CgppZighKEdldC1Qcm9jZXNzIG5vZGUgLUVycm9yQWN0aW9uIFNpbGVudGx5Q29udGludWUgfCBXaGVyZS1PYmplY3QgUGF0aCAtbWF0Y2ggJGNvbW1pdElkKSkgewppZihUZXN0LVBhdGggJGxvZ2ZpbGUpIHsKZGVsICRsb2dmaWxlCn0KJHNwbGF0ID0gQHsKRmlsZVBhdGggPSAicG93ZXJzaGVsbC5leGUiCldpbmRvd1N0eWxlID0gImhpZGRlbiIKQXJndW1lbnRMaXN0ID0gQCgKIi1FeGVjdXRpb25Qb2xpY3kiLCAiVW5yZXN0cmljdGVkIiwgIi1Ob0xvZ28iLCAiLU5vUHJvZmlsZSIsICItTm9uSW50ZXJhY3RpdmUiLCAiLWMiLCAiJiBgIiRzZXJ2ZXJEaXJcc2VydmVyLmNtZGAiIC0taG9zdD0xMjcuMC4wLjEgLS1lbmFibGUtcmVtb3RlLWF1dG8tc2h1dGRvd24gLS1wb3J0PTAgJHRlbGVtZXRyeSAqPiAnJGxvZ2ZpbGUnIgopClBhc3NUaHJ1ID0gJFRydWUKfQplY2hvICJTdGFydGluZyBzZXJ2ZXIgd2l0aCBjb21tYW5kLi4uICYgJyRzZXJ2ZXJEaXJcc2VydmVyLmNtZCcgLS1ob3N0PTEyNy4wLjAuMSAtLWVuYWJsZS1yZW1vdGUtYXV0by1zaHV0ZG93biAtLXBvcnQ9MCAkdGVsZW1ldHJ5ICo+ICckbG9nZmlsZSciCiRsYXVuY2hlZFNlcnZlclBpZCA9IChTdGFydC1Qcm9jZXNzIEBzcGxhdCkuSUQKfSBlbHNlIHsKZWNobyAidnNjb2RlLXNlcnZlciB3aXRoICRjb21taXRJZCBpcyBhbHJlYWR5IHJ1bm5pbmcuIgp9Cgokc3BsYXQgPSBAewpQYXRoID0gJGxvZ2ZpbGUKUGF0dGVybiA9ICJFeHRlbnNpb24gaG9zdCBhZ2VudCBsaXN0ZW5pbmcgb24gKFxkKykiCn0KCiR0aW1lb3V0RGF0ZSA9IChHZXQtRGF0ZSkuQWRkU2Vjb25kcyg0KQp3aGlsZSAoKEdldC1EYXRlKSAtbHQgJHRpbWVvdXREYXRlKSB7CmlmKFRlc3QtUGF0aCAkbG9nZmlsZSkgewokZ3JvdXBzID0gKFNlbGVjdC1TdHJpbmcgQHNwbGF0KS5NYXRjaGVzLkdyb3VwcwppZigkZ3JvdXBzKSB7CiRwb3J0ID0gJGdyb3Vwc1sxXS5WYWx1ZQpicmVhawp9Cn0KU3RhcnQtU2xlZXAgLU1pbGxpc2Vjb25kcyA1MDAKfQoKaWYgKCEkcG9ydCkgewplY2hvICJTZXJ2ZXIgZGlkIG5vdCBzdGFydCBzdWNjZXNzZnVsbHkuIEZ1bGwgc2VydmVyIGxvZyBhdCAkbG9nZmlsZSA+Pj4iCmNhdCAkbG9nZmlsZQplY2hvICI8PDwgRW5kIG9mIHNlcnZlciBsb2ciCmVjaG8gImYwZTI5MDQ4Yjg4NiMjMzIjIyIKcmV0dXJuCn0KfSBjYXRjaCB7CmVjaG8gInZzY29kZS1zZXJ2ZXIgZmFpbGVkIHRvIHN0YXJ0LiAtICQoJF8uVG9TdHJpbmcoKSkiCn0gZmluYWxseSB7CiRmaWxlLkNsb3NlKCkKfQoKCnRyeSB7CiR3aW5WZXJzaW9uID0gKEdldC1DaW1JbnN0YW5jZSBXaW4zMl9PcGVyYXRpbmdTeXN0ZW0pLlZlcnNpb24KfSBjYXRjaCB7CmVjaG8gIkZhaWxlZCB0byBmaW5kIFdpbmRvd3MgdmVyc2lvbiAtICQoJF8uVG9TdHJpbmcoKSkiCiR3aW5WZXJzaW9uID0gInVua25vd24iCn0KCmVjaG8gImYwZTI5MDQ4Yjg4Njogc3RhcnQiCgplY2hvICJhZ2VudFBvcnQ9PSRwb3J0PT0iCmVjaG8gIm9zUmVsZWFzZUlkPT13aW5kb3dzPT0iCmVjaG8gIm9zVmVyc2lvbj09JHdpblZlcnNpb249PSIKZWNobyAiYXJjaD09JHZzY29kZUFyY2g9PSIKZWNobyAicGxhdGZvcm09PXdpbmRvd3M9PSIKZWNobyAiZjBlMjkwNDhiODg2OiBlbmQiCgoKCmVjaG8gIkluc3RhbGwgc2NyaXB0IGlzICRwaWQsIHdhdGNoaW5nIHNzaGQgcGFyZW50ICRzc2hkUElEIgp3aGlsZSAoJFRydWUpIHsKZXhpdElmTmVlZGVkClN0YXJ0LVNsZWVwIDMwCn0K')))); powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -NonInteractive -EncodedCommand $e"
[16:29:27.326] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[16:29:27.471] "install" terminal received data: "�[2J�[m�[H�]0;C:\WINDOWS\System32\cmd.exe��[?25h"
[16:29:27.472] Got some output, clearing connection timeout
[16:29:27.916] "install" terminal received data: "�[25l�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K�[H�[?25h"
[16:29:31.542] "install" terminal received data: "#< CLIXML"
[16:29:31.566] "install" terminal received data: "f0e29048b886: running"
[16:29:31.814] "install" terminal received data: "Acquiring lock on C:\Users\Allied-Adm
in\.vscode-server-insiders\bin\efd754
8df65de8e775015b67863190e6bd4934fc\vs
code-remote-lock.efd7548df65de8e77501
5b67863190e6bd4934fc�[K"
[16:29:31.850] "install" terminal received data: "Looking for existing server in C:\Use
rs\Allied-Admin\.vscode-server-inside
rs\bin\efd7548df65de8e775015b67863190
e6bd4934fc�[K
vscode-server already installed. Skip
ping download...�[K
vscode-server with efd7548df65de8e775
015b67863190e6bd4934fc is already run
ning.�[K"
[16:29:31.874] "install" terminal received data: "f0e29048b886: start   
agentPort==8644==     
osReleaseId==windows==
osVersion==6.3.9600==
arch==x64==
platform==windows==
f0e29048b886: end
"
[16:29:31.875] Received install output:    
agentPort==8644==     
osReleaseId==windows==
osVersion==6.3.9600==
arch==x64==
platform==windows==

[16:29:31.876] Remote server is listening on port 8644
[16:29:31.876] Parsed server configuration: {"agentPort":8644,"osReleaseId":"windows","arch":"x64","platform":"windows"}
[16:29:31.882] Remote command length: 204/8192 characters
[16:29:31.883] Running script with connection command: ssh -T -L 127.0.0.1:14599:127.0.0.1:8644 probe powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -NonInteractive -Command "$e=[Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes([Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('ZWNobyBTU0hfQVVUSF9TT0NLPSRTU0hfQVVUSF9TT0NLICYmIGVjaG8gLWUgJ0Nvbm5lY3RlZCB0byBTU0ggSG9zdCAtIFBsZWFzZSBkbyBub3QgY2xvc2UgdGhpcyB0ZXJtaW5hbCcgJiYgd2hpbGUgdHJ1ZTsgZG8gc2xlZXAgMTgwOyBlY2hvIC1uICcgJzsgZG9uZQ==')))); powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -NonInteractive -EncodedCommand $e"
[16:29:31.884] Spawning tunnel with: ssh -T -L 127.0.0.1:14599:127.0.0.1:8644 probe powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -NonInteractive -Command "$e=[Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes([Text.Encoding]::UTF8.GetString([Convert]::FromBase64String('ZWNobyBTU0hfQVVUSF9TT0NLPSRTU0hfQVVUSF9TT0NLICYmIGVjaG8gLWUgJ0Nvbm5lY3RlZCB0byBTU0ggSG9zdCAtIFBsZWFzZSBkbyBub3QgY2xvc2UgdGhpcyB0ZXJtaW5hbCcgJiYgd2hpbGUgdHJ1ZTsgZG8gc2xlZXAgMTgwOyBlY2hvIC1uICcgJzsgZG9uZQ==')))); powershell -ExecutionPolicy Unrestricted -NoLogo -NoProfile -NonInteractive -EncodedCommand $e"
[16:29:31.884] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[16:29:31.921] "install" terminal received data: "
Install script is 7764, watching sshd�"
[16:29:31.937] "install" terminal received data: "
 parent 8164"
[16:29:32.004] "SSH Tunnel" terminal received data: "�[2J�[m�[H�]0;C:\WINDOWS\System32\cmd.exe��[?25h"
[16:29:32.430] "SSH Tunnel" terminal received data: "�[25l�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K�[H�[?25h"
[16:29:32.904] "install" terminal received data: "�[25l�[H5b67863190e6bd4934fc�[K
Looking for existing server in C:\Use
rs\Allied-Admin\.vscode-server-inside
rs\bin\efd7548df65de8e775015b67863190
e6bd4934fc�[K
vscode-server already installed. Skip
ping download...�[K
vscode-server with efd7548df65de8e775
015b67863190e6bd4934fc is already run
ning.�[K
f0e29048b886: start�[K
agentPort==8644==�[K
osReleaseId==windows==�[K
osVersion==6.3.9600==�[K
arch==x64==�[K
platform==windows==�[K
f0e29048b886: end�[K
Install script is 7764, watching sshd
 parent 8164�[K
�[K�[?25h"
[16:29:32.913] "SSH Tunnel" terminal received data: "�[25l�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K�[H�[?25h"
[16:29:36.034] "SSH Tunnel" terminal received data: "#< CLIXML"
[16:29:36.042] "SSH Tunnel" terminal received data: "<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><S S="Error">At line:1 char:35_x000D__x000A_</S><S S="
Error">+ echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK &amp;&amp; echo -e 'Connected to SSH Host - ..._x000D__x000A_</S><S S="Error">+�[K
                       ~~_x000D__x000A_</S><S S="Error">The token '&amp;&amp;' is not a valid statement separator in this version._x00
0D__x000A_</S><S S="Error">At line:1 char:106_x000D__x000A_</S><S S="Error">+ ... Connected to SSH Host - Please do not close this ter
minal' &amp;&amp; while t ..._x000D__x000A_</S><S S="Error">+                                                                ~~_x000D_
_x000A_</S><S S="Error">The token '&amp;&amp;' is not a valid statement separator in this version._x000D__x000A_</S><S S="Error">At li
ne:1 char:123_x000D__x000A_</S><S S="Error">+ ... H Host - Please do not close this terminal' &amp;&amp; while true; do sleep 1 ..._x0
00D__x000A_</S><S S="Error">+                                                                  ~_x000D__x000A_</S><S S="Error">Missing
 statement body in do loop._x000D__x000A_</S><S S="Error">    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecord
Ex _x000D__x000A_</S><S S="Error">   ception_x000D__x000A_</S><S S="Error">    + FullyQualifiedErrorId : InvalidEndOfLine_x000D__x000A
_</S><S S="Error"> _x000D__x000A_</S></Objs>�[K"
[16:29:36.479] "SSH Tunnel" terminal command done
[16:29:36.480] SSH tunnel command completed unexpectedly
[16:29:36.540] "install" terminal received data: "�[25l�[H5b67863190e6bd4934fc�[K
Looking for existing server in C:\Users\Allied-Admin\.vscode-server-insiders\bin\efd7548df65de8e775015b67863190e6bd4934fc�[K
vscode-server already installed. Skipping download...�[K
vscode-server with efd7548df65de8e775015b67863190e6bd4934fc is already running.�[K
f0e29048b886: start�[K
agentPort==8644==�[K
osReleaseId==windows==�[K
osVersion==6.3.9600==�[K
arch==x64==�[K
platform==windows==�[K
f0e29048b886: end�[K
Install script is 7764, watching sshd parent 8164�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K
�[K�[13;1H�[?25h"

@roblourens
Copy link
Member

Do you have remote.SSH.enableDynamicForwarding disabled? Too many settings... if not can you share the ssh settings you have set?

@soul4soul
Copy link

Is this issue in Windows itself or the version of ssh shipped with windows?

Do to #2512 I installed the SSH nightly extension. I have since set the extension settings back to default. On my host I have openssh installed from Windows optional feature and ssh installed from git. The VS Code SSH extension is once again finding ssh.exe which is installed with git. Using ssh installed with git appears to be the preferred approach #2512 Comment.

I'm trying to connect to a Windows 8 machine running OpenSSH_for_Windows_8.0p1, LibreSSL 2.6.5.

log
[09:29:23.764] Log Level: 2
[09:29:23.771] remote-ssh-nightly@2020.3.25802
[09:29:23.771] win32 x64
[09:29:23.773] SSH Resolver called for "ssh-remote+sl92", attempt 1
[09:29:23.773] SSH Resolver called for host: sl92
[09:29:23.773] Setting up SSH remote "sl92"
[09:29:23.776] Acquiring local install lock: C:\Users\MY_USERNAME\AppData\Local\Temp\vscode-remote-ssh-sl92-install.lock
[09:29:23.790] Looking for existing server data file at c:\Users\MY_USERNAME\AppData\Roaming\Code\User\globalStorage\ms-vscode-remote.remote-ssh-nightly\vscode-ssh-host-sl92-fe22a9645b44368865c0ba92e2fb881ff1afce94-2020.3.25802\data.json
[09:29:23.791] Using commit id "fe22a9645b44368865c0ba92e2fb881ff1afce94" and quality "stable" for server
[09:29:23.793] Install and start server if needed
[09:29:23.816] Checking ssh with "ssh -V"
[09:29:23.902] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
[09:29:23.902] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[09:29:23.943] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
[09:29:23.943] Checking ssh with "C:\Program Files\Git\usr\bin\ssh.exe -V"
[09:29:24.961] > OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
[09:29:24.964] Using SSH config file "C:\Users\MY_USERNAME\.ssh\config"
[09:29:24.964] askpass server listening on \\.\pipe\vscode-ssh-askpass-788f48975985de322806231b1d86341d8f6519f3-sock
[09:29:24.965] Spawning local server with {"ipcHandlePath":"\\\\.\\pipe\\vscode-ssh-askpass-dfd1c8c1c0c6c3cc3bfd417b6d848a2dc78e5196-sock","sshCommand":"C:\\Program Files\\Git\\usr\\bin\\ssh.exe","sshArgs":["-v","-T","-D","64423","-F","C:\\Users\\MY_USERNAME\\.ssh\\config","sl92"],"dataFilePath":"c:\\Users\\MY_USERNAME\\AppData\\Roaming\\Code\\User\\globalStorage\\ms-vscode-remote.remote-ssh-nightly\\vscode-ssh-host-sl92-fe22a9645b44368865c0ba92e2fb881ff1afce94-2020.3.25802\\data.json"}
[09:29:24.965] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"c:\\Users\\MY_USERNAME\\.vscode\\extensions\\ms-vscode-remote.remote-ssh-nightly-2020.3.25802\\out\\local-server\\askpass.bat","VSCODE_SSH_ASKPASS_NODE":"C:\\Program Files\\Microsoft VS Code\\Code.exe","VSCODE_SSH_ASKPASS_MAIN":"c:\\Users\\MY_USERNAME\\.vscode\\extensions\\ms-vscode-remote.remote-ssh-nightly-2020.3.25802\\out\\askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"\\\\.\\pipe\\vscode-ssh-askpass-788f48975985de322806231b1d86341d8f6519f3-sock"}
[09:29:24.973] Spawned 14060
[09:29:25.059] > local-server> Spawned ssh: 18248
[09:29:25.109] stderr> OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
[09:29:25.232] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:kUZusHekV+ojgPCkz/zGjmYEVntjNHw4sBTy5GlRfzI
[09:29:25.372] Got askpass request: {"request":"The authenticity of host 'sl92 (172.16.20.132)' can't be established."}
[09:29:25.373] Detected first line fingerprint confirmation message
[09:29:25.373] Showing fingerprint confirmation dialog
[09:29:25.374] Listening for interwindow password on \\.\pipe\vscode-ssh-askpass-7460cb6c2b4d3cc1c61c6a48a8632127999cbf24-sock
[09:29:25.374] Writing password prompt to globalState
[09:29:27.170] Got fingerprint response: yes
[09:29:27.170] Interactor gave response: yes
[09:29:27.170] Cleaning up other-window auth server
[09:29:27.198] stderr> Failed to add the host to the list of known hosts (/h/.ssh/known_hosts).
[09:29:27.410] Got askpass request: {"request":"engineer@sl92's password: "}
[09:29:27.410] Showing password prompt
[09:29:27.412] Listening for interwindow password on \\.\pipe\vscode-ssh-askpass-1f1b2eae9875f7c1f2d32bbdbfa78097d9a423c6-sock
[09:29:27.412] Writing password prompt to globalState
[09:29:29.293] Got password response
[09:29:29.293] Interactor gave response: ****
[09:29:29.293] Cleaning up other-window auth server
[09:29:29.332] stderr> Authenticated to sl92 ([172.16.20.132]:22).
[09:29:29.451] stderr> shell request failed on channel 2
[09:29:29.455] > local-server> ssh child died, shutting down
[09:29:29.462] Local server exit: 0
[09:29:29.463] Received install output: OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:kUZusHekV+ojgPCkz/zGjmYEVntjNHw4sBTy5GlRfzI
Failed to add the host to the list of known hosts (/h/.ssh/known_hosts).
Authenticated to sl92 ([172.16.20.132]:22).
shell request failed on channel 2

[09:29:29.463] Stopped parsing output early. Remaining text: OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019debug1: Server host key: ecdsa-sha2-nistp256 SHA256:kUZusHekV+ojgPCkz/zGjmYEVntjNHw4sBTy5GlRfzIFailed to add the host to the list of known hosts (/h/.ssh/known_hosts).Authenticated to sl92 ([172.16.20.132]:22).shell request failed on channel 2
[09:29:29.463] Failed to parse remote port from server output
[09:29:29.464] Resolver error: 
[09:29:29.466] ------

@roblourens
Copy link
Member

@soul4soul that looks like an older version of the nightly ssh extension

@JustinGrote
Copy link
Author

JustinGrote commented Oct 27, 2020

@roblourens as far as I can see this is still a problem

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:yYWMXXuJSdNDQ/lhxr/l72WDtqZGE+f53B660KXhkjs
Authenticated to xxx.xxx ([xxx.xxx.217.178]:10001).
shell request failed on channel 2

[16:29:14.800] Stopped parsing output early. Remaining text: OpenSSH_for_Windows_8.1p1, LibreSSL 2.9.2debug1: Server host key: ecdsa-sha2-nistp256 SHA256:yYxxxxx/lhxr/l72WDtqZGE+f53B660KXhkjsAuthenticated to xxx.link ([xxx.xxx.217.178]:10001).shell request failed on channel 2
[16:29:14.801] Failed to parse remote port from server output

@roblourens
Copy link
Member

roblourens commented Nov 5, 2020

Can you share the full log and the ssh settings you have? There have been a bunch of changes since March.

You may need

"remote.SSH.useLocalServer": false,
"remote.SSH.remotePlatform": {
        "<host name>": "windows"
},

if you have overridden these but the defaults should work

@github-actions
Copy link

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

@github-actions
Copy link

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

@github-actions
Copy link

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

@roblourens roblourens removed the info-needed Issue requires more information from poster label Dec 26, 2020
@spdkils
Copy link

spdkils commented Feb 5, 2021

Suffering the same fate. Reason for upgrading to the newer 8.1 build was because there is a known issues with SCP and the 7.x builds of OpenSSH.

So remote coding works on 7.x but then I can't use the server for SCP from Cisco devices.

Upgrade to 8.x, and now I can SCP, but I can't get VSCode to work with any permutation.

So.... just thought I'd drop this out here, there are good reasons to upgrade to 8 for compatibility... But then we get sunk on remote development.

@bagajjal
Copy link

bagajjal commented Feb 5, 2021

Is there an issue opened in the Win32-OpenSSH repo?
If not, please create a new issue with detailed repro steps and upload the ssh server side logs .

@JustinGrote
Copy link
Author

@bagajjal The issue is how vscode remoting interprets the shell messages, v8 sends different messages and remoting doesn't parse them correctly, so this is not an OpenSSH issue to fix but a vscode one. I would fix it myself but unfortunately the source is not open for review.

@bagajjal
Copy link

bagajjal commented Feb 5, 2021

@JustinGrote, Thanks for your quick reply.
I will check with the "Rob Lourens" internally to see why they are blocked.

@roblourens
Copy link
Member

roblourens commented Feb 5, 2021

I tested this with 8.1 and the only issue I see is the previously identified one, which only reproes when the useLocalServer setting is enabled. It works fine for me with that disabled. Could you try it again and share the log if it doesn't work?

The useLocalServer mode won't be usable as-is on Windows servers between 8.1 and 8.4. This mode is already disabled on windows clients by default because of other issues, and windows clients -> windows server is much more common than mac or linux to windows server. But I know that many users have enabled it because it brings a better experience. But on the client side we should be able to identify this particular error and fall back to reconnect with "powershell" passed as the command to SSH which avoids the issue. An impact on the user will be that they will have to type their password twice. I might write to the "remotePlatform" setting so it only happens once.

@grgsolymosi
Copy link

Probably the same issue still here.
Set all parameter off as you proposed.

`[07:01:32.330] Log Level: 2
[07:01:32.332] remote-ssh@0.64.0
[07:01:32.332] linux x64
[07:01:32.333] SSH Resolver called for "ssh-remote+199.212.215.125", attempt 1
[07:01:32.334] "remote.SSH.useLocalServer": false
[07:01:32.334] "remote.SSH.showLoginTerminal": false
[07:01:32.334] "remote.SSH.remotePlatform": {"199.212.215.125":"windows"}
[07:01:32.335] "remote.SSH.sshPath": undefined
[07:01:32.335] "remote.SSH.sshConfigurationFile": undefined
[07:01:32.335] "remote.SSH.useFlock": true
[07:01:32.335] "remote.SSH.lockfilesInTmp": false
[07:01:32.335] "remote.SSH.localServerDownload": off
[07:01:32.336] "remote.SSH.remoteServerListenOnSocket": false
[07:01:32.336] "remote.SSH.showLoginTerminal": false
[07:01:32.336] "remote.SSH.defaultExtensions": []
[07:01:32.336] SSH Resolver called for host: 199.212.215.125
[07:01:32.336] Setting up SSH remote "199.212.215.125"
[07:01:32.345] Using commit id "622cb03f7e070a9670c94bae1a45d78d7181fbd4" and quality "stable" for server
[07:01:32.348] Install and start server if needed
[07:01:32.355] Checking ssh with "ssh -V"
[07:01:32.361] > OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020

[07:01:32.364] Remote command length: 5899/8192 characters
[07:01:32.365] Running script with connection command: ssh -T -o ClearAllForwardings=true -o ConnectTimeout=15 '173.212.215.125' powershell
[07:01:32.826] > Admin@199.212.215.125's password:
[07:01:32.827] Got some output, clearing connection timeout
[07:01:32.828] Showing password prompt
[07:01:38.558] Got password response
[07:01:38.558] "install" wrote data to terminal: "****************"
[07:01:38.569] >
[07:05:49.029] > Timeout, server 199.212.215.125 not responding.
[07:05:49.290] "install" terminal command done
[07:05:49.291] Install terminal quit with output: Timeout, server 199.212.215.125 not responding.
[07:05:49.291] Received install output: Timeout, server 199.212.215.125 not responding.
[07:05:49.291] Stopped parsing output early. Remaining text: Timeout, server 199.212.215.125 not responding.
[07:05:49.291] Failed to parse remote port from server output
[07:05:49.300] Resolver error: Error:
at Function.Create (/home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:1:64126)
at Object.t.handleInstallOutput (/home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:1:62865)
at k (/home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:1:313462)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async /home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:1:311339
at async Object.t.withShowDetailsEvent (/home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:1:406337)
at async Object.t.resolve (/home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:1:315001)
at async /home/ge/.vscode/extensions/ms-vscode-remote.remote-ssh-0.64.0/out/extension.js:127:110431
[07:05:49.305] ------
`
And the terminal stalls:

image

FYI SSH works fine from console, several remotes work fine to Linux remotes.

@roblourens
Copy link
Member

I have a workaround for this which will be in tomorrow's Remote-SSH Nightly build. Would appreciate anyone who has run into this trying it out.

Essentially it will retry on the first attempt (so you may see a password prompt twice) then write the platform to the remote.SSH.remotePlatform setting it won't have to retry in the future.

@JustinGrote
Copy link
Author

@roblourens I'll give it a shot

@TylerLeonhardt
Copy link
Member

@JustinGrote did you get the chance to give this a shot?

@TylerLeonhardt TylerLeonhardt added the author-verification-requested Issues potentially verifiable by issue author label Apr 30, 2021
@JustinGrote
Copy link
Author

I think I had other issues that werent directly related. I'll put it on my to-do list, ping me in a week :)

@roblourens
Copy link
Member

Verification steps - if you have all windows updates installed, you may have SSH 7.9+ already. If not, install it from https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v8.1.0.0p1-Beta (download and run install-sshd.ps1). Try to connect with Remote-SSH. If you use password auth, you may get prompted twice. If you connect successfully, it works.

@TylerLeonhardt
Copy link
Member

image

looks like it's working. I'm marking as verified but would love other folks in this thread to give it a go

@TylerLeonhardt TylerLeonhardt added the verified Verification succeeded label May 1, 2021
@github-actions github-actions bot locked and limited conversation to collaborators May 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
author-verification-requested Issues potentially verifiable by issue author bug Issue identified by VS Code Team member as probable bug ssh Issue in vscode-remote SSH upstream Issue identified as 'upstream' component related (exists outside of VS Code Remote) verified Verification succeeded windows Issue on Windows
Projects
None yet
Development

No branches or pull requests

7 participants