Need more info: Mosh fails to connect with Ubuntu

[carloscabanero]

https://www.reddit.com/r/BlinkShell/comments/1477eb6/mosh_fails_immediately_to_ubuntu_box/

[ghost]

@carloscabanero I am also running into this issue (started recently, maybe after one of the recent updates?). My notes:

• Box is running Debian 11
• No changes have been made to the server I am trying to connect to within the last month
• Mosh is version 1.4 on the server
• I am able to mosh into this server just fine from multiple other boxes (all running mosh 1.4)

From that reddit post:

• Adding -T does not solve it
• Output of requested command:

blink> ssh -vvvv -o compression=no -t mainbox -- mosh-server new -s -c 256 -l LC_ALL=en_US.UTF-8
Setting up connection callbacks.
Starting connection to mainbox
ssh_connect: libssh 0.9.3 (c) 2003-2019 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
ssh_connect: Socket connecting, now waiting for the callbacks to work
ssh_connect: current state : 1
ssh_socket_exception_callback: Socket exception callback: 2 (2)
ssh_socket_exception_callback: Socket error: No such file or directory
connError(msg: "Session Exception - Socket error: No such file or directory")
Starting connection to mainbox
ssh_connect: current state : 9
Error connecting to mainbox. connError(msg: "Socket error: No such file or directory")
Session deinit

[ghost]

Just digging into things that changed recently... seems like it has something to do with --experimental-remote-ip (no idea what this does or why the addition of it would break being able to connect at all for me after using blink for YEARS). If I set --experimental-remote-ip=remote it connects just fine.

[ghost]

AHH, more info: maybe ipv6 related based on:

• mosh -I mainbox blerg@url-of-server.com -> fails to connect
• mosh -I mainbox blerg@ipv4-address-of-server -> connects successfully

The failed to connect message I get shows the ipv4 and ipv6 of my server, weirdly:

blink> mosh -I mainbox blerg@url-of-server.com
Connected to MY_IPV6_SERVER_ADDRESS



mosh did not make a successful connection to MY_IPV4_SERVER_ADDRESS:60001.
Please verify that UDP port 60001 is not firewalled and can reach the server.

(By default, mosh uses a UDP port between 60000 and 61000. The -p option
                                                                        selects a specific UDP port number.)

[carloscabanero]

OOOHHH, alright, I think we may have it. So I did not break anything, the "default" for IP resolution is still the same. BUT, I do have an idea of what may be going on since working on this. Do you have any Host configuration there as well?

"Remote" resolution should work for you in the meantime (and you are now able to set that as default for the Host in the UI). Remote will perform resolution at the "remote" side (d'oh), and it will go with its preferred method. "Local" resolution will use the Host configuration (this may be a solution as well, although it is more intended for ProxyCommand). The "default" is doing a resolution over SSH, and I think that may be where the confusion is coming from.

[ghost]

OOOHHH, alright, I think we may have it. So I did not break anything, the "default" for IP resolution is still the same. BUT, I do have an idea of what may be going on since working on this. Do you have any Host configuration there as well?

"Remote" resolution should work for you in the meantime (and you are now able to set that as default for the Host in the UI). Remote will perform resolution at the "remote" side (d'oh), and it will go with its preferred method. "Local" resolution will use the Host configuration (this may be a solution as well, although it is more intended for ProxyCommand). The "default" is doing a resolution over SSH, and I think that may be where the confusion is coming from.

Nothing in the Host configuration (I do not currently use that Blink feature, but probably will now to avoid having to type out the --experimental-remote-ip=remote bit all the time).

[gcv]

I recently started having problems with Blink and mosh also. Connecting through ssh works as it always has. Connecting through mosh only works if I bypass Host configuration and provide the full connection string (--server=/path/to/mosh-server user@hostname.domain etc.) or I do --experimental-remote-ip=remote (but then I can use a named Host configuration).

There’s no IPv6 involved, and the failure happens even with hosts on the local LAN with local DNS.

Error: Could not resolve address locally for host.

[carloscabanero]

I recently started having problems with Blink and mosh also. Connecting through ssh works as it always has. Connecting through mosh only works if I bypass Host configuration and provide the full connection string (--server=/path/to/mosh-server user@hostname.domain etc.) or I do --experimental-remote-ip=remote (but then I can use a named Host configuration).

Not sure this is related, but I will review the default flags when starting the ssh session. So sometimes mosh-server will not be in the path runs the command. This happens in macOS with homebrew. You should still be able to configure the mosh-server path on the Host configuration though.

There’s no IPv6 involved, and the failure happens even with hosts on the local LAN with local DNS.

Error: Could not resolve address locally for host.

For this second case, can you post here the verbose output?

[gcv]

You should still be able to configure the mosh-server path on the Host configuration though.

That’s right. The Mosh server path is set in the Host configuration. I’m saying that it doesn’t work unless I also say --experimental-remote-ip=remote.

verbose output

blink> mosh --verbose ws1-lan
MoshClient:DEBUG:".local/bin/mosh-server" new -s -c 256 -l LC_ALL=en_US.UTF-8
MoshClient:DEBUG:ssh -v -o compression=no -t ws1-lan -- ".local/bin/mosh-server" new -s -c 256 -l LC_ALL=en_US.UTF-8
socket_callback_connected: Socket connection callback: 1 (0)
ssh_known_hosts_read_entries: Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,password
ssh_agent_get_ident_count: Answer type: 12, expected answer: 12
ssh_packet_global_request: Invalid SSH_MSG_GLOBAL_REQUEST packet
channel_request: Channel request env failed
MoshClient:DEBUG:getaddrinfo failed with code: 8
Could not resolve address locally for host.

ws1-lan is a host alias configured in Hosts. This, OTOH, works:

blink> mosh --verbose --experimental-remote-ip=remote ws1-lan
MoshClient:DEBUG:".local/bin/mosh-server" new -s -c 256 -l LC_ALL=en_US.UTF-8
MoshClient:DEBUG:ssh -v -o compression=no -t ws1-lan -- echo "MOSH SSH_CONNECTION $SSH_CONNECTION" && ".local/bin/mosh-server" new -s -c 256 -l LC_ALL=en_US.UTF-8
socket_callback_connected: Socket connection callback: 1 (0)
ssh_known_hosts_read_entries: Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,password
ssh_agent_get_ident_count: Answer type: 12, expected answer: 12
ssh_packet_global_request: Invalid SSH_MSG_GLOBAL_REQUEST packet
channel_request: Channel request env failed
Connected to 10.0.1.11

[yury]

Ok, found reproducible use case.

This affects users with old hosts configurations.
In order to simulate that I disabled (temporary) deserialization for that field. Just commented that line

blink/BlinkConfig/BKHosts.m

Line 69 in d011b98

_moshExperimentalIP = [coder decodeObjectOfClasses:numbers forKey:@"moshExperimentalIP"];

Following code in that case defaults to nil value:

blink/Sessions/MoshSession.m

Lines 348 to 349 in de72f99

	NSString *experimentalIP = host.moshExperimentalIP ? experimentalIPStrings[host.moshExperimentalIP] : nil;
	self.sessionParams.experimentalRemoteIp = self.sessionParams.experimentalRemoteIp ?: experimentalIP;

Next this check failed bc self.sessionParams.experimentalRemoteIp is nil

blink/Sessions/MoshSession.m

Lines 430 to 434 in d011b98

	} else if ([@"default" isEqual:self.sessionParams.experimentalRemoteIp]) {
	ipPattern = @"Connected to (\\S*)$";
	ipMatchIdx = 1;
	ipFormat = [NSRegularExpression regularExpressionWithPattern:ipPattern options:0 error:nil];
	}

This code with fix

blink/Sessions/MoshSession.m

Lines 348 to 349 in d011b98

	NSString *experimentalIP = host.moshExperimentalIP ? experimentalIPStrings[host.moshExperimentalIP] : @"default";
	self.sessionParams.experimentalRemoteIp = self.sessionParams.experimentalRemoteIp ?: experimentalIP;

This code runs after parsing ssh output

blink/Sessions/MoshSession.m

Lines 262 to 265 in de72f99

	self.sessionParams.experimentalRemoteIp = self.sessionParams.experimentalRemoteIp ?: @"default";
	if (![@[@"default", @"remote", @"local"] containsObject:self.sessionParams.experimentalRemoteIp]) {
	return [self dieMsg:@"Unknown experimental IP mode. Use one of: default, remote, local"];
	}

[yury]

Ok, closing this issue.
Feel free to reopen if needed.