Issue for large number of ssh connections with DH key exchange on CentOS 8.2

[vishalsevani]

In my application I open large number of ssh connections (about 200) in parallel and then execute some commands on remote nodes.

My application runs properly on Ubuntu 20.04 with node 12.18.3.

However on CentOS 8.2 with nodejs 12.18.3, the ssh connections get timed out and also the CPU usage is constantly about 100%, so I cannot do any other activity via UI.

If I reduce the number of parallel ssh connections to about 50, then the application works on CentOS 8.2, but still the CPU is 100% when the ssh connections are going on.

Also the ssh is using DH key exchange as large number of devices in my network do not support other key exchanges. For the devices that support other key exchange algorithms like curve25519-sha256@libssh.org, my application runs properly on CentOS 8.2 as well. So the problem is with DH algorithm with large number of ssh connections on CentOS 8.2.

I can see a related issue for MAC OS here but the linked node issue is fixed, so why still the problem on CentOS 8.2 with node 12.18.3. Also just to reiterate, I don't see problem on Ubuntu 20.04 with node 12.18.3

Iam posting a code snippet to reproduce the error,

var Client = require('ssh2').Client;

function getSSHConn(ip) {

    var conn = new Client();
    conn.on('ready', function() {
        console.log('ready' + ip);
        conn.end();
    }).on('error', function(err) {
        console.log(err);
    }).connect({
        host: ip,
        username: 'admin',
        password: 'abc123'
    });
  }

const ips = [array of 200 ips]
for (var i = 0; i < ips.length; i ++) {
        getSSHConn(ips[i]);
}

I have done CPU profiling and majority of ticks are in crypto::DH::New

[C++]:
   ticks  total  nonlib   name
  18281   95.5%   97.8%  node::crypto::DiffieHellman::New(v8::FunctionCallbackInfo<v8::Value> const&)
     97    0.5%    0.5%  node::native_module::NativeModuleEnv::CompileFunction(v8::FunctionCallbackInfo<v8::Value> const&)
     89    0.5%    0.5%  node::crypto::DiffieHellman::GenerateKeys(v8::FunctionCallbackInfo<v8::Value> const&)
     33    0.2%    0.2%  void node::StreamBase::JSMethod<&node::StreamBase::WriteBuffer>(v8::FunctionCallbackInfo<v8::Value> const&)

Any comments/help/clarification I can get on the issue?

[mscdex]

but the linked node issue is fixed

I'm not sure where you got that impression, especially since the linked issue is still open. Unless node is floating some patch (which AFAICT is not the case) to undo the changes OpenSSL made to more strictly validate Diffie-Hellman parameters, the issue will continue to exist (unless OpenSSL reverts their changes at some point of course).

As you saw, using curve25519 works around this issue and is probably a better key exchange method anyway.

Just to reiterate what I said in the original issue on this, there is currently nothing this module or any other (userland) module can do to avoid this performance issue. If it's something even node core can resolve without patching OpenSSL, obviously those changes will have to be made in node core.

[vishalsevani]

I'm not sure where you got that impression, especially since the linked issue is still open. Unless node is floating some patch (which AFAICT is not the case) to undo the changes OpenSSL made to more strictly validate Diffie-Hellman parameters, the issue will continue to exist (unless OpenSSL reverts their changes at some point of course).

Yeah I got confused. There is another node issue which is closed now.

Any idea why Iam not seeing problems on Ubuntu 20.04 but only CentOS 8.2? The OpenSSL that node uses is bundled with the node or it is system wide?. I can see that on Ubuntu 20.04 I have openssl 1.1.1f while on CentOS it is openssl 1.1.1c. I will try updating the openssl on CentOS and see if it resolves the issue.

[mscdex]

Any idea why Iam not seeing problems on Ubuntu 20.04 but only CentOS 8.2?

None offhand. If you didn't compile your own node binaries, you might try downloading the source code for node 12.18.3 on both systems (CentOS 8.2 and Ubuntu 20.04), compile that version of node on both, and then run your script that uses ssh2 on both systems and see if you see the same behavior. Some Linux distributions may provide a node binary that has distribution-specific patches applied on top or they may be linking to a shared OpenSSL library, both of which could affect node's behavior.

[mscdex]

As noted, the solution going forward here is to use curve25519 for your key exchanges wherever possible. The performance regressions as previously noted are beyond ssh2's control.