Reduce connection timeout for intra-cluster connections

[DaveCTurner]

A 30 second timeout for establishing node-to-node connections within a cluster is unreasonably long. We need longer timeouts for connections out of the cluster, so simply reducing transport.tcp.connect_timeout isn't feasible, but a separate connection profile for node-to-node connections with a separately configurable timeout would mean that attempts to connect to an unresponsive node would be able to fail much more quickly.

Relates #28920 in which cluster state application is blocked for multiple minutes while repeated attempts to connect to unresponsive nodes take place.

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

We assumed it would be simpler to change the profiles used by NodeConnectionService and the pinging but we should double check. Maybe it is better to reduce transport.tcp.connect_timeout and increase it for the cases we need (transport client & CCS come to mind)

[DaveCTurner]

Altering the profile used by the NodeConnectionsService involves a nontrivial amount of plumbing, because it needs to be based on TcpTransport#defaultConnectionProfile so that it has the correct numbers of each kind of channel. Yet, reducing all connection timeouts except for the cases (we think) we need seems like an invitation for trouble. This needs further discussion.

[DaveCTurner]

We discussed this and decided it's likely that connection management is at risk of broader changes than this, so it makes sense to park this for now. To be revisited when the dust has settled.