Drain HTTP connections more gracefully at shutdown

[DaveCTurner]

Description

Today when a node receives a SIGTERM we stop the HttpServerTransport which simply closes any inbound HTTP connections, regardless of the state they're in. This means that stopping a node will result in a small spike in error responses, which means a rolling restart is not truly a "zero-downtime" operation and carries some risk.

Instead I think we should stop accepting new connections and allow some time for in-flight requests to complete before closing existing connections. We'd send reponses in this phase with the Connection: close header indicating that the connection cannot be re-used for subsequent requests.

Note that this would still be something of a best-effort thing: a client may continue to send requests after the one that received the Connection: close response (e.g. if using pipelining) or the connection may be idle (so there's no response to which to attach the Connection: close header). It would still improve things in practice tho.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[stu-elastic]

Fixed by #96363