Client http status code returned for deadline_exceeded

[DMarby]

Currently, when a server handler returns a (context) deadline exceeded error, this maps to a 408 http status code.

The 408 (Request Timeout) status code is generally used to indicate that the server did not receive a complete request message within the time that it was prepared to wait, for example if a client haven't yet sent the headers for a request.

As this is a server timeout, we should consider returning a 500 error instead, such as 504 (Gateway Timeout).

[akshayjshah]

Per status.proto from googleapis, we should be using 504 for deadline exceeded and 499 for cancelled. Let's figure out how we can make this change without breaking clients!

@chrispine for scheduling and prioritization.

[jhump]

As this is a server timeout, we should consider returning a 500 error instead

FWIW, the "deadline_exceeded" error is usually a client timeout and occurs when the RPC deadline (provided by the client) passes. Though servers are free to use it as an error code when other internal timeouts in the handler lapse, too (I'm guessing that's how you are seeing it used). The fact that it could be used for either is a frustrating aspect of gRPC error codes: their taxonomy does not attempt to clearly attribute errors to the client or the server like HTTP does (4xx vs 5xx).

Having said that, I concur with the conclusion that we should change the HTTP status code used here. A big motivation to make this change In my opinion is that the HTTP spec has very particular semantics for 408:
https://datatracker.ietf.org/doc/html/rfc7231#section-6.5.7

In particular, that last sentence:

If the client has an outstanding request in transit, the
client MAY repeat that request on a new connection.

There are clients (browsers and HTTP client libraries) that do, in practice, retry requests that fail with a 408 status, even non-idempotent requests (i.e. POSTs).