.withDeadlineAfter(..) not working for us in 1.52.0 and above

[burk]

What version of gRPC-Java are you using?

1.51.3, 1.52.0 and 1.57.2

What is your environment?

Linux container in kubernetes in Google Cloud, Eclipse JDK 17

What did you expect to see?

We start up to 20 long-running gRPC requests concurrently, using a fixed thread pool. We use service.withDeadlineAfter(deadline.getSeconds(), TimeUnit.SECONDS) to time out the requests after 15 minutes. If the requests time out we split the work up into smaller requests.

In versions up to and including 1.51.3, this works fine. In versions 1.52.0 and above, the initial requests with the 15 minute deadline never time out.

What did you see instead?

Expected the requests to time out like before.

Steps to reproduce the bug

Please let me know any additional information I can provide.

Looking at the changelogs, it seems maybe this change could be related? #9626

[burk]

Oh, maybe this is the same issue as this: #10336 ? I do think we have a general retry policy in place.

[ejona86]

Yeah, my first thought was #10336. I do know we looked into it more in the past month, but didn't find the direct cause.

I'm going to assume this a duplicate of #10336, as the more we investigate the easier it appears to be hit. If you happen to disable retries and it still happens, then comment here and we can reopen. But I do think it is a safe assumption it is the same issue.

CC @larry-safran