grpc-java is not compatible with the latest Netty version 4.1.101.Final

[idelpivnitskiy]

What version of gRPC-Java are you using?

v1.59.0, all previous versions have the same issue.

What is your environment?

macOS, but doesn't matter for this issue

What did you expect to see?

grpc-java to work with the latest Netty 4.1.101.Final at runtime.

What did you see instead?

io.grpc.StatusRuntimeException: UNKNOWN
	at app//io.grpc.Status.asRuntimeException(Status.java:539)
	at app//io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
	at app//io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
	at app//io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
	at app//io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
	at app//io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
	at app//io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at app//io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base@21.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@21.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@21.0.1/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.UnsupportedOperationException
	at io.grpc.netty.AbstractHttp2Headers.isEmpty(AbstractHttp2Headers.java:40)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:419)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:352)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:188)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

Steps to reproduce the bug

1. Bump Netty version to 4.1.101.Final.
2. Execute any test that sends a request and receives a response with trailers.

Additional details

The changeset #13603 uses headers.isEmpty() method. gprc-java has its custom implementation of Netty's Http2Headers interface, where they implement only size() but not isEmpty() method. Its parent AbstractHttp2Headers implements this method as throwing UnsupportedOperationException.

[idelpivnitskiy]

Ok, I see someone opened a similar issue, will close this one as a duplicate: #10663

[idelpivnitskiy]

Actually, that was a PR, not an issue :)
Will reopen and ask that contributor to relate them

[ejona86]

Looks like a sort of useless optimization in Netty, to avoid creating an iterator if there are no entries. But yeah, we'll want to implement the method.

[seregamorph]

Hello. Can you please release the version with this fix as it blocks the netty update from 4.1.100 to 4.1.101, which is included to todays spring-boot 2.7.18 upgrade. Same for spring-boot 3.0.13, 3.1.6, 3.2.0 - all released today.

Several notes: Yes, it's technically possible to define older version of netty with newer spring-boot, but it's not enough just to override a property netty.version in our case, as there are other sources of dependencyManagement in our project where this version is hardcoded, not resolved via ${netty.version} placeholder. It's even not enough to define import like

      <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-bom</artifactId>
        <version>${netty.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>

and only per-each-artifactId dependencyManagement will work correctly. But it's worth to mention, that netty has dozens of libraries so we'd like to avoid such verbose dependencyManagement. So the best option is to just migrate to future 1.60.0 grpc 🙏

[seregamorph]

cc @temawi , please check the comment above ^

[ejona86]

1.59.1 is released and has this as a backport.

[seregamorph]

Thank you!