Support a handler for checking connection status using Ping frame in HTTP/2 #3612

raccoonback · 2025-02-02T03:57:11Z

Description

In some cases, HTTP/2 connections may remain open even when they are no longer functional. By introducing a periodic Ping frame health check, we can proactively detect and close unhealthy connections.

The Http2ConnectionLivenessHandler schedules and sends Ping frames at a user-defined interval to verify the connection's liveness. If an acknowledgment (ACK) is not received within the specified time, the connection is considered unhealthy and is closed automatically.

However, if other frames are actively being sent or received, the scheduler does not send a Ping frame. This is because the server may delay ACK responses for various reasons. To prevent unnecessary connection termination, Ping frames are only sent when no read or write activity is detected.

Additionally, a configurable retry threshold for Ping frame transmission has been introduced. If a Ping frame fails to receive an ACK response, it will be retried up to the specified threshold before considering the connection unhealthy. This allows fine-tuning of the failure detection mechanism, balancing between aggressive failure detection and avoiding premature disconnections.

Scheduler Flow

Ping ACK Flow

Key Changes

Added Http2ConnectionLivenessHandler handler to check connection health with Ping frames at a configurable interval.
Introduced a retry threshold setting to limit the number of Ping transmission attempts before marking the connection as unhealthy.
Specify the ping frame ack allowable range through Http2SettingsSpec.

HttpClient.create()
	.protocol(HttpProtocol.H2)
	.secure()            
	.http2Settings(
		 builder -> builder.pingAckTimeout(Duration.ofMillis(600))  // Max wait time for ACK
			.pingScheduleInterval(Duration.ofMillis(300))  // Interval for sending PING frames
			.pingAckDropThreshold(2)  // Maximum retries before considering the connection dead
         );

Related Issue

Fix HTTP/2 PING frame handling #3301

raccoonback · 2025-02-08T05:52:25Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		 * @param pingAckTimeout the interval in between consecutive PING frames
+		 *                       and ACK responses. Must be a positive value.
+		 */
+		default Builder pingAckTimeout(Duration pingAckTimeout) {


Sets the interval for sending HTTP/2 PING frames and receiving ACK responses.

raccoonback · 2025-02-08T05:52:38Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		default Builder pingScheduleInterval(Duration pingScheduleInterval) {
+			return this;
+		}


Sets the execution interval for the scheduler that sends HTTP/2 PING frames

raccoonback · 2025-02-08T05:52:56Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		default Builder pingAckDropThreshold(Integer pingAckDropThreshold) {
+			return this;
+		}


Sets the threshold for retrying HTTP/2 PING frame transmissions. (default. 0)

raccoonback · 2025-02-08T05:53:44Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		/**
+		 * Sets the interval for sending HTTP/2 PING frames and receiving ACK responses.
+		 *
+		 * <p>
+		 * This method configures the time interval at which PING frames are sent to the peer.
+		 * The interval should be chosen carefully to balance between detecting connection issues
+		 * and minimizing unnecessary network traffic.
+		 * </p>
+		 *
+		 * <p>
+		 * If the interval is set too short, it may cause excessive network overhead.
+		 * If set too long, connection failures may not be detected in a timely manner.
+		 * </p>
+		 *
+		 * @param pingAckTimeout the interval in between consecutive PING frames
+		 *                       and ACK responses. Must be a positive value.
+		 */
+		default Builder pingAckTimeout(Duration pingAckTimeout) {
+			return this;
+		}
+
+		/**
+		 * Sets the execution interval for the scheduler that sends HTTP/2 PING frames
+		 * and periodically checks for ACK responses.
+		 *
+		 * <p>
+		 * This method configures the time interval at which the scheduler runs
+		 * to send PING frames and verify if ACK responses are received within
+		 * the expected timeframe.
+		 * Proper tuning of this interval helps in detecting connection issues
+		 * while avoiding unnecessary network overhead.
+		 * </p>
+		 *
+		 * <p>
+		 * If the interval is too short, it may increase network and CPU usage.
+		 * Conversely, setting it too long may delay the detection of connection failures.
+		 * </p>
+		 *
+		 * @param pingScheduleInterval the interval in at which the scheduler executes.
+		 *                       Must be a positive value.
+		 */
+		default Builder pingScheduleInterval(Duration pingScheduleInterval) {
+			return this;
+		}
+
+		/**
+		 * Sets the threshold for retrying HTTP/2 PING frame transmissions.
+		 *
+		 * <p>
+		 * This method defines the maximum number of attempts to send a PING frame
+		 * before considering the connection as unresponsive.
+		 * If the threshold is exceeded without receiving an ACK response,
+		 * the connection may be closed or marked as unhealthy.
+		 * </p>
+		 *
+		 * <p>
+		 * A lower threshold can detect connection failures more quickly but may lead
+		 * to premature disconnections. Conversely, a higher threshold allows more retries
+		 * but may delay failure detection.
+		 * </p>
+		 *
+		 * <p>
+		 * If this value is not specified, it defaults to 0, meaning only one attempt to send a PING frame is made without retries.
+		 * </p>
+		 *
+		 * @param pingAckDropThreshold the maximum number of PING transmission attempts.
+		 *                 Must be a positive integer.
+		 *                 The default value is 0, meaning no retries will occur and only one PING frame will be sent.
+		 */
+		default Builder pingAckDropThreshold(Integer pingAckDropThreshold) {
+			return this;
+		}


It is declared as a default method for backward compatibility.

raccoonback · 2025-02-08T05:55:15Z

reactor-netty-http/src/main/java/reactor/netty/http/client/Http2ConnectionLivenessHandler.java

+	private final long pingAckTimeoutNanos;
+	private final long pingScheduleIntervalNanos;
+	private final int pingAckDropThreshold;


This is the configuration value provided by the user.

raccoonback · 2025-02-08T06:22:04Z

reactor-netty-http/src/main/java/reactor/netty/http/client/Http2ConnectionLivenessHandler.java

+		private boolean isOutOfTimeRange() {
+			return pingAckTimeoutNanos < Math.abs(lastReceivedPingTime - lastSendingPingTime);
+		}


It determines whether the Ping ACK was received within pingAckTimeoutNanos.

raccoonback · 2025-02-08T06:24:06Z

reactor-netty-http/src/main/java/reactor/netty/http/client/Http2ConnectionLivenessHandler.java

+	private class PingWriteListener implements ChannelFutureListener {
+
+		@Override
+		public void operationComplete(ChannelFuture future) throws Exception {
+			if (future.isSuccess()) {
+				if (log.isDebugEnabled()) {
+					log.debug("Wrote PING frame to {} channel.", future.channel());
+				}
+
+				isPingAckPending = true;
+				lastSendingPingTime = System.nanoTime();
+			}
+			else if (log.isDebugEnabled()) {
+				log.debug("Failed to wrote PING frame to {} channel.", future.channel());
+			}
+		}
+	}


If the new Ping frame is successfully sent, the status is changed to 'ping awaiting', and lastSendingPingTime is updated to the current time.

raccoonback · 2025-02-08T06:25:03Z

reactor-netty-http/src/main/java/reactor/netty/http/client/HttpClientConfig.java

+		 .addBefore(NettyPipeline.ReactiveBridge, NettyPipeline.H2LivenessHandler,
+				 new Http2ConnectionLivenessHandler(codec.encoder(), pingAckTimeout, pingScheduleInterval, pingAckDropThreshold))


configure Http2ConnectionLivenessHandler.

raccoonback · 2025-02-08T06:27:08Z

...y-http/src/test/java/reactor/netty/resources/PooledConnectionProviderDefaultMetricsTest.java

+		REGISTRY.clear();
+


Although not related to the purpose of this PR, handling has been implemented for failing tests.
To prevent interference with other tests, the metrics are initialized.

raccoonback · 2025-02-08T06:29:41Z

reactor-netty-http/src/test/java/reactor/netty/http/client/HttpClientTest.java

+			Mockito.verify(spyLogger, times(0))
+					.warn(Mockito.eq("Connection pool creation limit exceeded: {} pools created, maximum expected is {}"),
+							Mockito.eq(2), Mockito.eq(1));


Although not related to the purpose of this PR, handling has been implemented for failing tests.
In PooledConnectionProvider, the connectionPoolCount is handled atomically to ensure that the log warning log.warn("Connection pool creation limit exceeded: {} pools created, maximum expected is {}") is not called multiple times.
As a result, the test outcomes have been adjusted.

violetagg · 2025-02-11T17:09:30Z

@raccoonback Can you rebase against branch 1.2.x and switch the target branch for the PR to be 1.2.x. We branched and now in main it is the next development 1.3.x while we want this feature for version 1.2.x

Signed-off-by: raccoonback <kosb15@naver.com>

… in HTTP/2 Signed-off-by: raccoonback <kosb15@naver.com>

Signed-off-by: raccoonback <kosb15@naver.com>

…e ping interval - ref. https://github.com/reactor/reactor-netty/actions/runs/13095356929/job/36537031690?pr=3612 Signed-off-by: raccoonback <kosb15@naver.com>

Signed-off-by: raccoonback <kosb15@naver.com>

- Added a method to configure the execution interval of the scheduler that sends HTTP/2 PING frames and periodically checks for ACK responses - Introduced a retry threshold setting to limit the number of PING transmission attempts before considering the connection as unresponsive - Default values: - Scheduler interval must be explicitly set - Retry threshold defaults to 0 (no retries, only one PING attempt) Signed-off-by: raccoonback <kosb15@naver.com>

raccoonback · 2025-02-11T23:50:11Z

@violetagg
Hello!
I have rebased the branch onto 1.2.x and changed the target branch of the PR to 1.2.x.

raccoonback marked this pull request as ready for review February 2, 2025 08:17

raccoonback force-pushed the issue-3301 branch 9 times, most recently from 171dfc5 to 202d036 Compare February 4, 2025 00:11

raccoonback marked this pull request as draft February 5, 2025 16:36

raccoonback force-pushed the issue-3301 branch from 18196bd to 231b27d Compare February 7, 2025 16:34

raccoonback marked this pull request as ready for review February 7, 2025 17:12

raccoonback commented Feb 8, 2025

View reviewed changes

violetagg added the type/enhancement A general enhancement label Feb 10, 2025

violetagg added this to the 1.2.4 milestone Feb 10, 2025

raccoonback added 11 commits February 12, 2025 08:46

Support configuring the Ping frame interval in the HTTP/2 protocol

1280248

Signed-off-by: raccoonback <kosb15@naver.com>

Add a handler to check connection status via Ping frame health checks…

f28be6f

… in HTTP/2 Signed-off-by: raccoonback <kosb15@naver.com>

Update doc related with HTTP/2

22ed250

Signed-off-by: raccoonback <kosb15@naver.com>

Ensure backward compatibility while adding support for configuring th…

78fad85

…e ping interval - ref. https://github.com/reactor/reactor-netty/actions/runs/13095356929/job/36537031690?pr=3612 Signed-off-by: raccoonback <kosb15@naver.com>

Fix unnecessary warnings in test code

f35fbbc

Signed-off-by: raccoonback <kosb15@naver.com>

Http2ChannelDuplexHandler -> ChannelDuplexHandler

b29231b

Signed-off-by: raccoonback <kosb15@naver.com>

Reduce the scope of access control

1688b0c

Signed-off-by: raccoonback <kosb15@naver.com>

Fix broken test code

994b52a

Signed-off-by: raccoonback <kosb15@naver.com>

Change comparison type for PING frame

ccb3a6b

Signed-off-by: raccoonback <kosb15@naver.com>

Fix broken test

92be9aa

Signed-off-by: raccoonback <kosb15@naver.com>

raccoonback force-pushed the issue-3301 branch from 231b27d to a06c842 Compare February 11, 2025 23:47

raccoonback changed the base branch from main to 1.2.x February 11, 2025 23:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

raccoonback commented Feb 2, 2025 •

edited

Loading

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

violetagg commented Feb 11, 2025

raccoonback commented Feb 11, 2025

		.addBefore(NettyPipeline.ReactiveBridge, NettyPipeline.H2LivenessHandler,
		new Http2ConnectionLivenessHandler(codec.encoder(), pingAckTimeout, pingScheduleInterval, pingAckDropThreshold))

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

Are you sure you want to change the base?

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

Conversation

raccoonback commented Feb 2, 2025 • edited Loading

Description

Scheduler Flow

Ping ACK Flow

Key Changes

Related Issue

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

violetagg commented Feb 11, 2025

raccoonback commented Feb 11, 2025

raccoonback commented Feb 2, 2025 •

edited

Loading