go.mod,net/tstun,wgengine/netstack: implement gVisor TCP GSO for Linux #12869
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1aeaeee to
08921d5
Compare
|
|
|
SNAT incremental checksum update results in an invalid (partial/pseudoheader) checksum when gVisor feeds us a single segment with NeedsCsum=true and a partial TCP checksum. SNAT could move post-GSO, though not ideal from a performance perspective. In theory incremental update should still work just fine against a psuedoheader checksum as the additive property of the checksum still applies. SNAT changes the src L3 addr, which is part of the pseudoheader checksum. 36b6a73 demonstrates the problem via unit test |
|
22ee88a simplifies the test. I don't yet understand why the additive property of checksumming isn't applying the same here. |
70ea9fa to
e192d63
Compare
|
When gVisor hands us a partial transport checksum it must be inverted (ones' complement) prior to SNAT incremental checksum update. NATPing integration tests are now passing after accounting for this, and I'm hands off. |
|
@raggi @sailorfrag this is ready for review; GRO PR will follow shortly after this one |
sailorfrag
approved these changes
Jul 30, 2024
This commit implements TCP GSO for packets being read from gVisor on Linux. Windows support will follow later. The wireguard-go dependency is updated in order to make use of newly exported GSO logic from its tun package. A new gVisor stack.LinkEndpoint implementation has been established (linkEndpoint) that is loosely modeled after its predecessor (channel.Endpoint). This new implementation supports GSO of monster TCP segments up to 64K in size, whereas channel.Endpoint only supports up to 32K. linkEndpoint will also be required for GRO, which will be implemented in a follow-on commit. TCP throughput from gVisor, i.e. TUN read direction, is dramatically improved as a result of this commit. Benchmarks show substantial improvement through a wide range of RTT and loss conditions, sometimes as high as 5x. The iperf3 results below demonstrate the effect of this commit between two Linux computers with i5-12400 CPUs. There is roughly ~13us of round trip latency between them. The first result is from commit 57856fc without TCP GSO. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.51 GBytes 2.15 Gbits/sec 154 sender [ 5] 0.00-10.00 sec 2.49 GBytes 2.14 Gbits/sec receiver The second result is from this commit with TCP GSO. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec 6 sender [ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec receiver Updates #6816 Signed-off-by: Jordan Whited <jordan@tailscale.com>
b9f812c to
f0016cc
Compare
jwhited
added a commit
that referenced
this pull request
Aug 15, 2024
…for Linux (#12869)" This reverts commit 7bc2dda. Updates tailscale/corp#22348 Signed-off-by: Jordan Whited <jordan@tailscale.com>
Asutorufa
pushed a commit
to Asutorufa/tailscale
that referenced
this pull request
Aug 23, 2024
tailscale#12869) This commit implements TCP GSO for packets being read from gVisor on Linux. Windows support will follow later. The wireguard-go dependency is updated in order to make use of newly exported GSO logic from its tun package. A new gVisor stack.LinkEndpoint implementation has been established (linkEndpoint) that is loosely modeled after its predecessor (channel.Endpoint). This new implementation supports GSO of monster TCP segments up to 64K in size, whereas channel.Endpoint only supports up to 32K. linkEndpoint will also be required for GRO, which will be implemented in a follow-on commit. TCP throughput from gVisor, i.e. TUN read direction, is dramatically improved as a result of this commit. Benchmarks show substantial improvement through a wide range of RTT and loss conditions, sometimes as high as 5x. The iperf3 results below demonstrate the effect of this commit between two Linux computers with i5-12400 CPUs. There is roughly ~13us of round trip latency between them. The first result is from commit 57856fc without TCP GSO. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 2.51 GBytes 2.15 Gbits/sec 154 sender [ 5] 0.00-10.00 sec 2.49 GBytes 2.14 Gbits/sec receiver The second result is from this commit with TCP GSO. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec 6 sender [ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec receiver Updates tailscale#6816 Signed-off-by: Jordan Whited <jordan@tailscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit implements TCP GSO for packets being read from gVisor on
Linux. Windows support will follow later. The wireguard-go dependency is
updated in order to make use of newly exported GSO logic from its tun
package.
A new gVisor stack.LinkEndpoint implementation has been established
(linkEndpoint) that is loosely modeled after its predecessor
(channel.Endpoint). This new implementation supports GSO of monster TCP
segments up to 64K in size, whereas channel.Endpoint only supports up to
32K. linkEndpoint will also be required for GRO, which will be
implemented in a follow-on commit.
TCP throughput from gVisor, i.e. TUN read direction, is dramatically
improved as a result of this commit. Benchmarks show substantial
improvement through a wide range of RTT and loss conditions, sometimes
as high as 5x.
The iperf3 results below demonstrate the effect of this commit between
two Linux computers with i5-12400 CPUs. There is roughly ~13us of round
trip latency between them.
The first result is from commit 57856fc without TCP GSO.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.51 GBytes 2.15 Gbits/sec 154 sender
[ 5] 0.00-10.00 sec 2.49 GBytes 2.14 Gbits/sec receiver
The second result is from this commit with TCP GSO.
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
Test Complete. Summary Results:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec 6 sender
[ 5] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec receiver
Updates #6816