Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVPN-1687: Fix memory usage #159

Merged
merged 8 commits into from
Feb 11, 2025
Merged

Conversation

kp-mariappan-ramasamy
Copy link
Contributor

@kp-mariappan-ramasamy kp-mariappan-ramasamy commented Feb 11, 2025

Description

Fixes and some tracing logs related to memory usage of Lightway server.

Motivation and Context

In prod, some of the lightway server instances were taking more memory.
This is because a lot of connections are stayed in LinkUp state and stayed there for 24 hours until age out timer kicked out.
This is unnecessary, since the connections which are not online within 1 minute will not be online.
So free those connections to save resources.

Also fixed a bug where we use two different instances of plugin for same connection for ingress and egress.

How Has This Been Tested?

Tested by using crafted client which stops after linkup and verified that server kickouts that connection after 1 miunute.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • The correct base branch is being used, if not main

So far we are tracking only TCP connection disconnects. Move this
metric to ConnectionManager::disconnect so that it will be tracked
for UDP disconnects also.
This will help to track the number of connections linked-up, but
not become online successfully.
In some cases, the connection is becoming link-up, but never become online.
Since our inactive connection age out interval is 1 day, we are accumulating lot
of inactive connections and waste memory.

Introduce a new timeout STALE_AGE (60s) and disconnect the client,
if it is not online by that time.
For outside plugins, we apply the plugins in two places.
1. In lightway_core::connection -> Ingress packets
2. In lightway_core::connection::WolfsslIoAdapter -> Egress packets

So we had to store the outside PluginList in both places.
But we were building PluginList twice from PluginFactory, instead of
using the same instance. This will case ingress and egress use different
instance of Plugin.

This commit fixes that by using the same instance in both places.
We are counting client disconnects as unknown error metric, which is
not an actual error.
Add a separate metric to track client initiated disconnects.
@kp-mariappan-ramasamy kp-mariappan-ramasamy requested a review from a team as a code owner February 11, 2025 02:32
Copy link

Code coverage summary for 09bcd3f:

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lightway-app-utils/src/args/cipher.rs                              4                 4     0.00%           1                 1     0.00%           5                 5     0.00%           0                 0         -
lightway-app-utils/src/args/connection_type.rs                     4                 4     0.00%           1                 1     0.00%           5                 5     0.00%           0                 0         -
lightway-app-utils/src/args/duration.rs                            7                 7     0.00%           5                 5     0.00%          13                13     0.00%           0                 0         -
lightway-app-utils/src/args/ip_map.rs                             11                11     0.00%           3                 3     0.00%          15                15     0.00%           0                 0         -
lightway-app-utils/src/args/logging.rs                            14                14     0.00%           2                 2     0.00%          16                16     0.00%           0                 0         -
lightway-app-utils/src/connection_ticker.rs                       39                 4    89.74%          17                 2    88.24%         156                 5    96.79%           0                 0         -
lightway-app-utils/src/dplpmtud_timer.rs                          63                 7    88.89%          22                 4    81.82%         282                11    96.10%           0                 0         -
lightway-app-utils/src/event_stream.rs                             5                 0   100.00%           3                 0   100.00%          13                 0   100.00%           0                 0         -
lightway-app-utils/src/iouring.rs                                157               142     9.55%          23                17    26.09%         306               281     8.17%           0                 0         -
lightway-app-utils/src/metrics.rs                                  2                 2     0.00%           2                 2     0.00%           4                 4     0.00%           0                 0         -
lightway-app-utils/src/sockopt/ip_mtu_discover.rs                 23                23     0.00%           4                 4     0.00%          93                93     0.00%           0                 0         -
lightway-app-utils/src/sockopt/ip_pktinfo.rs                       4                 4     0.00%           1                 1     0.00%          16                16     0.00%           0                 0         -
lightway-app-utils/src/tun.rs                                     76                76     0.00%          22                22     0.00%          94                94     0.00%           0                 0         -
lightway-app-utils/src/utils.rs                                   13                13     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
lightway-client/src/args.rs                                       32                32     0.00%          28                28     0.00%          35                35     0.00%           0                 0         -
lightway-client/src/io/inside/tun.rs                              39                39     0.00%           7                 7     0.00%          48                48     0.00%           0                 0         -
lightway-client/src/io/outside/tcp.rs                             48                48     0.00%          10                10     0.00%          45                45     0.00%           0                 0         -
lightway-client/src/io/outside/udp.rs                             60                60     0.00%          12                12     0.00%          61                61     0.00%           0                 0         -
lightway-client/src/keepalive.rs                                 245                33    86.53%          56                 8    85.71%         372                26    93.01%           0                 0         -
lightway-client/src/lib.rs                                       176               176     0.00%          19                19     0.00%         272               272     0.00%           0                 0         -
lightway-client/src/main.rs                                       48                48     0.00%           7                 7     0.00%         166               166     0.00%           0                 0         -
lightway-core/src/borrowed_bytesmut.rs                            86                 1    98.84%          24                 0   100.00%         196                 1    99.49%           0                 0         -
lightway-core/src/builder_predicates.rs                           20                10    50.00%           4                 2    50.00%          32                16    50.00%           0                 0         -
lightway-core/src/cipher.rs                                        8                 0   100.00%           3                 0   100.00%          13                 0   100.00%           0                 0         -
lightway-core/src/connection.rs                                  486               237    51.23%          48                19    60.42%         669               283    57.70%           0                 0         -
lightway-core/src/connection/builders.rs                          59                21    64.41%          18                 7    61.11%         236                54    77.12%           0                 0         -
lightway-core/src/connection/dplpmtud.rs                         761                90    88.17%          66                 1    98.48%         947                11    98.84%           0                 0         -
lightway-core/src/connection/fragment_map.rs                     132                11    91.67%          27                 0   100.00%         290                 7    97.59%           0                 0         -
lightway-core/src/connection/io_adapter.rs                       137                17    87.59%          36                 5    86.11%         313                23    92.65%           0                 0         -
lightway-core/src/connection/key_update.rs                        23                 7    69.57%           5                 0   100.00%          38                19    50.00%           0                 0         -
lightway-core/src/context.rs                                      89                25    71.91%          24                 7    70.83%         197                40    79.70%           0                 0         -
lightway-core/src/context/ip_pool.rs                               7                 3    57.14%           1                 0   100.00%           6                 0   100.00%           0                 0         -
lightway-core/src/context/server_auth.rs                          14                11    21.43%           4                 3    25.00%          24                20    16.67%           0                 0         -
lightway-core/src/io.rs                                            3                 3     0.00%           3                 3     0.00%          10                10     0.00%           0                 0         -
lightway-core/src/lib.rs                                           9                 6    33.33%           4                 1    75.00%          18                 9    50.00%           0                 0         -
lightway-core/src/metrics.rs                                       9                 9     0.00%           7                 7     0.00%          17                17     0.00%           0                 0         -
lightway-core/src/packet.rs                                       27                 7    74.07%           4                 1    75.00%          30                 6    80.00%           0                 0         -
lightway-core/src/plugin.rs                                       66                 8    87.88%          23                 5    78.26%         160                 9    94.38%           0                 0         -
lightway-core/src/utils.rs                                        99                19    80.81%          20                 0   100.00%         178                11    93.82%           0                 0         -
lightway-core/src/version.rs                                      37                 0   100.00%          19                 0   100.00%          89                 0   100.00%           0                 0         -
lightway-core/src/wire.rs                                        140                19    86.43%          32                 0   100.00%         256                 2    99.22%           0                 0         -
lightway-core/src/wire/auth_failure.rs                             9                 1    88.89%           3                 0   100.00%          19                 0   100.00%           0                 0         -
lightway-core/src/wire/auth_request.rs                           147                12    91.84%          34                 0   100.00%         287                 0   100.00%           0                 0         -
lightway-core/src/wire/auth_success_with_config_ipv4.rs           69                 4    94.20%          13                 0   100.00%         145                 0   100.00%           0                 0         -
lightway-core/src/wire/data.rs                                    20                 1    95.00%           7                 0   100.00%          43                 0   100.00%           0                 0         -
lightway-core/src/wire/data_frag.rs                               41                 1    97.56%          17                 0   100.00%          92                 0   100.00%           0                 0         -
lightway-core/src/wire/ping.rs                                    22                 3    86.36%           7                 0   100.00%          68                 0   100.00%           0                 0         -
lightway-core/src/wire/pong.rs                                    15                 2    86.67%           5                 0   100.00%          34                 0   100.00%           0                 0         -
lightway-core/src/wire/server_config.rs                           20                 2    90.00%           5                 0   100.00%          44                 0   100.00%           0                 0         -
lightway-server/src/args.rs                                       29                29     0.00%          29                29     0.00%          29                29     0.00%           0                 0         -
lightway-server/src/auth.rs                                      104                36    65.38%          19                 5    73.68%         148                28    81.08%           0                 0         -
lightway-server/src/connection.rs                                 51                51     0.00%          23                23     0.00%         110               110     0.00%           0                 0         -
lightway-server/src/connection_manager.rs                         87                87     0.00%          25                25     0.00%         209               209     0.00%           0                 0         -
lightway-server/src/connection_manager/connection_map.rs          87                 9    89.66%          26                 2    92.31%         273                 9    96.70%           0                 0         -
lightway-server/src/io/inside/tun.rs                              25                25     0.00%           8                 8     0.00%          34                34     0.00%           0                 0         -
lightway-server/src/io/outside/tcp.rs                             80                80     0.00%           8                 8     0.00%          99                99     0.00%           0                 0         -
lightway-server/src/io/outside/udp.rs                            142               142     0.00%          18                18     0.00%         262               262     0.00%           0                 0         -
lightway-server/src/io/outside/udp/cmsg.rs                        47                16    65.96%          14                 5    64.29%         233                56    75.97%           0                 0         -
lightway-server/src/ip_manager.rs                                 82                25    69.51%          19                 4    78.95%         254                28    88.98%           0                 0         -
lightway-server/src/ip_manager/ip_pool.rs                        101                 1    99.01%          32                 0   100.00%         305                 0   100.00%           0                 0         -
lightway-server/src/lib.rs                                        93                93     0.00%          10                10     0.00%         148               148     0.00%           0                 0         -
lightway-server/src/main.rs                                       65                65     0.00%           9                 9     0.00%         175               175     0.00%           0                 0         -
lightway-server/src/metrics.rs                                    91                89     2.20%          80                78     2.50%         216               212     1.85%           0                 0         -
lightway-server/src/statistics.rs                                 53                23    56.60%           8                 4    50.00%         103                42    59.22%           0                 0         -
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                           4762              2048    56.99%        1037               445    57.09%        9077              3201    64.74%           0                 0         -

✅ Region coverage 56% passes
✅ Line coverage 64% passes

@kp-mariappan-ramasamy kp-mariappan-ramasamy merged commit c0a32e3 into main Feb 11, 2025
11 checks passed
@kp-mariappan-ramasamy kp-mariappan-ramasamy deleted the cvpn-1687-fix-memory-usage branch February 11, 2025 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants