Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BSD: add kqueue support #387

Merged
merged 2 commits into from
May 26, 2022
Merged

BSD: add kqueue support #387

merged 2 commits into from
May 26, 2022

Conversation

hg
Copy link
Contributor

@hg hg commented May 17, 2022

Similar to #266, but for FreeBSD/OpenBSD/NetBSD/macOS.

event.c is in need of splitting into multiple files. I'd rather do that in a separate PR.

Performance

It would be great to test this on physical machines with a fast network between them, if only I had the hardware. So here are results for a FreeBSD VM (13.1) on a Linux desktop (5.17.5).

Profiles

Start tincd, run 30 seconds of iperf3, stop tincd.

select

%   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 27.6       5.27     5.27       10   527.41   527.41  __sys_write [3]
 17.9       8.69     3.42        0  100.00%           __sys_select [13]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 15.0      11.56     2.87        0  100.00%           _mcount [16]
 14.9      14.40     2.84        0  100.00%           __sys_sendto [17]
  9.8      16.28     1.87        0  100.00%           _recvfrom [20]
  2.0      16.67     0.39 12124817     0.00     0.00  logger [11]
  1.8      17.01     0.34        0  100.00%           .mcount (501)
  1.5      17.30     0.29        1   291.34  6852.63  event_loop [2]
  1.5      17.58     0.28       20    14.22    14.22  __sys_read [39]
  0.7      17.72     0.14  2017455     0.00     0.00  __svfscanf [36]
  0.6      17.85     0.12  3026242     0.00     0.00  __vdso_gettc [48]
  0.6      17.96     0.11  1012231     0.00     0.00  __vfprintf [44]
  0.6      18.07     0.11  5054394     0.00     0.00  memcpy [50]
  0.5      18.16     0.09  3029506     0.00     0.00  route [14]
  0.4      18.24     0.08  2017424     0.00     0.00  receive_meta [6]

kqueue

%   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 27.4       4.92     4.92       10   492.41   492.41  __sys_write [5]
 17.1       8.00     3.07        0  100.00%           __sys_sendto [17]
 16.5      10.96     2.97        0  100.00%           _mcount [18]
 12.3      13.18     2.21        0  100.00%           _recvfrom [21]
 12.2      15.36     2.19  3879960     0.00     0.00  __sys_kevent [23]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2.0      15.72     0.36 11660378     0.00     0.00  logger [12]
  2.0      16.07     0.35        0  100.00%           .mcount (498)
  1.5      16.34     0.27       20    13.46    13.46  _read [40]
  1.0      16.52     0.17  1940474     0.00     0.00  __svfscanf [38]
  0.8      16.66     0.14   973307     0.00     0.00  __vfprintf [45]
  0.6      16.77     0.11        1   114.30  8491.48  event_loop [2]
  0.6      16.87     0.11  4860650     0.00     0.00  memcpy [48]
  0.5      16.97     0.10  1940434     0.00     0.00  receive_meta [4]
  0.5      17.06     0.09  2913615     0.00     0.00  route [7]
  0.4      17.13     0.08  1941372     0.00     0.00  __vdso_gettc [58]

Baseline

Direct connection, no tincd.

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.46 GBytes  38.3 Gbits/sec    0   1.67 MBytes
[  5]   1.00-2.00   sec  4.72 GBytes  40.6 Gbits/sec    0   1.67 MBytes
[  5]   2.00-3.00   sec  4.74 GBytes  40.7 Gbits/sec    0   1.67 MBytes
[  5]   3.00-4.00   sec  4.78 GBytes  41.1 Gbits/sec    0   1.67 MBytes
[  5]   4.00-5.00   sec  4.73 GBytes  40.6 Gbits/sec    0   1.67 MBytes
[  5]   5.00-6.00   sec  4.77 GBytes  41.0 Gbits/sec    0   1.67 MBytes
[  5]   6.00-7.00   sec  4.76 GBytes  40.9 Gbits/sec    0   1.67 MBytes
[  5]   7.00-8.00   sec  4.72 GBytes  40.6 Gbits/sec    0   1.67 MBytes
[  5]   8.00-9.00   sec  4.74 GBytes  40.7 Gbits/sec    0   1.67 MBytes
[  5]   9.00-10.00  sec  4.69 GBytes  40.3 Gbits/sec    0   1.67 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  47.1 GBytes  40.5 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  47.1 GBytes  40.5 Gbits/sec                  receiver

$ wrk -c400 -d5s http://192.168.122.4
Running 5s test @ http://192.168.122.4
  2 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.92ms  557.62us  20.31ms   97.69%
    Req/Sec    29.05k   848.97    30.30k    80.00%
  289106 requests in 5.02s, 234.36MB read
Requests/sec:  57641.66
Transfer/sec:     46.73MB

epoll + select

$ wrk -c400 -d30s http://10.0.0.2
Running 30s test @ http://10.0.0.2
  2 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    37.70ms   82.95ms 845.70ms   87.98%
    Req/Sec    31.46k     5.44k   52.88k    72.67%
  1878491 requests in 30.03s, 1.49GB read
Requests/sec:  62554.59
Transfer/sec:     50.71MB

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   109 MBytes   917 Mbits/sec    0   1.69 MBytes
[  5]   1.00-2.00   sec  98.8 MBytes   828 Mbits/sec    0   1.69 MBytes
[  5]   2.00-3.00   sec   108 MBytes   902 Mbits/sec  485   1.30 MBytes
[  5]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0   1.41 MBytes
[  5]   4.00-5.00   sec   110 MBytes   923 Mbits/sec    0   1.51 MBytes
[  5]   5.00-6.00   sec   101 MBytes   849 Mbits/sec    0   1.57 MBytes
[  5]   6.00-7.00   sec   105 MBytes   881 Mbits/sec    0   1.61 MBytes
[  5]   7.00-8.00   sec  97.5 MBytes   818 Mbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec  96.2 MBytes   807 Mbits/sec    0   1.61 MBytes
[  5]   9.00-10.00  sec   109 MBytes   912 Mbits/sec    0   1.65 MBytes
[  5]  10.00-11.00  sec   108 MBytes   902 Mbits/sec  729    597 KBytes
[  5]  11.00-12.00  sec   110 MBytes   923 Mbits/sec    0    724 KBytes
[  5]  12.00-13.00  sec   118 MBytes   986 Mbits/sec   34    646 KBytes
[  5]  13.00-14.00  sec   116 MBytes   975 Mbits/sec    0    773 KBytes
[  5]  14.00-15.00  sec   114 MBytes   954 Mbits/sec    0    881 KBytes
[  5]  15.00-16.00  sec   119 MBytes   996 Mbits/sec    2    718 KBytes
[  5]  16.00-17.00  sec   116 MBytes   975 Mbits/sec   20    646 KBytes
[  5]  17.00-18.00  sec   118 MBytes   986 Mbits/sec    0    775 KBytes
[  5]  18.00-19.00  sec   122 MBytes  1.03 Gbits/sec   10    655 KBytes
[  5]  19.00-20.00  sec   120 MBytes  1.01 Gbits/sec    0    786 KBytes
[  5]  20.00-21.00  sec   119 MBytes   996 Mbits/sec   21    673 KBytes
[  5]  21.00-22.00  sec   114 MBytes   954 Mbits/sec   19    576 KBytes
[  5]  22.00-23.00  sec   118 MBytes   986 Mbits/sec    0    717 KBytes
[  5]  23.00-24.00  sec   116 MBytes   975 Mbits/sec    3    649 KBytes
[  5]  24.00-25.00  sec   116 MBytes   975 Mbits/sec    9    576 KBytes
[  5]  25.00-26.00  sec   115 MBytes   965 Mbits/sec    0    716 KBytes
[  5]  26.00-27.00  sec   114 MBytes   954 Mbits/sec    0    827 KBytes
[  5]  27.00-28.00  sec   119 MBytes   996 Mbits/sec    2    665 KBytes
[  5]  28.00-29.00  sec   116 MBytes   975 Mbits/sec    0    789 KBytes
[  5]  29.00-30.00  sec   119 MBytes   996 Mbits/sec    0    898 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  3.29 GBytes   943 Mbits/sec  1334             sender
[  5]   0.00-30.01  sec  3.29 GBytes   942 Mbits/sec                  receiver

epoll + kqueue

$ wrk -c400 -d30s http://10.0.0.2
Running 30s test @ http://10.0.0.2
  2 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    34.13ms   79.44ms 986.57ms   88.72%
    Req/Sec    30.34k     5.57k   52.94k    70.00%
  1811878 requests in 30.03s, 1.43GB read
Requests/sec:  60327.43
Transfer/sec:     48.90MB

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   119 MBytes   998 Mbits/sec  706    634 KBytes
[  5]   1.00-2.00   sec   118 MBytes   986 Mbits/sec   38    536 KBytes
[  5]   2.00-3.00   sec   120 MBytes  1.01 Gbits/sec    0    689 KBytes
[  5]   3.00-4.00   sec   119 MBytes   996 Mbits/sec    2    631 KBytes
[  5]   4.00-5.00   sec   115 MBytes   965 Mbits/sec    0    755 KBytes
[  5]   5.00-6.00   sec   118 MBytes   986 Mbits/sec    0    868 KBytes
[  5]   6.00-7.00   sec   120 MBytes  1.01 Gbits/sec  150    505 KBytes
[  5]   7.00-8.00   sec   121 MBytes  1.02 Gbits/sec    0    663 KBytes
[  5]   8.00-9.00   sec   122 MBytes  1.03 Gbits/sec    0    796 KBytes
[  5]   9.00-10.00  sec   120 MBytes  1.01 Gbits/sec   98    659 KBytes
[  5]  10.00-11.00  sec   121 MBytes  1.02 Gbits/sec    0    792 KBytes
[  5]  11.00-12.00  sec   125 MBytes  1.05 Gbits/sec    0    902 KBytes
[  5]  12.00-13.00  sec   116 MBytes   975 Mbits/sec   23    576 KBytes
[  5]  13.00-14.00  sec   119 MBytes   996 Mbits/sec    0    716 KBytes
[  5]  14.00-15.00  sec   118 MBytes   986 Mbits/sec   27    609 KBytes
[  5]  15.00-16.00  sec   120 MBytes  1.01 Gbits/sec    0    748 KBytes
[  5]  16.00-17.00  sec   118 MBytes   986 Mbits/sec   59    650 KBytes
[  5]  17.00-18.00  sec   120 MBytes  1.01 Gbits/sec   21    577 KBytes
[  5]  18.00-19.00  sec   122 MBytes  1.03 Gbits/sec   11    502 KBytes
[  5]  19.00-20.00  sec   121 MBytes  1.02 Gbits/sec    0    663 KBytes
[  5]  20.00-21.00  sec   118 MBytes   986 Mbits/sec    0    790 KBytes
[  5]  21.00-22.00  sec   118 MBytes   986 Mbits/sec   40    560 KBytes
[  5]  22.00-23.00  sec   119 MBytes   996 Mbits/sec    0    706 KBytes
[  5]  23.00-24.00  sec   124 MBytes  1.04 Gbits/sec    0    829 KBytes
[  5]  24.00-25.00  sec   124 MBytes  1.04 Gbits/sec   76    703 KBytes
[  5]  25.00-26.00  sec   124 MBytes  1.04 Gbits/sec   35    585 KBytes
[  5]  26.00-27.00  sec   125 MBytes  1.05 Gbits/sec    0    732 KBytes
[  5]  27.00-28.00  sec   122 MBytes  1.03 Gbits/sec   20    665 KBytes
[  5]  28.00-29.00  sec   119 MBytes   996 Mbits/sec    0    789 KBytes
[  5]  29.00-30.00  sec   120 MBytes  1.01 Gbits/sec   49    478 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  3.52 GBytes  1.01 Gbits/sec  1355             sender
[  5]   0.00-30.01  sec  3.52 GBytes  1.01 Gbits/sec                  receiver

wrk results are very unstable and could easily be swapped the other way. The only real difference I'm seeing are somewhat lower latencies with kqueue.

@hg
Copy link
Contributor Author

hg commented May 17, 2022 •
edited
Loading

Fedora job will fail because F36 is using OpenSSL 3.0. Fixed in #386.

There's also this patch which removes unnecessary calls to kevent(), improving iperf3 throughput by 4%.

Not sure how safe it is, though, since it depends on flags always being in agreement with internal kqueue state, unlike how both select and epoll support are implemented.

Edit: not needed anymore, see below.

@gsliepen
Copy link
Owner

I have some older machines with dual Ethernet ports I could convince to run FreeBSD. Results from a VM should indeed be taken with a very large grain of salt.

@hg
Copy link
Contributor Author

hg commented May 18, 2022

kqueue setup is now reduced to a single syscall thanks to EV_RECEIPT which I originally missed in the man page.

iperf3 benchmark 
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   128 MBytes  1.07 Gbits/sec  289    628 KBytes
[  5]   1.00-2.00   sec   125 MBytes  1.05 Gbits/sec    0    768 KBytes
[  5]   2.00-3.00   sec   126 MBytes  1.06 Gbits/sec    8    667 KBytes
[  5]   3.00-4.00   sec   125 MBytes  1.05 Gbits/sec   25    594 KBytes
[  5]   4.00-5.00   sec   124 MBytes  1.04 Gbits/sec    0    737 KBytes
[  5]   5.00-6.00   sec   124 MBytes  1.04 Gbits/sec   22    669 KBytes
[  5]   6.00-7.00   sec   125 MBytes  1.05 Gbits/sec    0    802 KBytes
[  5]   7.00-8.00   sec   125 MBytes  1.05 Gbits/sec   65    488 KBytes
[  5]   8.00-9.00   sec   122 MBytes  1.03 Gbits/sec    0    653 KBytes
[  5]   9.00-10.00  sec   124 MBytes  1.04 Gbits/sec    0    786 KBytes
[  5]  10.00-11.00  sec   125 MBytes  1.05 Gbits/sec   45    675 KBytes
[  5]  11.00-12.00  sec   125 MBytes  1.05 Gbits/sec   39    581 KBytes
[  5]  12.00-13.00  sec   125 MBytes  1.05 Gbits/sec    0    732 KBytes
[  5]  13.00-14.00  sec   121 MBytes  1.02 Gbits/sec    1    649 KBytes
[  5]  14.00-15.00  sec   124 MBytes  1.04 Gbits/sec   37    554 KBytes
[  5]  15.00-16.00  sec   122 MBytes  1.03 Gbits/sec    0    706 KBytes
[  5]  16.00-17.00  sec   121 MBytes  1.02 Gbits/sec   10    643 KBytes
[  5]  17.00-18.00  sec   122 MBytes  1.03 Gbits/sec    4    584 KBytes
[  5]  18.00-19.00  sec   124 MBytes  1.04 Gbits/sec    0    730 KBytes
[  5]  19.00-20.00  sec   124 MBytes  1.04 Gbits/sec   35    658 KBytes
[  5]  20.00-21.00  sec   122 MBytes  1.03 Gbits/sec   37    573 KBytes
[  5]  21.00-22.00  sec   128 MBytes  1.07 Gbits/sec    0    724 KBytes
[  5]  22.00-23.00  sec   122 MBytes  1.03 Gbits/sec   45    622 KBytes
[  5]  23.00-24.00  sec   129 MBytes  1.08 Gbits/sec    0    769 KBytes
[  5]  24.00-25.00  sec   121 MBytes  1.02 Gbits/sec   77    662 KBytes
[  5]  25.00-26.00  sec   129 MBytes  1.08 Gbits/sec    0    798 KBytes
[  5]  26.00-27.00  sec   128 MBytes  1.07 Gbits/sec   65    665 KBytes
[  5]  27.00-28.00  sec   125 MBytes  1.05 Gbits/sec    0    796 KBytes
[  5]  28.00-29.00  sec   130 MBytes  1.09 Gbits/sec   25    680 KBytes
[  5]  29.00-30.00  sec   125 MBytes  1.05 Gbits/sec    0    813 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  3.65 GBytes  1.05 Gbits/sec  829             sender
[  5]   0.00-30.01  sec  3.65 GBytes  1.05 Gbits/sec                  receiver

@hg
Copy link
Contributor Author

hg commented May 18, 2022

The difference is more significant on OpenBSD (same limitations — it's a similarly configured virtual machine).

select

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  21.6 MBytes   181 Mbits/sec    0   52.3 KBytes
[  5]   1.00-2.00   sec  22.0 MBytes   185 Mbits/sec    0   86.3 KBytes
[  5]   2.00-3.00   sec  22.1 MBytes   185 Mbits/sec    0    127 KBytes
[  5]   3.00-4.00   sec  21.6 MBytes   182 Mbits/sec    8   93.3 KBytes
[  5]   4.00-5.00   sec  21.7 MBytes   182 Mbits/sec    5    119 KBytes
[  5]   5.00-6.00   sec  22.4 MBytes   188 Mbits/sec    9    123 KBytes
[  5]   6.00-7.00   sec  22.1 MBytes   185 Mbits/sec    6    107 KBytes
[  5]   7.00-8.00   sec  21.5 MBytes   180 Mbits/sec    5    127 KBytes
[  5]   8.00-9.00   sec  22.1 MBytes   185 Mbits/sec   14    110 KBytes
[  5]   9.00-10.00  sec  22.1 MBytes   185 Mbits/sec    8   94.7 KBytes
[  5]  10.00-11.00  sec  22.1 MBytes   186 Mbits/sec   12    119 KBytes
[  5]  11.00-12.00  sec  22.1 MBytes   186 Mbits/sec   10    100 KBytes
[  5]  12.00-13.00  sec  22.1 MBytes   185 Mbits/sec    4    129 KBytes
[  5]  13.00-14.00  sec  21.7 MBytes   182 Mbits/sec   16    110 KBytes
[  5]  14.00-15.00  sec  21.7 MBytes   182 Mbits/sec    5    112 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec   329 MBytes   184 Mbits/sec  102             sender
[  5]   0.00-15.00  sec   328 MBytes   184 Mbits/sec                  receiver

kqueue

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  33.7 MBytes   282 Mbits/sec    0   55.1 KBytes
[  5]   1.00-2.00   sec  35.8 MBytes   300 Mbits/sec    0   86.3 KBytes
[  5]   2.00-3.00   sec  37.6 MBytes   315 Mbits/sec    0    120 KBytes
[  5]   3.00-4.00   sec  37.7 MBytes   316 Mbits/sec    0    154 KBytes
[  5]   4.00-5.00   sec  36.2 MBytes   304 Mbits/sec    0    189 KBytes
[  5]   5.00-6.00   sec  36.4 MBytes   305 Mbits/sec    0    225 KBytes
[  5]   6.00-7.00   sec  35.8 MBytes   300 Mbits/sec    0    256 KBytes
[  5]   7.00-8.00   sec  36.1 MBytes   303 Mbits/sec    0    305 KBytes
[  5]   8.00-9.00   sec  36.2 MBytes   303 Mbits/sec    0    332 KBytes
[  5]   9.00-10.00  sec  36.8 MBytes   309 Mbits/sec    0    365 KBytes
[  5]  10.00-11.00  sec  36.5 MBytes   307 Mbits/sec    0    532 KBytes
[  5]  11.00-12.00  sec  37.0 MBytes   311 Mbits/sec    0    532 KBytes
[  5]  12.00-13.00  sec  36.0 MBytes   302 Mbits/sec    0    532 KBytes
[  5]  13.00-14.00  sec  36.0 MBytes   302 Mbits/sec    0    532 KBytes
[  5]  14.00-15.00  sec  37.0 MBytes   311 Mbits/sec    0    532 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-15.00  sec   545 MBytes   305 Mbits/sec    0             sender
[  5]   0.00-15.02  sec   543 MBytes   303 Mbits/sec                  receiver

gsliepen
gsliepen reviewed May 20, 2022
View reviewed changes
@hg hg force-pushed the kqueue branch 3 times, most recently from d30a588 to 42a1a60 Compare May 21, 2022 10:38
@hg hg changed the title BSD/macOS: add kqueue support BSD: add kqueue support May 21, 2022
@hg
Copy link
Contributor Author

hg commented May 21, 2022

I split event.c as it's pretty difficult to work with already. Two I/O tree updates were missing in Windows code because it's hard to keep track of all the #ifdefs.

There's a small amount of copy-pasted code in functions like io_add/io_set.

Getting rid of it requires introducing more "public" functions and more calls between translation units. I didn't think it to be worth it, but if you'd rather not have duplicate logic, let me know.

@hg hg force-pushed the kqueue branch 2 times, most recently from d37beee to 2d79077 Compare May 22, 2022 07:28
gsliepen
gsliepen reviewed May 22, 2022
View reviewed changes
Copy link
Owner

@gsliepen gsliepen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I will try it out on FreeBSD on bare metal first, which will probably be next weekend.

@gsliepen
Copy link
Owner

Ok, tried it on FreeBSD on two identical nodes with a gigabit Ethernet switch between them. I'm not trusting iperf results anymore since I got this:

Connection Throughput
Direct 615 Mbit/s
1.1 646 Mbit/s
hg/kqueue 690 Mbit/s

I've rerun the tests, there's a standard deviation of a few Mbit/s. I don't know why the direct connection is slower. Anyway, no regression for your patch, even a ~7% boost in performance (although I would take that with a grain of salt).

@gsliepen gsliepen merged commit 021293e into gsliepen:1.1 May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gsliepen gsliepen gsliepen left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hg @gsliepen