Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Malicious IPA hangs in real world setup with 3 helpers #685

Closed
akoshelev opened this issue Jun 7, 2023 · 19 comments
Closed

Malicious IPA hangs in real world setup with 3 helpers #685

akoshelev opened this issue Jun 7, 2023 · 19 comments
Labels
bug Something isn't working

Comments

@akoshelev
Copy link
Collaborator

I have set up 3 machines that represent helper parties and another one that plays the role of report collector (test_mpc). Helpers are using Docker to run IPA code, RC just straight calls cargo to run the test.

The results are disappointing. Semi-honest IPA scales up to 1M event with no issues. Malicious IPA breaks somewhere between 1000 and 5000 input reports.

This will be a tracking issue to fix this bug
@akoshelev akoshelev added the bug Something isn't working label Jun 7, 2023
@akoshelev
Copy link
Collaborator Author

@martinthomson suggested to use artificially high maximum number of HTTP2 streams (10k from the current 4k)

@akoshelev akoshelev mentioned this issue Jun 7, 2023
Merged
@akoshelev
Copy link
Collaborator Author

First thing I am going to try is to run this without Docker

@akoshelev
Copy link
Collaborator Author

fun stuff - forgot that you need to dance around permissions to allow binding to low numbered ports

./target/release/helper --network network.toml --identity 3 --tls-cert h3.pem --tls-key h3.key --port 443
2023-06-07T23:48:13.006296Z  INFO ipa::cli::verbosity: Logging setup at level info
thread 'main' panicked at 'Failed to bind server to a port', /home/ec2-user/workspace/ipa/src/net/server/mod.rs:189:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'Failed to serve: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', /home/ec2-user/workspace/ipa/src/net/server/mod.rs:243:18
[ec2-user@ip-172-31-56-177 ipa]$ sudo setcap CAP_NET_BIND_SERVICE=+eip ./target/release/helper

@akoshelev
Copy link
Collaborator Author

having the same issue without docker as well. we hang at user capping

2023-06-07T23:58:10.857052Z  INFO ipa_query:attribute:apply_window: ipa::protocol::attribution::apply_attribution_window: close time.busy=501µs time.idle=18.9µs
2023-06-07T23:58:10.857068Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: new
2023-06-07T23:58:11.804931Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: close time.busy=527ms time.idle=421ms
2023-06-07T23:58:11.804955Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
...
nothing is happening after

@akoshelev
Copy link
Collaborator Author

next step is to try what Martin has suggested

@akoshelev
Copy link
Collaborator Author

changing this to 10k

ipa/src/net/server/mod.rs

Line 237 in 6292d3a

.http2_max_concurrent_streams(Some(256))

@akoshelev
Copy link
Collaborator Author

akoshelev commented Jun 8, 2023 •
edited
Loading

I am not surprised but it worked for 5k input

2023-06-08T17:01:53.693025Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
2023-06-08T17:02:17.855622Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: close time.busy=13.0s time.idle=11.1s
2023-06-08T17:02:17.855654Z  INFO ipa_query:attribute:aggregate_credit: ipa::protocol::attribution::aggregate_credit: new
2023-06-08T17:02:20.147518Z  INFO ipa_query:attribute:aggregate_credit: ipa::protocol::attribution::aggregate_credit: close time.busy=1.20s time.idle=1.09s
2023-06-08T17:02:20.147554Z  INFO ipa_query:attribute:validate{step=protocol/after_convert_all_bits/validate}: ipa::protocol::context::validator: new
2023-06-08T17:02:20.170653Z  INFO ipa_query:attribute:validate{step=protocol/after_convert_all_bits/validate}: ipa::protocol::context::validator: close time.busy=127µs time.idle=23.0ms
2023-06-08T17:02:20.174874Z  INFO ipa_query:attribute: ipa::protocol::attribution: close time.busy=17.1s time.idle=13.7s
2023-06-08T17:02:20.175747Z  INFO ipa_query: ipa::query::runner::ipa: close time.busy=36.3s time.idle=34.5s

@akoshelev
Copy link
Collaborator Author

Scaling up to 50k

@akoshelev
Copy link
Collaborator Author

50k worked just fine

2023-06-08T17:14:52.968401Z  INFO ipa_query:attribute:validate{step=protocol/after_convert_all_bits/validate}: ipa::protocol::context::validator: close time.busy=156µs time.idle=13.2ms
2023-06-08T17:14:52.981242Z  INFO ipa_query:attribute: ipa::protocol::attribution: close time.busy=178s time.idle=122s
2023-06-08T17:14:52.989412Z  INFO ipa_query: ipa::query::runner::ipa: close time.busy=372s time.idle=281s

@akoshelev
Copy link
Collaborator Author

Overconfident me decided to run it with 500k input

@akoshelev
Copy link
Collaborator Author

It finished within 2hours which is good

@akoshelev
Copy link
Collaborator Author

However I still don't know the root cause. I strongly suspected hyperium/hyper#2419 to be it because we were using hyper 0.14.20 that does not have the fix for it. However, after upgrading to 0.14.26 I am still seeing hangs in user capping

2023-06-08T21:21:03.936735Z  INFO ipa_query:attribute:validate{step=protocol/binary_validator/validate}: ipa::protocol::context::validator: new
2023-06-08T21:21:03.939183Z  INFO ipa_query:attribute:validate{step=protocol/binary_validator/validate}: ipa::protocol::context::validator: close time.busy=793µs time.idle=1.66ms
2023-06-08T21:21:04.321705Z  INFO ipa_query:attribute:apply_window: ipa::protocol::attribution::apply_attribution_window: new
2023-06-08T21:21:04.322332Z  INFO ipa_query:attribute:apply_window: ipa::protocol::attribution::apply_attribution_window: close time.busy=608µs time.idle=19.6µs
2023-06-08T21:21:04.322351Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: new
2023-06-08T21:21:05.375147Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: close time.busy=533ms time.idle=520ms
2023-06-08T21:21:05.375175Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new

cargo.toml

hex = { version = "0.4", optional = true, features = ["serde"] }
hkdf = "0.12.3"
hpke = { version = "0.10.0", default-features = false, features = ["x25519-dalek"] }
hyper = { version = "0.14.26", optional = true, features = ["client", "server", "h2", "stream"] }
hyper-tls = { version = "0.5.0", optional = true }

@akoshelev
Copy link
Collaborator Author

HTTP2 settings negotiated between client and server (both are helpers)

Frame 15: 99 bytes on wire (792 bits), 99 bytes captured (792 bits)
Ethernet II, Src: 0e:df:af:76:80:0d (0e:df:af:76:80:0d), Dst: 0e:be:e3:d5:6f:db (0e:be:e3:d5:6f:db)
Internet Protocol Version 4, Src: 172.31.49.227, Dst: 172.31.49.57
Transmission Control Protocol, Src Port: 3000, Dst Port: 39880, Seq: 1, Ack: 271, Len: 33
HyperText Transfer Protocol 2
Stream: SETTINGS, Stream ID: 0, Length 24
Length: 24
Type: SETTINGS (4)
Flags: 0x00
0... .... .... .... .... .... .... .... = Reserved: 0x0
.000 0000 0000 0000 0000 0000 0000 0000 = Stream Identifier: 0
Settings - Max concurrent streams : 256
Settings - Initial Windows size : 1048576
Settings - Max frame size : 16384
Settings - Max header list size : 16777216

@akoshelev
Copy link
Collaborator Author

Hmm we are genuinely use >200 streams in capping

I added a tracker to count number of active streams: 8660d8f

Here is what I am seeing before user capping gets stuck:

2023-06-08T23:32:08.024202Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
2023-06-08T23:32:08.024355Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits]: 0
2023-06-08T23:32:08.024386Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits/duplicate_multiply]: 1
2023-06-08T23:32:08.127072Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits] closed: 2
2023-06-08T23:32:08.127104Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits/duplicate_multiply] closed: 1
2023-06-08T23:32:08.137016Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit0/upgrade/upgrade_bit_triple0]: 0
2023-06-08T23:32:08.137063Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit0/upgrade/upgrade_bit_triple1]: 1
2023-06-08T23:32:08.137098Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit1/upgrade/upgrade_bit_triple0]: 2
2023-06-08T23:32:08.137135Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit1/upgrade/upgrade_bit_triple1]: 3
2023-06-08T23:32:08.137183Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit2/upgrade/upgrade_bit_triple0]: 4
...

2023-06-08T23:32:09.671726Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit15]: 224
2023-06-08T23:32:09.671744Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit15/duplicate_multiply]: 225
2023-06-08T23:32:09.757334Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit16]: 226
2023-06-08T23:32:09.757377Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit16/duplicate_multiply]: 227
2023-06-08T23:32:09.757406Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit17]: 228
2023-06-08T23:32:09.757431Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit17/duplicate_multiply]: 229

I think my counter is sloppy but nevertheless we are dangerously close to the point where we can't really open a new stream because of 256 concurrent streams limit. Sad that our code is structured the way it can't make progress on the active steps without opening a new stream, @martinthomson work on streaming should help here as well

@akoshelev
Copy link
Collaborator Author

Semi-honest capping is much more frugal in streams usage

2023-06-08T23:40:44.468150Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
2023-06-08T23:40:44.468259Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/mask_source_credits]: 0
2023-06-08T23:40:44.557988Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: channel[H2,step=protocol/after_convert_all_bits/user_capping/mask_source_credits] closed: 1
2023-06-08T23:40:44.601742Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit0/xor1]: 0
2023-06-08T23:40:44.601799Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit1/xor1]: 1
2023-06-08T23:40:44.601833Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit2/xor1]: 2
2023-06-08T23:40:44.601870Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit3/xor1]: 3
2023-06-08T23:40:44.601902Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit4/xor1]: 4
...
2023-06-08T23:40:46.552108Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/dot_product]: 96
2023-06-08T23:40:46.552154Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit0]: 97
2023-06-08T23:40:46.552182Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit1]: 98
2023-06-08T23:40:46.696108Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit2]: 99
2023-06-08T23:40:46.696159Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit3]: 100
2023-06-08T23:40:46.696189Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit4]: 101
2023-06-08T23:40:46.840098Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit5]: 102
2023-06-08T23:40:46.840156Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit6]: 103
2023-06-08T23:40:46.840189Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit7]: 104
2023-06-08T23:40:46.984126Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit8]: 105
2023-06-08T23:40:46.984338Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit9]: 106
2023-06-08T23:40:46.984375Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit10]: 107
2023-06-08T23:40:47.128113Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit11]: 108
2023-06-08T23:40:47.128172Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit12]: 109
2023-06-08T23:40:47.128210Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit13]: 110
2023-06-08T23:40:47.272115Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit14]: 111
2023-06-08T23:40:47.272175Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit15]: 112
2023-06-08T23:40:47.272205Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit16]: 113
2023-06-08T23:40:47.416108Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit17]: 114
2023-06-08T23:40:47.416163Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit18]: 115
2023-06-08T23:40:47.416203Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit19]: 116
2023-06-08T23:40:47.560107Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit20]: 117
2023-06-08T23:40:47.560155Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit21]: 118
2023-06-08T23:40:47.560183Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit22]: 119
2023-06-08T23:40:47.704098Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit23]: 120
2023-06-08T23:40:47.704163Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit24]: 121
2023-06-08T23:40:47.704201Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit25]: 122
2023-06-08T23:40:47.848144Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit26]: 123
2023-06-08T23:40:47.848218Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit27]: 124
2023-06-08T23:40:47.848243Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit28]: 125
2023-06-08T23:40:47.992102Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit29]: 126
2023-06-08T23:40:47.992165Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit30]: 127
2023-06-08T23:40:47.992222Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/dot_product]: 128
2023-06-08T23:40:48.205794Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/and]: 129

@martinthomson
Copy link
Member

I fear that work on streaming will make this worse not better, because it might allow for progress on more steps at a time.

@akoshelev
Copy link
Collaborator Author

I fear that work on streaming will make this worse not better, because it might allow for progress on more steps at a time.

hmm, true. I don't think I understand all the consequences, but maybe we can just remove the limit on number concurrent streams. There must be some bookkeeping cost on both client and server side but it seems we would need to pay it regardless if we really need 10k concurrent streams.

@martinthomson
Copy link
Member

I doubt we'll get to that number, but as long as there is no cost for having that many streams possible, the cost of maintaining state for the streams we do use should be small.

akoshelev added a commit to akoshelev/raw-ipa that referenced this issue Jun 8, 2023
@akoshelev
Remove the limit on number of concurrent streams
954ca4f 
See private-attribution#685 for details. We certainly need more than 256 streams. Having an upper limit close to that number leaves us vulnerable to having this issue again in the future.

So I'd rather not limit it and rely on performance metrics if we start seeing impact of too many streams open.

I also don't see an easy way to test this, so tests are not included :(
@akoshelev akoshelev mentioned this issue Jun 9, 2023
Merged
@akoshelev
Copy link
Collaborator Author

akoshelev commented Jun 9, 2023 •
edited
Loading

fixed by #690

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@martinthomson @akoshelev