Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Networking] Enhance RPC Inspection with Configurable Thresholds and Granular Metrics Collection #5234

Merged
merged 65 commits into from
Jan 22, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
83409c3
refactors duplicate string tracker
thep2p Jan 10, 2024
ff71dee
refactors duplicate string tracker
thep2p Jan 10, 2024
9de01c8
refactors errors to reflect the number of duplicate messages
thep2p Jan 10, 2024
fa9f0ba
extends gossipsub rpc inspector parameters
thep2p Jan 10, 2024
5a2c263
refactors tracker logic in rpc tracker
thep2p Jan 10, 2024
6bf67e8
fixes default config
thep2p Jan 10, 2024
1c8d7d0
fixes flag names
thep2p Jan 11, 2024
bee18ea
fixes config name mismatch
thep2p Jan 11, 2024
4c6b0d8
fixes flag names
thep2p Jan 11, 2024
f30d441
Merge remote-tracking branch 'origin/master' into yahya/6927-duplicat…
thep2p Jan 11, 2024
13e6b87
lint fix
thep2p Jan 11, 2024
23a8f98
lint fix
thep2p Jan 11, 2024
6a69555
switches uints to ints
thep2p Jan 11, 2024
b5b34da
adds tests for ihave message id duplications
thep2p Jan 11, 2024
807dae5
adds tests for graft and prune duplicate topic ids below threshold
thep2p Jan 11, 2024
e9d2e53
adds test for duplicate prune topic id above the threshold
thep2p Jan 11, 2024
8e3a70c
adds test for duplicate graft topic id below the threshold
thep2p Jan 11, 2024
ebf4b75
removes threshold for individual topic ids
thep2p Jan 11, 2024
0b40435
simplifies test messages
thep2p Jan 11, 2024
7991095
fixes flags
thep2p Jan 11, 2024
833dfa4
fixes lint issues with insecure
thep2p Jan 12, 2024
ad1d841
revises config names and supports documents
thep2p Jan 12, 2024
ccb8a74
adds metrics for iwant and ihave inspections
thep2p Jan 15, 2024
26e0968
adds metrics for invalid topic ids
thep2p Jan 15, 2024
f297003
wires all invalid topic id metrics
thep2p Jan 15, 2024
8675cf6
adds metrics for inspection result
thep2p Jan 15, 2024
c0d2f42
adds metrics for publish inspection support
thep2p Jan 15, 2024
70940ab
adds metrics for graft and prune inspection
thep2p Jan 15, 2024
efe63e2
adds metrics for publish inspection
thep2p Jan 15, 2024
7da8f08
updates mocks
thep2p Jan 15, 2024
84c576a
updates noop collector
thep2p Jan 15, 2024
3f967c7
Merge remote-tracking branch 'origin/master' into yahya/6927-duplicat…
thep2p Jan 15, 2024
94fbb48
fixes merge conflicts
thep2p Jan 15, 2024
22dc578
fixes iwant cache miss behavior
thep2p Jan 16, 2024
6d20dbb
fixes valid rpc test
thep2p Jan 16, 2024
a95eb29
adds TestIWantInspection_DuplicateMessageIds_AboveThreshold
thep2p Jan 16, 2024
52cbd02
adds TestIWantInspection_DuplicateMessageIds_BelowThreshold
thep2p Jan 16, 2024
26adb68
adds TestIHaveInspection_DuplicateMessageIds_BelowThreshold
thep2p Jan 16, 2024
ccd0ad5
adds TestIHaveInspection_DuplicateTopicIds_BelowThreshold
thep2p Jan 17, 2024
8279cd9
adds TestIHaveInspection_DuplicateTopicIds_AboveThreshold
thep2p Jan 17, 2024
3baf643
adds TestGraftInspection_InvalidTopic
thep2p Jan 17, 2024
1e7d3fb
adds TestPrueInspection_DuplicateTopicIds_BelowThreshold
thep2p Jan 17, 2024
6f2ce0b
adds TestPruneInspection_DuplicateTopicIds_AboveThreshold
thep2p Jan 17, 2024
dce5bda
Merge branch 'master' into yahya/6927-duplicate-ihave-topics
thep2p Jan 17, 2024
2c72e3d
adds TestGraftInspection_DuplicateTopicIds_BelowThreshold
thep2p Jan 17, 2024
15922ee
Merge branch 'yahya/6927-duplicate-ihave-topics' of github.com:onflow…
thep2p Jan 17, 2024
176c42f
adds TestGraftInspection_DuplicateTopicIds_AboveThreshold
thep2p Jan 17, 2024
98d1294
removes cache miss threshold
thep2p Jan 17, 2024
3116787
lint fix
thep2p Jan 17, 2024
9b7ed6a
lint fix
thep2p Jan 17, 2024
030604e
fixes metrics
thep2p Jan 17, 2024
bc4c41a
updates mocks
thep2p Jan 17, 2024
f2a14b8
updates mocks
thep2p Jan 17, 2024
bd0dbdb
fixes insecure package tests
thep2p Jan 17, 2024
f2fddf3
fixes insecure package tests
thep2p Jan 17, 2024
a6ae09e
relaxes inspection conditions
thep2p Jan 17, 2024
7bdedb2
removes debug line
thep2p Jan 17, 2024
7c9c04e
Merge branch 'master' into yahya/6927-duplicate-ihave-topics
thep2p Jan 18, 2024
90bfcec
fixes TestGossipSubSpamMitigationIntegration
thep2p Jan 18, 2024
38ad17e
Merge branch 'yahya/6927-duplicate-ihave-topics' of github.com:onflow…
thep2p Jan 18, 2024
0b07c2b
fixes broken promises tests
thep2p Jan 18, 2024
20effab
empty commit
thep2p Jan 19, 2024
c955a24
Merge branch 'master' into yahya/6927-duplicate-ihave-topics
thep2p Jan 19, 2024
db64207
lint fix
thep2p Jan 22, 2024
fbe37a5
Merge remote-tracking branch 'origin/master' into yahya/6927-duplicat…
thep2p Jan 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixes flag names
  • Loading branch information
thep2p committed Jan 11, 2024
commit 4c6b0d8424b8d08118112d63398bc797029277f6
15 changes: 14 additions & 1 deletion network/netconf/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@ func AllFlagNames() []string {
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.NotificationCacheSizeKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.MaxSampleSizeKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.MaxMessageIDSampleSizeKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.MaxTotalDuplicateTopicIdThresholdKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.MaxTotalDuplicateMessageIdThresholdKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.DuplicateTopicIdThresholdKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.GraftPruneKey, p2pconfig.MaxTotalDuplicateTopicIdThresholdKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.GraftPruneKey, p2pconfig.MaxSampleSizeKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IWantConfigKey, p2pconfig.MaxSampleSizeKey),
BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IWantConfigKey, p2pconfig.MaxMessageIDSampleSizeKey),
Expand Down Expand Up @@ -259,8 +263,17 @@ func InitializeNetworkFlags(flags *pflag.FlagSet, config *Config) {
config.GossipSub.RpcInspector.Validation.PublishMessages.ErrorThreshold,
"the threshold at which an error will be returned if the number of invalid RPC messages exceeds this value")
flags.Uint(BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.MaxTotalDuplicateTopicIdThresholdKey),
config.GossipSub.RpcInspector.Validation.IHave.MaxTotalDuplicateTopicIdThreshold,
"the max allowed duplicate topic IDs across all iHave control messages in a single RPC message")
flags.Uint(BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.MaxTotalDuplicateMessageIdThresholdKey),
config.GossipSub.RpcInspector.Validation.IHave.MaxTotalDuplicateMessageIdThreshold,
"the max allowed duplicate topic IDs in a single iHave control message")
"the max allowed duplicate message ids in a single iHave control message")
flags.Uint(BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.IHaveConfigKey, p2pconfig.DuplicateTopicIdThresholdKey),
config.GossipSub.RpcInspector.Validation.IHave.DuplicateTopicIdThreshold,
"the number of times a topic ID can be repeated in iHave control messages across the same RPC message before it is considered a duplicate")
flags.Uint(BuildFlagName(gossipsubKey, p2pconfig.RpcInspectorKey, p2pconfig.ValidationConfigKey, p2pconfig.GraftPruneKey, p2pconfig.MaxTotalDuplicateTopicIdThresholdKey),
config.GossipSub.RpcInspector.Validation.GraftPrune.MaxTotalDuplicateTopicIdThreshold,
"the max allowed duplicate topic IDs across all graft/prune control messages in a single RPC message")
flags.Duration(BuildFlagName(gossipsubKey, p2pconfig.SubscriptionProviderKey, p2pconfig.UpdateIntervalKey),
config.GossipSub.SubscriptionProvider.UpdateInterval,
"interval for updating the list of subscribed topics for all peers in the gossipsub, recommended value is a few minutes")
Expand Down
4 changes: 0 additions & 4 deletions network/p2p/config/gossipsub_rpc_inspectors.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,6 @@ type PublishMessageInspectionParameters struct {
ErrorThreshold int `validate:"gte=500" mapstructure:"error-threshold"`
}

const (
MaxTotalDuplicateTopicIdThreshold = "max-total-duplicate-topic-id-threshold"
)

type GraftPruneRpcInspectionParameters struct {
// MaxSampleSize the max sample size used for control message validation of GRAFT and PRUNE. If the total number of control messages (GRAFT or PRUNE)
// exceeds this max sample size then the respective message will be truncated to this value before being processed.
Expand Down