Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adaptive radix tree aggregate matching #2

Merged
merged 4 commits into from
Jan 4, 2024
Merged

Conversation

tonyhb
Copy link
Contributor

@tonyhb tonyhb commented Dec 23, 2023

This PR introduces aggregate adaptive radix tree matching for expressions. The general idea here is to ignore expressions that don't matter given input data. For example:

  • If you have 100m expressions matching on account IDs (event.data.account_id = $foo)
  • And you process 10 events
  • We want to evaluate 10 expressions instead of 1B expressions.

In order to do that, we must parse the CEL expression into its AST, normalize into a canonical representation, then parse each operand into an aggregate tree for matching.

Each expression is broken down into groups. For example:

a == "yes" && b == "no"

Is broken down into a single matching group containing two predicate operators for A and B. For each ident matching strings, ("a" and "b"), we create a new adaptive radix tree matching the given literal value.

When an event is received, we iterate through the keys in the given event. If there's an ART for the key in the event, we match the event's data against the tree. This returns potential expressions to evaluate.

From there, we evaluate only all matched expressions (plus any expressions that cannot be evaluated by aggregate trees). This cuts down the number of expressions evaluated by orders of magnitude.

@tonyhb tonyhb force-pushed the feature/art-prefix-searching branch from 1a8cd3d to 89d7643 Compare December 23, 2023 21:42
@tonyhb tonyhb force-pushed the feature/art-prefix-searching branch from ba05941 to c958f69 Compare January 4, 2024 14:36
@tonyhb tonyhb force-pushed the feature/art-prefix-searching branch from c958f69 to 5693949 Compare January 4, 2024 14:37
@tonyhb tonyhb merged commit 49e3d60 into main Jan 4, 2024
2 checks passed
@tonyhb tonyhb deleted the feature/art-prefix-searching branch January 5, 2024 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant