JSONSelection refactoring, $variable syntax, and an extensive new README.md

[benjamn]

Now that the implementation of the JSONSelection syntax lives in this repository, I've taken the time to reorganize the single selection_parser.rs file into a directory called json_selection containing a mod.rs file, with the various components are split out into separate files.

Most importantly, I've also written almost 4500 words in a comprehensive new README.md that lives in the json_selection directory. Please make time to read this documentation to understand the current version of the syntax.

As explained in README.md, I added support for $variable expressions, which will eventually support the $this and $args namespaces, as well as a special $ variable that always refers to the current value being processed. The README.md also describes -> method syntax, which will be useful for arbitrary inline data transformations, but I have not yet added support for that feature to the implementation. The JSLiteral and related grammar rules are only used with -> method syntax, so you can ignore them for now.

An important change since the previous version of the JSONSelection syntax is that leading . characters are no longer required for PathSelection chains. This means you can write post.author.name instead of .post.author.name (though the latter is still supported for backwards compatibility). When you need to express a PathSelection consisting of a single key, you can now write $.data instead of .data, taking advantage of the new $ variable.

Now that this code lives in the same repository with other code that uses it, I was able to update all existing consumption sites to accommodate my various renamings (like Selection to JSONSelection, and Property to Key), but if you have outstanding PRs or branches that you're working on, you may encounter some minor rebase conflicts. Let me know if anything I've changed is confusing.

[router-perf]

CI performance tests

• step - Basic stress test that steps up the number of users over time
• events_big_cap_high_rate_callback - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity using callback mode
• large-request - Stress test with a 1 MB request payload
• events - Stress test for events with a lot of users and deduplication ENABLED
• xxlarge-request - Stress test with 100 MB request payload
• events_without_dedup - Stress test for events with a lot of users and deduplication DISABLED
• xlarge-request - Stress test with 10 MB request payload
• step-jemalloc-tuning - Clone of the basic stress test for jemalloc tuning
• events_callback - Stress test for events with a lot of users and deduplication ENABLED in callback mode
• no-graphos - Basic stress test, no GraphOS.
• reload - Reload test over a long period of time at a constant rate of users
• events_big_cap_high_rate - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity
• events_without_dedup_callback - Stress test for events with a lot of users and deduplication DISABLED using callback mode
• const - Basic stress test that runs with a constant number of users

[lennyburdette]

this is awesome!

quick non-blocking Q: $args and $this are not "special" — you can use any identifier in a VarPath, right? i think that's the right move, and we'll have semantic validation that disallows things like $foo in a different part of the code?

[benjamn]

@lennyburdette Yep, $args and $this are not special, and we still need to write code that passes them into the ApplyTo::apply_with_vars method. The only special (always defined) variable at the moment is $.

As for validation, referring to undefined variables (or undefined paths like $var.does.not.exist) currently produces a runtime ApplyToError, which is non-fatal for the rest of the selection (similar to GraphQL result.errors). If the nonexistent value was the immediate child of an object property, the output object simply won't have that property, but if it was an array element, we need a placeholder value to preserve the array structure, so the value becomes null.

[lennyburdette]

Cool, that makes sense. It'll be fun for us (@dylan-apollo) to make static validation match the ApplyTo runtime behavior

[benjamn]

I went ahead and merged this after @lennyburdette's review, since it's vulnerable to merge conflicts and I've been rebasing frequently. I don't mean to shut down discussion or comments from the other reviewers, or anyone who has thoughts/questions.