[expressions] AST Builder

[lukeelmers]

Closes #56748

Prior POCs & discussions: #62109, #62334, #34318

cc @elastic/kibana-canvas, @elastic/kibana-app, @elastic/kibana-app-arch

Summary

Adds two utilities for managing Expression ASTs: buildExpression and buildExpressionFunction. They provide a type-safe mechanism for manipulating & migrating expressions without doing the work of manually traversing the AST.

If you work on an app that uses Expressions, and you find yourself updating an Expression AST directly or -- gasp 😱 -- concatenating Expression strings -- this will make your life easier.

Usage

// App that works with expressions can now build an AST using
// `buildExpressionFunction`, which takes a function name, and object of args:
const fn = buildExpressionFunction('hello', { world: ['whee'] });
fn.addArgument('first', true)
  .addArgument('second', 0)
  .replaceArgument('second', [2])
  .removeArgument('first');

fn.toAst(); // returns function AST

// `buildExpression` takes an expression string, AST,
// or array of `buildExpressionFunction`
const exp = buildExpression([
  buildExpressionFunction('a', {});
  buildExpressionFunction('b', {});
]); // or buildExpression(ast); or buildExpression('a | b');

// add a new function
exp.functions.push(
  buildExpressionFunction('c', { hi: ['friends'] })
);

exp.toAst(); // returns expression AST
exp.toString(); // returns expression string

// You can pass another `buildExpression` to any argument as a subexpression
const exp = buildExpression('foo | bar');
exp.functions.push(
  buildExpressionFunction('baz', {
    subexpressionArg: buildExpression('subexp a=true'),
  })
);

// Type safety provided by generic type params:
const exp = buildExpression([
  buildExpressionFunction<MyFnExpressionFunctionDefinition>('myFn', { hello: [true] });
]);
const fns = exp.findFunction<MyFnExpressionFunctionDefinition>('myFn');

// Eventually the goal is for plugins registering functions to individually export
// their own function builders that return `ExpressionAstFunctionBuilder`, so
// that dependencies stay explicit and you automatically get type safety:
// pluginA
export const myFnBuilder: (args: MyArgs) => ExpressionAstFunctionBuilder<MyFnDef> = (args) => {
  return buildExpressionFunction('myFnName', args);
};
// pluginB
import { myFnBuilder } from '../plugin_a';
const exp = buildExpression([myFnBuilder(args)]);

There's also a buildExpression.findFunction method which aims to make migrations easier by recursively searching for all functions in an expression matching a given name (including subexpressions). It returns references to the buildExpressionFunctions for each of the found functions, which allows you to iterate over each of them to do your migration. There's a unit test showing an example of this, but basically:

// Migrates all `b` functions in the expression to change the `version` arg to 2
const exp = buildExpression('a | b version=1 | c subexp={b version=1}');
exp.findFunction('b').forEach(fn => {
  const arg = fn.getArgument('version');
  if (arg) {
    fn.replaceArgument('version', [2]);
  }
});
exp.toString(); // `a | b version=2 | c subexp={b version=2}`

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for keyboard-only and screenreader accessibility
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
• This was checked for cross-browser compatibility, including a check against IE11

Dev Docs

The expressions plugin has introduced a set of helpers which make it easier to manipulate expression ASTs. Please refer to the PR for more detailed examples.

// also available on `expressions/public/server`
import {
  buildExpression,
  buildExpressionFunction
} from '../../src/plugins/expressions/public';

// `buildExpression` takes an expression string, AST, or array of `buildExpressionFunction`
const exp = buildExpression([
  // `buildExpressionFunction` takes an expression function name, and object of args
  buildExpressionFunction('myFn', { hello: [true] });
]);

const anotherFn = buildExpressionFunction('anotherFn', { world: [false] });
exp.functions.push(anotherFn);
fn.replaceArgument('world', [true]);

exp.toAst(); // prints the latest AST

// you can get added type-safety by providing a generic type argument:
const exp = buildExpression([
  buildExpressionFunction<MyFnExpressionFunctionDefinition>('myFn', { hello: [true] });
]);
const fns = exp.findFunction<MyFnExpressionFunctionDefinition>('myFn');

[lukeelmers]

split this into a separate file for consistency with how we split parse_expression into a separate file

[lukeelmers]

I ran into some issues with the typings for parse and format which were requiring unnecessary casting -- so I updated these types to use conditionals which is more accurate.

[lukeelmers]

Adding this felt like the safest thing to do, however AnyExpressionFunctionDefinition is extremely permissive -- it basically uses any for everything. This could probably be made more strict.

[ppisljar]

code LGTM

[lukeelmers]

@elasticmachine merge upstream

[crob611]

This looks really nice. I know there are a lot of places in Canvas where we manually traverse through an expression to update it (I think maybe all of the "sidebar" stuff that updates the expression does that). I also do a lot of manual building of expressions when dealing with an embeddable's input changing through user interaction. (Like when a user pans a map, the expression needs to be updated to reflect the new center coordinates) and it would be much nicer to build it using this that through the string manipulation I currently do.

Off the top of my head, I can't think of any use cases we might have for it that this wouldn't cover.

Really nice work. Anxious to try it out.

[lukeelmers]

@elasticmachine merge upstream

[lukeelmers]

Highlighting this, as it's probably the main type-safety caveat:

The return value of buildExpression is added as an option for any expression function argument, so if someone is creating a subexpression they must take care to ensure that the expected return value of that whole expression actually satisfies the requirements of the argument it is passed to.

This is unfortunately not something we can determine for them in TS, as the output of an expression is equivalent to the output of the last function in that expression, which is something we don't know at compile time.

[lukeelmers]

These should be optional as default values are provided in the function definition

[lukeelmers]

@elasticmachine merge upstream

[lukeelmers]

@elasticmachine merge upstream

[lukeelmers]

@elasticmachine merge upstream

[wylieconlon]

As discussed offline, the findFunction doesn't seem useful to me unless it wraps sub-expressions in the builder as well, because the literal sub-expressions can't be passed back into the builder in their current state.

[lukeelmers]

findFunction doesn't seem useful to me unless it wraps sub-expressions in the builder as well,

Thanks @wylieconlon -- I've updated this based on our discussion. Any subexpression arguments to a function are now wrapped in the builder, rather than as raw AST. That means calling expressionBuilder.functions, expressionBuilder.findFunction, and expressionFunctionBuilder.arguments will all return builder instances for subexpressions. (And calling toAst on either will still unwrap them as you'd expect)

[lukeelmers]

@elasticmachine merge upstream

[lukeelmers]

Discussed this PR when doing some planning for the expressions service today, and @timroes & I agreed to revert the changes to Lens, which were only added to prove that this builder works.

We felt that touching migrations which have shipped already introduces unnecessary risk to a PR which is otherwise simply adding a new feature.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: ebd8677

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
data	205	+2	203
expressions	102	+3	99
total	-	+5	-

History

• 💔 Build #60355 failed 56fe20d
• 💔 Build #59786 failed 7c296ac
• 💚 Build #59608 succeeded 61d3508
• 💚 Build #58971 succeeded 7ae1994
• 💚 Build #58482 succeeded 25dddb70143086a50f6640de9c100f533eb81730

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

[kibanamachine]

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.