[ML] Plugin support for scoped clients

[rylnd]

Describe a specific use case for the feature:
Interaction with ML during alert execution, using the alert's permissions (role/licensing authorization).

Describe the feature:
The ML Plugin's contract exposes a number of useful services for consumers to use.

However, all of these service providers require an ApiCaller argument (i.e. a cluster client) to function, and this client must define ml-specific paths in order to function.

As a temporary workaround for 7.7, I exposed the ML client as part of ML's contract. This provides consumers with a ml-path-enriched client to pass to those services. Additionally, it can be used in a request handler to make calls on behalf of the current user by using the asScoped method.

However, there is one major limitation with this approach, and that's alerting. Alerts execute independent of a request (so we cannot use asScoped) but execute on behalf of the user that created them. To support that, alerts are provided a cluster client scoped to their creator (via an API key mimicking their permission). However, due to the paths requirement mentioned above, we cannot use said client with ML services, and are left with the following options:

1. Write custom queries agains ML indices using the scoped client
   • data access is respected, but ML logic is leaked
   • SIEM currently does this to query anomalies during ML Rule execution
2. Use ML services with mlClient.callAsInternalUser)
   • ML logic is not leaked, but neither is data access respected
   • SIEM currently does this to check jobs' status during ML Rule execution

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[rylnd]

Pinging @elastic/kibana-alerting-services in case they might be able to offer a quick solution here:

Our limitations are as follows:

• ML calls require the mlClient as it defines the ML ES endpoints that are called, but:
• mlClient cannot be scoped to the alert's user (via asScoped) because we have no request in the execution context

After looking through the alerting code, it looks like you already do something similar to what I wanted to propose: generate a dummy request in order to scope the provided AlertServices.

If alerting could expose that request (or at least the API key) to the executor, we'd be able to scope the ML client without any other changes. Thoughts?

[rylnd]

@jgowdyelastic I Spoke briefly with @mikecote about the above. From a technical perspective this should indeed be a quick solution, but he's going to talk first with the kibana security team about the implications of exposing that information to the alert.

@mikecote one quick thought: instead of exposing the credentials directly, alerting could provide a function to scope an arbitrary client:

const scopedMlClient = alertingServices.scopeClusterClient(mlClient);

[mikecote]

I've made a comment here to see what drawbacks we would have exposing the request in the alerting framework.

Is there a 7.8 alert that is waiting for a scoped ML client or is the goal to have a client exposed within 7.8?

@mikecote one quick thought: instead of exposing the credentials directly, alerting could provide a function to scope an arbitrary client:

const scopedMlClient = alertingServices.scopeClusterClient(mlClient);

Something like that could work, depending on the answer from platform team, we could also take on code debt and provide a scoped ML client within the services the alert executor receives in the time being.