Enhance Query API key information API

[SiddharthMantri]

Summary:

Enhance Query API key information API to support API key management page use cases

Description:

To improve user experience Customers with a large number of API keys are unable to use the API key management page today as there is a lack of pagination support in the GET endpoint. To enable such users to use this feature, we (Kibana security team) are moving away from the Get API key information API to the Query API. We require some changes to the Query API key API to support enhanced searching and term aggregation.

• Search Capabilities

  • Description: Extend the current search capabilities of the API endpoint to accept new query fields.
  • Requirement: Add a new query field called type to the existing ones
  • Possible values: cross_cluster, rest. The filter will only call one or the other and never both at the same time

• Term Aggregation:

  • Description: When loading a subset of API keys, the endpoint should provide all associated usernames, types and if the keys are active/expired, irrespective of the number of keys loaded.
  • Use Case: For instance, a user visits a UI page and loads 10 API keys. These 10 keys are a subset of a total of 20 keys, each created by a different user. The UI's Owner dropdown should display all 20 usernames, even if only 10 are loaded. Similarly, having that information for active/expired keys and types of keys would be needed as well.
  • Note: The term aggregation should honor the privileges manage_api_key and manage_own_api_key for the results. For those users who have the manage_own_api_key privileges, the results should not include the keys that do not belong to them.

UI

This is the existing UI which loads all the API keys and then displays them using an in-memory table.

We intend to move away from this in-memory table. To support a 1:1 mapping between the old and new features, we would also require some enhancements to the API to support use cases of the search bar

Free text search
If a user searches with the term exp, the table currently displays results as follows

This also works similarly if the user searches by API key “type” as ‘personal/cross_cluster` or any other criteria on that page.

To keep the user experience similar across this change, a free text query capability like this one would be a very welcome addition to the Query API key information API.

We’d like to inspect the following fields for free text search:

• Username
• Type
• Name
• Id

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[albertzaharovits]

@SiddharthMantri

1. I think we can expose allowing searching on the API Key type (rest or cross_cluster) without any problems.
2. Wrt to the "Free text search" I think ES should expose the multi-match query type https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html (it does not today), and Kibana should call that to search on both the username and name fields (given the user typed-in text)
3. Wrt to aggregations, regular ES searches actually specify an aggs param to _search requests https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html. Currently, the Query API key doesn't allow for it though. There are 2 options: we either allow aggs similarly to the _search endpoint (I don't think we need to do any validation, but I'll check for it), or we offer some hard-coded specific (term) aggregations, e.g. for username, type and expiry ones, that can be toggled via a simple boolean/enum parameter to the Query API Key API.

I'm going to raise this with the broader team tomorrow, but I would like to first check with you if the above sounds that it's sufficient to achieve the UI features you've described.

[n1v0lg]

@albertzaharovits re:

I think we can expose allowing searching on the API Key type (rest or cross_cluster) without any problems.

It's a little more complicated than it looks, though nothing insurmountable. What complicates this is that API keys created before we introduced the type field, the field is (logically) undefined; these API keys are implicitly rest keys. This means that you can't simply query for rest and get all rest keys; you need to query for s.th. like (rest OR undefined) -- this could happen client-side or we come up with an alternative approach for special handling of the type in the QueryApiKey API. ES-5990 (no link since it's a private Jira issue) gives more context. We mainly didn't prioritize working on this because there was no clear need for it. Now that we have an ask though, we can revisit 👍

[SiddharthMantri]

@albertzaharovits

Wrt point 3, If i understand correctly, we could query the API as

{
  "aggs": {
    "agg-api-keys": {
      "terms": {
        "field": "username"
      }
    }
  }
}

And then add multiple aggs for each of the fields so that we get the "hits" for each of these fields?

Overall, from reading the docs, this looks like a good solution to how we're expecting the UI to function.

[albertzaharovits]

And then add multiple aggs for each of the fields so that we get the "hits" for each of these fields?

Yeah, something like that. It's on the caller to define the aggs names and types (even nested ones).

[albertzaharovits]

Wrt to aggregations, regular ES searches actually specify an aggs param to _search requests https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html. Currently, the Query API key doesn't allow for it though. There are 2 options: we either allow aggs similarly to the _search endpoint (I don't think we need to do any validation, but I'll check for it), or we offer some hard-coded specific (term) aggregations, e.g. for username, type and expiry ones, that can be toggled via a simple boolean/enum parameter to the Query API Key API.

We discussed this in yesterday team's meeting. We're OK to provide the aggs parameter to the Query API Key API (_security/_query/api_key) in an identical fashion to how it's used for the regular _search endpoint, possibly limiting the types of aggs.

[kc13greiner]

Heya @albertzaharovits! Do you have a rough estimate on when this work will be picked up? There is growing interest in this feature!

[albertzaharovits]

@kc13greiner I'm starting to work on this now.

[albertzaharovits]

@kc13greiner @SiddharthMantri I've raised a draft PR that should address all the requirements in the description.
I would appreciate if you could give it a try while it goes through reviews and testing, in order to ensure that it really meets the requirements. Thanks!

[albertzaharovits]

This has been resolved by implementing:

• support for querying by key type (rest or cross_cluster)
• support for aggregations (some types only)
• support for the multi-field simple query string query type

[albertzaharovits]

As well as support for the match query type.