[Ingest Manager] Agent bulk actions UI #77690

jen-huang · 2020-09-16T23:24:04Z

Resolves #72712. Resolves #72653.

Summary

This PR adds bulk reassign agents and bulk unenroll agents capabilities. Both the UI and API for these capabilities are license protected (Gold+):

Added UI for bulk actions selection per design spec
Revised UI for single unenroll agent modal (same modal for bulk unenroll) per design spec
Added POST /api/ingest_manager/fleet/agents/bulk_reassign endpoint which expects payload

{
  agents: [array of agent IDs | kuery string],
  policy_id: [ID of new policy to reassign agents to]
}

Added POST /api/ingest_manager/fleet/agents/bulk_unenroll endpoint which expects payload

{
  agents: [array of agent IDs | kuery string],
  force: [boolean to force unenroll or not]
}

Adjusted GET /api/ingest_manager/fleet/agents API response to include totalInactive property, which is the total number of inactive agents from the returned list of agents (in other words total - totalInactive = # of active agents).
Added API integration tests for bulk unenroll to existing unenroll test suite. Added new reassign test suite that includes tests for bulk reassign.

In addition, this PR also includes some refactoring related to supporting this work:

class LicenseServer moved to /common, isGoldPlus() method added to it, and now both client and server plugins instantiates it. On server it is a simple service, and on client a useLicense() hook was added to use it.
Added normalizeKuery() method for use in server code, DRY-ing up all places where the same code was used.
Added findAllSOs() method for use in server code to retrieve all SOs without paging parameters (aka returns all results by paging though pageSize: 10000 limit).

Screenshots

It's hard to capture all the permutations of this UI, so here are some gifs 😄 Note that I changed table size to be shorter for demo purposes.

Basic table selection behavior

Reassigning agents by manual selection

Force unenroll agents based on current table query (aka select all pages)

Taking actions based on current table query (including when inactive agents are shown)

… action selection UI that may or may not include active agents

…orce option

elasticmachine · 2020-09-17T20:04:06Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ph · 2020-09-17T20:26:43Z

@jen-huang I can't wait to try that out tomorrow.

jfsiii · 2020-09-17T22:26:36Z

x-pack/plugins/ingest_manager/server/routes/agent/handlers.ts

+      };
+    }, {});
+    return response.ok({ body });
+  } catch (e) {


Mind replacing this with the new defaultIngestErrorHandler pattern?

Like df19026#diff-a4f9e2e344a4b58d7209d56878afed76

nice, didn't know about this. I updated here as well as the equivalent single agent handler too for consistency, as well as the unenroll agent(s) handlers. I guess we have a task somewhere to replace and use it in all handlers?

I guess we have a task somewhere to replace and use it in all handlers?

@jen-huang better yet, a PR #77975 🎉

jfsiii · 2020-09-17T22:28:22Z

.../public/applications/ingest_manager/sections/fleet/components/agent_unenroll_modal/index.tsx

+        throw error;
+      }
+      setIsSubmitting(false);
+      const successMessage = forceUnenroll


I am struggling to read this nested ternary. Would you mind splitting it up?

jfsiii · 2020-09-17T22:35:39Z

x-pack/plugins/ingest_manager/server/services/agents/unenroll.ts

+    | {
+        agentIds: string[];
+      }
+    | {


jfsiii · 2020-09-17T22:39:11Z

x-pack/plugins/ingest_manager/server/services/agents/unenroll.ts

+  const apiKeys: string[] = [];
+
+  // Get all API keys that need to be invalidated
+  agentsToUpdate.map((agent) => {


Maybe .forEach or for-of since we're not using the return value?

jfsiii · 2020-09-17T22:46:22Z

x-pack/plugins/ingest_manager/server/services/saved_object.ts

+        page: currentPage,
+      });
+      if (currentPageSavedObjects.length) {
+        savedObjectResults = [...savedObjectResults, ...currentPageSavedObjects];


Can we use an area method like .concat or .push here? Spreading thousands of objects is taxing on the engine

jfsiii · 2020-09-17T22:49:28Z

x-pack/test/ingest_manager_api_integration/apis/fleet/agents/reassign.ts

+        .send({
+          policy_id: 'INVALID_ID',
+        })
+        .expect(500);


Seems more like a 4xx error but not a big deal rn.

agreed and will defer too, we don't have a super great pattern of handling errors thrown from services in route handlers right now

actually apparently we are able to forward the Boom.notFound error code :D updated this assertion to 404

👍 I did the same in #77975 & @nchaulet agrees #77975 (comment)

…policy to es archiver data

nchaulet · 2020-09-18T19:10:48Z

Out of curiosity did you tested this with a high number of agents? like 1k? 10k?
I am curious how actions like force unenroll are working as we are doing a lot of thing per agent, invalidating two key, update a SO.

jen-huang · 2020-09-18T21:12:31Z

Good question @nchaulet, I just tested with 1000 and 10,000 agents. For 1000 agents, the force unenroll action took a few seconds but completed well. For 10,000 agents, it failed with various errors that look to be caused by too many parallel ES requests:

[search_phase_execution_exception] response from /_security/api_key: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":[],"caused_by":{"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.action.search.FetchSearchPhase$1@18e5165e on EsThreadPoolExecutor[name = Jens-MacBook-Pro.local/system_read, queue capacity = 2000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@335e9005[Running, pool size = 4, active threads = 4, queued tasks = 2000, completed tasks = 176131]]"}},"status":429}

server    log   [14:03:47.271] [error][api-key][plugins][security] Failed to invalidate API key as current user: [es_rejected_execution_exception] rejected execution of org.elasticsearch.action.ActionRunnable$2@437aacef on EsThreadPoolExecutor[name = Jens-MacBook-Pro.local/system_read, queue capacity = 2000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@335e9005[Running, pool size = 4, active threads = 4, queued tasks = 2000, completed tasks = 177222]]
...
server    log   [14:04:21.780] [error][elasticsearch][security] Request error, retrying
DELETE http://localhost:9200/_security/api_key => connect EADDRNOTAVAIL 127.0.0.1:9200 - Local (0.0.0.0:0)
...
server    log   [14:05:55.074] [error][api-key][plugins][security] Failed to invalidate API key as current user: Request Timeout after 30000ms

The ES API keys interface doesn't provide a way to bulk invalidate API keys, so Kibana security interface doesn't either. I'll investigate next week if it is possible to batch the requests or run in them in the background. If you have any suggestions, let me know!

nchaulet · 2020-09-22T16:01:10Z

@jen-huang I am not sure how we can implement that, but maybe this can be done as a background task, at some point with a high number of agents it's sure that the request is going to be too long.

We can probably merge it like is it, and create an issue for scaling this.

jen-huang · 2020-09-22T18:24:51Z

@nchaulet I adjusted the code so that it invalidates API keys in batches instead, 500 keys at a time. This worked well for 10,000 agents, it took a reasonable amount of time to execute and didn't trigger any Kibana or ES errors 🎉

kibanamachine · 2020-09-22T20:04:32Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 0406652

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
ingestManager	551	+3	548

async chunks size

id	value	diff	baseline
ingestManager	1.1MB	+12.3KB	1.1MB

page load bundle size

id	value	diff	baseline
ingestManager	518.7KB	+4.2KB	514.5KB

distributable file count

id	value	diff	baseline
default	45943	+1	45942

History

💚 Build #75619 succeeded d447b0e
💔 Build #75319 failed beaf737
💚 Build #75044 succeeded bce6951
💔 Build #75034 failed 1062d93

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Add temporary client-side license service/hook * Initial pass at bulk actions UI (UI behavior only) * Initial pass at implementing reassign agent policy by agent IDs * Allow bulk reassign agent policy by kuery * Return total inactive agents in list agents API to better handle bulk action selection UI that may or may not include active agents * Add isGoldPlus method to license service * Add `normalizeKuery` function * Add `.findAllSOs` method and refactor bulk reassign to use that * Initial pass at backend work for bulk unenroll * Covert unenroll provider to unenroll modal and adjust UI to include force option * Move license protection to handler level, fix misc bugs * Add comments about `data` field response in create agent action(s) * Clean up license service * Fix i18n * Add tests for bulk. unenroll * Add tests for reassign and bulk reassign * Fix typing * Adjust single actions icon and text to be consistent * Fix i18n * PR feedback * Increment api key test assertion to account for adding another agent policy to es archiver data * Fix test * Fix duplicate declaration after merging * Add comments to SO find all function * Batch invalidate API keys requests # Conflicts: # x-pack/plugins/ingest_manager/server/services/index.ts

* master: (31 commits) skip tests for old pacakge (elastic#78194) [Ingest Pipelines] Add url generator for ingest pipelines app (elastic#77872) [Lens] Rename "telemetry" to "stats" (elastic#78125) [CSM] Url search (elastic#77516) [Drilldowns] Config to disable URL Drilldown (elastic#77887) [Lens] Combined histogram/range aggregation for numbers (elastic#76121) Remove legacy plugins support (elastic#77599) 'Auto' interval must be correctly calculated for natural numbers (elastic#77995) [CSM] fix ingest data retry order messed up (elastic#78163) Add response status helpers (elastic#78006) Bump react-beautiful-dnd (elastic#78028) [Security Solution][Detection Engine] Bubbles up more error messages from ES queries to the UI (elastic#78004) Index pattern - refactor constructor (elastic#77791) Add `xpack.security.sameSiteCookies` to docker allow list (elastic#78192) Remove [key: string]: any; from IIndexPattern (elastic#77968) Remove requirement for manage_index_templates privilege for Index Management (elastic#77377) [Ingest Manager] Agent bulk actions UI (elastic#77690) [Metrics UI] Add inventory view timeline (elastic#77804) Reporting/Docs: Updates for setting to enable CSV Download (elastic#78101) Update to latest rum-react (elastic#78193) ...

* [Ingest Manager] Agent bulk actions UI (#77690) * Add temporary client-side license service/hook * Initial pass at bulk actions UI (UI behavior only) * Initial pass at implementing reassign agent policy by agent IDs * Allow bulk reassign agent policy by kuery * Return total inactive agents in list agents API to better handle bulk action selection UI that may or may not include active agents * Add isGoldPlus method to license service * Add `normalizeKuery` function * Add `.findAllSOs` method and refactor bulk reassign to use that * Initial pass at backend work for bulk unenroll * Covert unenroll provider to unenroll modal and adjust UI to include force option * Move license protection to handler level, fix misc bugs * Add comments about `data` field response in create agent action(s) * Clean up license service * Fix i18n * Add tests for bulk. unenroll * Add tests for reassign and bulk reassign * Fix typing * Adjust single actions icon and text to be consistent * Fix i18n * PR feedback * Increment api key test assertion to account for adding another agent policy to es archiver data * Fix test * Fix duplicate declaration after merging * Add comments to SO find all function * Batch invalidate API keys requests # Conflicts: # x-pack/plugins/ingest_manager/server/services/index.ts * Fix export * Fix spacing Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

jen-huang added 25 commits August 26, 2020 13:19

Add temporary client-side license service/hook

024def9

Initial pass at bulk actions UI (UI behavior only)

42de98b

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

3f4479b

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

37af4bc

Initial pass at implementing reassign agent policy by agent IDs

3581922

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

131a956

Allow bulk reassign agent policy by kuery

f23e431

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

74a13a2

Return total inactive agents in list agents API to better handle bulk…

a50c710

… action selection UI that may or may not include active agents

Add isGoldPlus method to license service

5471f5d

Add normalizeKuery function

238c7ed

Add .findAllSOs method and refactor bulk reassign to use that

1fc9131

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

9eefabb

Initial pass at backend work for bulk unenroll

1398ed5

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

3969045

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

5cc9d07

Covert unenroll provider to unenroll modal and adjust UI to include f…

8ddd87a

…orce option

Move license protection to handler level, fix misc bugs

f156d56

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

1041e2b

Add comments about data field response in create agent action(s)

d39767c

Clean up license service

1062d93

Fix i18n

bce6951

Add tests for bulk. unenroll

6e72369

Add tests for reassign and bulk reassign

aa5344e

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

beaf737

jen-huang self-assigned this Sep 17, 2020

jen-huang added release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0 labels Sep 17, 2020

jen-huang added 2 commits September 17, 2020 12:35

Adjust single actions icon and text to be consistent

460844b

Fix i18n

98f967c

jen-huang marked this pull request as ready for review September 17, 2020 20:04

jen-huang requested a review from a team September 17, 2020 20:04

jen-huang mentioned this pull request Sep 17, 2020

[Ingest Manager] Adding bulk packages upgrade api #77827

Merged

jfsiii approved these changes Sep 17, 2020

View reviewed changes

jen-huang added 5 commits September 17, 2020 17:02

PR feedback

4ce2235

Increment api key test assertion to account for adding another agent …

5dd641f

…policy to es archiver data

Fix test

853dd9e

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

52ab1b9

Fix duplicate declaration after merging

d447b0e

jen-huang mentioned this pull request Sep 21, 2020

Support invalidating multiple API keys by their ID elastic/elasticsearch#47609

Closed

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

db2dee9

jen-huang added 3 commits September 22, 2020 10:52

Merge remote-tracking branch 'upstream/master' into ingest/bulk-actions

43f8e41

Add comments to SO find all function

86fa85e

Batch invalidate API keys requests

0406652

jen-huang merged commit 48b81a6 into elastic:master Sep 22, 2020

jen-huang deleted the ingest/bulk-actions branch September 22, 2020 21:23

jen-huang mentioned this pull request Sep 22, 2020

[7.x] [Ingest Manager] Agent bulk actions UI (#77690) #78220

Merged

jen-huang mentioned this pull request Sep 28, 2020

Supplement Fleet docs with bulk action information elastic/observability-docs#149

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Agent bulk actions UI #77690

[Ingest Manager] Agent bulk actions UI #77690

jen-huang commented Sep 16, 2020 •

edited

Loading

elasticmachine commented Sep 17, 2020

ph commented Sep 17, 2020

jfsiii Sep 17, 2020

jen-huang Sep 18, 2020

jfsiii Sep 21, 2020

jfsiii Sep 17, 2020

jfsiii Sep 17, 2020

jfsiii Sep 17, 2020

jfsiii Sep 17, 2020

jfsiii Sep 17, 2020

jen-huang Sep 18, 2020

jen-huang Sep 18, 2020 •

edited

Loading

jfsiii Sep 21, 2020

nchaulet commented Sep 18, 2020

jen-huang commented Sep 18, 2020

nchaulet commented Sep 22, 2020

jen-huang commented Sep 22, 2020 •

edited

Loading

kibanamachine commented Sep 22, 2020

[Ingest Manager] Agent bulk actions UI #77690

[Ingest Manager] Agent bulk actions UI #77690

Conversation

jen-huang commented Sep 16, 2020 • edited Loading

Summary

Screenshots

elasticmachine commented Sep 17, 2020

ph commented Sep 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jen-huang Sep 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nchaulet commented Sep 18, 2020

jen-huang commented Sep 18, 2020

nchaulet commented Sep 22, 2020

jen-huang commented Sep 22, 2020 • edited Loading

kibanamachine commented Sep 22, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

page load bundle size

distributable file count

History

jen-huang commented Sep 16, 2020 •

edited

Loading

jen-huang Sep 18, 2020 •

edited

Loading

jen-huang commented Sep 22, 2020 •

edited

Loading