Add configuration option to customize query report cap

[rachaelshaw]

Goal

User story
As a Fleet admin,
I want to customize the result cap for query reports
so that I can see results for all of my hosts.

Currently, for queries that run on >1000 hosts, query reports in the Fleet UI serve as previews of the data returned, rather than true reports of the latest results. (Those users need to send data to a log destination in order to build a complete up-to-date report, since reports in Fleet are clipped at 1,000.)

Context

• Requestor(s): @mikermcneil
• Product designer: @rachaelshaw

Changes

Product

• Config changes: Figma
• REST API changes:
  • Add server_settings.query_report_cap to Fleet configuration API
  • Add report_clipped to GET /api/latest/fleet/queries/:query_id/report response
• Outdated documentation changes: Update API docs to reflect REST API changes. Update GitOps reference PR to call out that you should enable reports for one query at time and monitor your infrastructure

Engineering

• Database schema migrations: TODO
• Load testing: TODO

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires load testing: TODO
• Risk level: Low / High TODO
• Risk description: TODO

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. Engineer (@____): Added comment to user story confirming successful completion of QA.
2. QA (@____): Added comment to user story confirming successful completion of QA.

[lucasmrod]

@rachaelshaw It seems we had the 1000 hardcoded on the UI too.

fleet/frontend/pages/queries/details/QueryDetailsPage/QueryDetailsPageConfig.tsx

Line 15 in 30e4b25

export const QUERY_REPORT_RESULTS_LIMIT = 1000;

fleet/frontend/pages/queries/details/QueryDetailsPage/QueryDetailsPage.tsx

Lines 202 to 203 in 30e4b25

	const isClipped =
	(queryReport?.results?.length ?? 0) >= QUERY_REPORT_RESULTS_LIMIT;

So I added a report_clipped: boolean response field to the GET /api/latest/fleet/queries/$query_id/report API. (Same field we already have in GET /api/latest/fleet/hosts/$host_id/queries/$query_id.)

Let me know if this makes sense.

[rachaelshaw]

@lucasmrod sounds good! Updated the API changes in the description

[noahtalerman]

Hey @rachaelshaw, tracking the TODO from our discussion during product design review in this issue so we see it during confirm and celebrate:

TODO Rachael: Update GitOps reference docs to call out that you should enable reports for one query at time and monitor your infrastructure

UPDATE: Add PR for API changes.

[marko-lisica]

Hey @zayhanlon & @Patagonia121 this story has shipped but before we notify customers we want to make sure docs PR is merged. See the comment above for TODO docs.

[rachaelshaw]

Docs PR here: #20414

[noahtalerman]

Hey @zayhanlon & @Patagonia121 this story has shipped but before we notify customers we want to make sure docs PR is merged.

Hey @zayhanlon and @Patagonia121, the API reference docs are shipped!

Customers can now increase the max number of query report results (per query) Fleet stores.

From the PATCH /config API endpoint reference:

From the GitOps reference:

[fleet-release]

Customize cap high,
More results in your eye.
Fleet expands the sky.