Ability to cache more than 1000 results for a scheduled query

[aysiu]

Problem

There isn't a clear path forward for reporting meaningfully from FleetDM for a large fleet if the scheduled query cached results are capped at 1,000 records. Log destinations are just a dump for data but don't automatically reconcile back to FleetDM, so a lot of decommissioned or lost/stolen devices will still be in the log destination results. The 1000-record cap seems arbitrary. If it's SaaS, you can charge more for more storage / database indexing, but if it's self-hosted, the limit should be configurable or able to be eliminated altogether.

What have you tried?

I've tried log destinations. It's just a mess of data, especially with snapshots. I've also tried using the API, but the API still leverages the cache that's maxed out at 1,000 records.

Potential solutions

Make 1,000 the default but be able to be moved upwards or just removed altogether (no limit), so people using FleetDM with larger fleets can get meaningful reporting from scheduled queries. Or, if that's too hard to add as a configurable option, just remove the limit completely.

What is the expected workflow as a result of your proposal?

Use the API to report on custom queries for the entire fleet or entire segments (separated by OS) of the fleet. If you have 2,000 or 5,000 or 10,000 or 50,000 devices, getting 1000 results back isn't super helpful for reporting purposes.

[iansltx]

Hi @aysiu, your issue appears to be the same as #19600, which we made configurable via query_report_cap in server configuration in v4.53.0. Please confirm whether that's what you're looking for (you can't remove the limit entirely, but you can raise it to an arbitrarily large number); thanks!

[iansltx]

If the above is the fix, I understand why you missed this issue; the most obvious place where query reporting is documented doesn't mention that you can configure the result cap upward. I'll PR in a docs update for that today.

[iansltx]

@aysiu Docs are updated. Does that address your issue?

[aysiu]

Thanks, @iansltx . I haven't had a chance yet to verify, but that looks promising.

[aysiu]

Is this something that needs to be modified by API PATCH call? Or can it also be done through the web UI (Settings > Organization Settings > Agent Options)?

[iansltx]

It needs to be modified via API PATCH.

[aysiu]

Thanks for clarifying.

One odd thing—The query_report_cap appears to be 0 before modification. Does 0 essentially mean the default, which is 1000?

[iansltx]

That's correct. 0 maps to the hard-coded default. Probably a "Golang zero value" convention.

[aysiu]

This seems to work. Thanks!

[fleet-release]

Cache limit expands,
Fleet's knowledge grows with it,
Clarity descends.

Like a city in clouds,
Data in volumes profound,
No device is lost in crowds.

In this vast expanse,
Every query finds its chance,
Users' trust enhances.