-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increase the maximum number of saved objects that could be installed with a Fleet package #148441
Conversation
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Pinging @elastic/security-solution (Team: SecuritySolution) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fleet change LGTM
c317562
to
8544b40
Compare
Thanks for taking a look at this. What's our "heap budget"?Ignoring how we implement the limit for now... To decide on a safe limit we need to know two things, how much heap is available to Kibana and how much heap the import consumes. The available heap differs a lot between dev and a Kibana build. On my laptop an idle Kibana running in dev mode from the Because Nodejs consumes more memory than what's available for kibana's javascript code, a 1GB cloud instance gets an Assuming you used a dev Kibana instance to test I'm guessing the import of 30k saved objects was getting close to consuming ~404MB. A production Kibana with some load would have less available heap and a higher risk of OOM. It's a bit arbitrary where we draw the line but it feels like if there's ~500MB free heap for an idle instance then consuming more than 256MB heap might start getting dangerous (we might be able to use metrics from our cloud clusters to make a bit more of an educated decision). Based on your projections: So an override limit of 15k might be sufficient for the time being. In the final implementation it would probably be a good idea to have an FTR test that installs the latest package against an 800MB (or even smaller) Kibana instance to ensure we don't hit an OOM. How do we enforce the "heap budget"?Something I just recently thought about is that rules might differ a lot in size based on the specific rule and even specific version. I'm guessing one rule might be a simple query and another one a very elaborate query? This might be a good reason to prefer the |
Thank you, @rudolf, for the detailed feedback.
Yes, if we look at the number of rules we've been releasing recently (see docs), I can assume that the 15k limit might be enough for 10+ upcoming releases. So that would give us some time to implement an alternative approach.
It's a good idea. We have a ticket for improving test coverage of detection rules package installation. Added a note to test for OOMs there: #148176.
Rules indeed vary in size, the distribution looks like this:
I couldn't say that the difference is vast, but we still need to take it into consideration. Overall, relying on the |
… with a Fleet package
8544b40
to
3b0be70
Compare
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Public APIs missing exports
Unknown metric groupsAPI count
History
To update your PR or re-run it, just comment with: cc @xcrzx |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
Resolves: #148175
Summary
If we try to install a package that contains more than 10000 saved objects, the installation fails with the following error:
The max number of objects to import is controlled by the
savedObjects.maxImportExportSize
config option inkibana.yml
. However, we cannot control that option when installing packages, so we must find out how to overcome this limitation.Implemented solution
I've created a setting for the saved object importer that allows you to override the default limit of 10,000 saved objects per import. This setting can be used when initializing the importer if it is determined that the limit can be increased safely.
Usage example:
I tested various limit values using the
security_detection_engine package
and found that a safe limit for our use case is around 30,000 saved objects. To test that, I limited Kibana's available memory to 1 GB using theNODE_OPTIONS=--max-old-space-size=1024
flag and tried importing packages of different sizes to determine the upper limit. I found that packages with 35,000 to 40,000 saved objects can cause Kibana to crash due to an out-of-memory error. However, this should be more than sufficient for the needs of the security solution, as we will likely never have more than 30,000 saved objects in a package.Alternative solutions
It may be possible to divide the installation of a package into smaller chunks, but this approach may only work for certain types of saved objects. One of the steps in the import process is reference validation, which requires all of the saved objects from a package to be loaded into memory at once. This may be fine for the
security_detection_engine
package, as it does not contain any references. However, creating separate import logic based on the contents of a package could result in additional maintenance costs on the Fleet side.An alternative solution would be to use the
maxImportPayloadBytes
setting instead ofmaxImportExportSize
for the saved objects importer. This is because the size of saved objects in a package can vary significantly, and a package with a small number of large objects could potentially create more memory pressure than a package with a large number of small objects. Therefore, using the payload size as a reference point may be a more stable approach.I welcome any thoughts or suggestions from the @elastic/kibana-core and @elastic/fleet teams regarding this issue and potential solutions.