-
Notifications
You must be signed in to change notification settings - Fork 483
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Fleet server performance bottlenecks due to JSON unmarshal #25512
Comments
associated risk (thx @mna): It's interesting to investigate, but the use of Some have accompaniying PR open for a year and a half. It's... not a good look. Would be good to also verify if they take shortcuts like in |
Another extremely popular JSON library is https://github.com/bytedance/sonic and it is also not memory-safe. If we must have memory safety, then I recommend using https://github.com/go-json-experiment/json There is another memory-safe alternative, but it doesn't seem very popular: |
For #26218 Basic Android MDM on/off backend functionality. Manually tested. The following env vars must be set: ``` FLEET_DEV_ANDROID_ENABLED=1 FLEET_DEV_ANDROID_SERVICE_CREDENTIALS=$(cat credentials.json) FLEET_DEV_ANDROID_PUBSUB_TOPIC=projects/your-project/topics/your-topic ``` I picked https://github.com/go-json-experiment/json as the JSON library, which seems like the safest option. - will become json/v2 at some point - currently used in production by other companies, like Tailscale - well-maintained - Some context here: #25512 Plan for next work: - refactoring from 1st PR - add pubsub with device enroll -> spec proxy for fleetdm.com - come back to this sub-task to add tests and finish TODOs # Checklist for submitter - [x] Added/updated automated tests - [x] Manual QA for all new/changed functionality
In case it might be a decent option, I ran across the simd-json library and this write up about simdjson |
@jacobshandling Thank you for sharing. I took a look at it. It only works on Intel/AMD x86 chips, so it might not be a good fit for our M1 dev machines. |
Problem: The Go json/v1 API we are using is at least 2X slower at unmarshaling than pretty much any other library. We do a lot of JSON unmarshaling between osquery results, vulnerability processing, etc.
Example performance issue: #24390
Another example from an old vuln processing pprof run:

The text was updated successfully, but these errors were encountered: