[Stack Monitoring] Document data stored for ES module

[jasonrhodes]

For the elasticsearch-xpack Metricbeat module, answer the following questions in a Markdown file:

• What endpoints does Metricbeat request data from, and what data is requested?
• Of the data we request, what gets stored in the ES indices? Do we have a description of each field anywhere? What is the source of truth?

Stretch goals (if these become too difficult to answer, we can separate to a new ticket):

• Of the documents and fields that we store, how many are queried by the Kibana app? How many are never queried?
• Of the documents and fields queried, which are used for which graphs and charts and tables etc in the UI?

Notes:

• Consider using monitoring.ui.debug_mode kibana.yml option to visualize queries being made per page

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

Not a markdown document, but kibana module centralizes the paths into constants: https://github.com/elastic/beats/blob/main/metricbeat/module/kibana/kibana.go#L37-L39

ES doesn't, but maybe it should: https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/cluster_stats/cluster_stats.go#L33

That would help answer the first bullet with the code directly I think.

The second point (what's stored) should be covered by https://github.com/elastic/beats/blob/main/metricbeat/module/elasticsearch/_meta/fields.yml

We should be able to add descriptions to those fields.

[weltenwort]

🤔 For day-to-day SDH work the inverse lookup seems useful: Which page/chart in the UI is powered by which aggregation over which documents retrieved from which monitoring endpoint.

[matschaffer]

Nice call @weltenwort - I think the debug logs help a little with that. But what I'd most like to see is something like the "Inspect" support that @smith added to APM UI not long ago. Being able to see which queries were used and easily click over to something like lens or dev tools should make it quite easy to see what doesn't line up I think.

[matschaffer]

As for the issue covered here. I noticed a few modules have _meta/README.md files that look similar to the dev_docs we're building for kibana plugins.

• https://github.com/elastic/beats/blob/main/metricbeat/module/kubernetes/_meta/README.md
• https://github.com/elastic/beats/blob/main/metricbeat/module/system/_meta/testing/README.md
• https://github.com/elastic/beats/blob/main/x-pack/metricbeat/module/airflow/_meta/README.md

I think we should document the modules along with the code as much as possible, so I propose adding info to _meta/README.md for each module.

cc @ruflin who can probably point us in the right direction to validate that as a documentation strategy.

[ruflin]

+1 on using _meta/README.md for the docs. I would have preferred just README.md inside the module directory which makes it easy to find on Github but I remember _meta is required as a prefix as it cleans up all these dev_docs before packaging and having it all in the _meta directory makes this simple.

[neptunian]

Of the documents and fields that we store, how many are queried by the Kibana app? How many are never queried?

Not sure how useful this would be for docs. What are you wanting to do with this information?

Of the documents and fields queried, which are used for which graphs and charts and tables etc in the UI?

For this I think they might make more sense going into the Kibana Stack Monitoring docs. We could have a doc for our visualizations that lists each chart with each elasticsearch field metric related to that chart , or just link to the the code files it's all located in.

[neptunian]

I opened a draft pr. I think it answers the first two questions without duplicating docs and helps to guide someone looking for more information. I think it's fairly straight forward to look at exported fields vs the ES API response. If we wanted to go a step further we could look at the schema of each metricset to map to the exact elasticsearch api field path but they are similar enough that I'm not sure its necessary.

We could improve the description of each field by updating the fields.yml files in the beats module as @matschaffer pointed out.

[matschaffer]

@neptunian @jasonrhodes after trying out the logstash version of this issue (#127808) I'm thinking we should tackle those stretch goals as their own issues.

#109890 is already an outstanding issue in that space.

If we can first build some tooling (like #123280) around checking what fields get queried and used by each part of the stack monitoring UI, that'll make the work of documenting what's used (or not) for each module a lot easier.