[DOCS] Describe how to use Metricbeat to collect monitoring data

[dedemorton]

This PR adds instructions for using Metricbeat to collect monitoring data. It provides content that's parallel to what we provide in other books.

However, I do not think the current user experience is very good. Having the docs live in different books makes the content very fragmented. Also it was sometimes hard for me to tell which content applied to the old way vs the new way of collecting metrics.

I hope after we get the initial content in place, we can go back and improve the user experience.

It's not just a problem with the docs. There are a lot of places where users can make mistakes that prevent things from working. Adding security to the mix makes it even more complicated...especially when features like ILM aren't even taken into consideration with the default users/roles.

Issues to resolve before merging:

• The remote_monitoring_agent role does not have the privileges required to create metricbeat indices if ILM is enabled. Should we tell users to disable ILM, or should we indicate that additional privileges are required when connecting to a cluster that supports ILM (default for beats is auto, which means that Beats use lifecycle management if the cluster supports it)?

Resolution: I've added a tip with a pointer to the section that describes how to secure Beats.

• What do we want to say to Functionbeat users about beats monitoring? Do we want to continue to document internal collection? Of course, using Metricbeat for collection doesn't make sense. Will users want to use Functionbeat itself to read logs from the CloudWatch log group to which Functionbeat writes logs?

Resolution: already done.

• Determine whether Metricbeat can monitor itself. Right now, this does not seem to work. Even though the monitoring data gets indexed into the .monitoring index in Kibana, it does not appear in the Stack monitoring view. This might be related to an issue with setting the UUID for the ES cluster (Be defensive about determining ES cluster UUID #13020). Need to confirm that using Metricbeat to monitor itself is a valid use case.

Resolution: removed the section.

To do after merging:

• Make parallel changes to introductory topics in the ES, LS, and Kib docs (to make the distinction between internal collection and metricbeat clearer).

[dedemorton]

@lcawl I want you to see where I am headed with this content.

I departed slightly from the structure that's used for Kib/ES. Here's why:

• it was hard to tell the difference between the two methods of collecting metrics, so I tweaked the link text in the overview. If you like the change, we can apply it to the ES and Kibana content.

• having the steps for exposing metrics for collection and collecting them with Metricbeat all in one long list was really confusing in the context of configuring Metricbeat to monitor Metricbeat. (I tried conditional coding to add qualifiers, but the language started getting very complicated and wordy, so I broke the list into two super-tasks.

I'm marking this in-progress because I want to do more testing

[lcawl]

@dedemorton that LGTM. I think it's the same intention as the https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html sub-pages, though these more explicit titles are likely better.

[dedemorton]

I think it's the same intention as the https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html sub-pages, though these more explicit titles are likely better.

@lcawl After we get this merged, I can make parallel changes to the Kibana and ES docs to use titles that are more explicit. I also think we should discuss the future of the stack monitoring docs (ways we can improve the user experience).

[ycombinator]

The remote_monitoring_agent role does not have the privileges required to create metricbeat indices if ILM is enabled. Should we tell users to disable ILM, or should we indicate that additional privileges are required when connecting to a cluster that supports ILM (default for beats is auto, which means that Beats use lifecycle management if the cluster supports it)?

I think we should indicate that additional privileges are needed. [EDIT] We could consider expanding the privileges of the built-in remote_monitoring_agent role to support ILM on metricbeat-* indices but I'm hesitant to do this just yet, given some simplifications potentially coming in the future.

What do we want to say to Functionbeat users about beats monitoring? Do we want to continue to document internal collection? Of course, using Metricbeat for collection doesn't make sense. Will users want to use Functionbeat itself to read logs from the CloudWatch log group to which Functionbeat writes logs?

For now I think we continue to document internal collection as an alternative approach, just like we do for Kibana and Elasticsearch monitoring. In the case of Functionbeat specifically, however, would be it possible NOT to document the Metricbeat collection approach at all (since it simply can't be setup by the user)?

Determine whether Metricbeat can monitor itself. Right now, this does not seem to work. Even though the monitoring data gets indexed into the .monitoring index in Kibana, it does not appear in the Stack monitoring view. This might be related to an issue with setting the UUID for the ES cluster (#13020). Need to confirm that using Metricbeat to monitor itself is a valid use case.

This is certainly a valid use case. #13020 was merged recently. Would you mind re-testing when you get a chance?

[dedemorton]

Regarding the feedback:

For now I think we continue to document internal collection as an alternative approach, just like we do for Kibana and Elasticsearch monitoring.

Yes, that's what I've done for now. I've set ifndef::serverless[] so those sections don't get included in the functionbeat docs.

[ycombinator]

LGTM. Thanks so much for creating these docs, @dedemorton!

[bmorelli25]

I ran into some troubles while testing this process with APM Server. It could very well be user error on my end, but here's what I found. @dedemorton perhaps we can chat when you return.

1. This didn't work for me. I kept getting Module beat-xpack doesn't exist!. It wasn't until I ran ./metricbeat modules list that I saw beat-xpack had already been enabled.
   

   beats/libbeat/docs/monitoring/monitoring-metricbeat.asciidoc

   Line 120 in 722c732

   metricbeat modules enable beat-xpack

2. Prior to this step, I had to remove the .disabled from the modules.d/beat-xpack.yml file.
   

   beats/libbeat/docs/monitoring/monitoring-metricbeat.asciidoc

   Line 129 in 722c732

   . Configure the `beat-xpack` module in {metricbeat}. +

3. When running Metricbeat on mac, I also had to run sudo chown root on modules.d/beat-xpack.yml which isn't listed anywhere.