[Kibana Docs] Provide explanation of Logstash Centralized Pipeline Management

[justinkambic]

Issue

Users have expressed confusion when using Kibana and Elastic Cloud Service about the purpose of the Logstash Centralized Pipeline Management plugin. They make the reasonable assumption that you can deploy Logstash directly from Kibana with no further action.

We should update the documentation to reflect that this is only part of the process, and link to the detailed Logstash documentation that explains how to leverage this feature.

[KOTungseth]

@ycombinator can you give us any advice on how to address this issue?

[ycombinator]

I was not privy to this issue until I got pinged on it just now. @justinkambic, first I want to make sure I'm understanding the confusion correctly.

Users have expressed confusion when using Kibana and Elastic Cloud Service about the purpose of the Logstash Centralized Pipeline Management plugin. They make the reasonable assumption that you can deploy Logstash directly from Kibana with no further action.

I'm curious if this assumption is only being made when users are running Kibana in Elastic Cloud. IOW, did you find that users of on-prem Kibana are not making this assumption?

Also, how frequent is this confusion? Is it a couple of users, a handful, or does this issue seem to come up repeatedly?

Just trying to assess the extent of this confusion.

[justinkambic]

@ycombinator I haven't heard anything related to this in quite a while, so it might've just been one or two instances. That said, I could see it still being confusing for Cloud users, as there's a big component of managing the stack that's abstracted away. If you're managing your stack on-prem, in my opinion it seems like there's less likelihood for confusion because these users are managing the infrastructure as well.

[KOTungseth]

@karenzone can you help us with one? Do you know anything about the background on this issue? We're stumped and are hoping that you can give us some insight since you are the Logstash writer.

[ycombinator]

@justinkambic said:

That said, I could see it still being confusing for Cloud users, as there's a big component of managing the stack that's abstracted away. If you're managing your stack on-prem, in my opinion it seems like there's less likelihood for confusion because these users are managing the infrastructure as well.

A user could be running their ES and Kibana instances in Cloud but LS nodes on-prem. They'd still be able to centrally manage their LS pipelines from the Kibana instance running in Cloud.

Is the confusion that some users expect to install Logstash in Cloud from the Central Management UI?

I haven't heard anything related to this in quite a while, so it might've just been one or two instances.

In that case could we close this issue for now? We can always reopen it if we hear something again, hopefully with a few more specifics?

[justinkambic]

In that case could we close this issue for now? We can always reopen it if we hear something again, hopefully with a few more specifics?

I am 👍 for that.

[DanRoscigno]

This came up again. A customer asked during one of the weekly Cloud AMA sessions. I popped open Kibana just now and went to the Logstash Pipeline entry. I did not expect to be able to deploy Logstash in Elastic Cloud, but I did expect to be able to push out a pipeline that I created to a Logstash instance. I could not figure out what to do next.

I then looked at the Ingest Node entry in the same menu, and it includes a link to the docs. I think it would be great to have a link to the docs from the Logstash Pipeline page so that customers know that they have to register their Logstash instances and how to get the pipelines pulled down from Elasticsearch to Logstash.

I looked at the code for the doc link in the Ingest Node page, but I am not comfortable writing a PR as it has been years since I built a dev build of Kibana.

[DanRoscigno]

cc: @murrayleitch