Add support for multiples pipelines

[ph]

Currently Logstash support only one pipeline active at any time. The metrics implementation could reuse the pipeline code (input, filter and output) to send events to the metrics backend. We don't want any hiccups of the main pipeline possibly impacting the collection and transport of the metrics.

This changes could also help us do live reload of configuration or cluster management.

Ref #3693 #3774 #3898

[colinsurprenant]

just for clarification, while a desirable feature to support metrics, multiple pipelines support is not strictly specific for metrics, so in that respect I would remove the "Metrics: " in the subject. Nitpicking, I know.

[ph]

@colinsurprenant agree, let me keep at least the metrics label so I can track it :P

[ph]

Each pipeline should have an ID to help identify it in the stats.

[jsvd]

The refactor work in #3872 opens up the possibility of multiple pipelines. That PR implements it as an array, but could be an hash, to include the id

[fbaligand]

I'm very happy to see this new feature in 2.1 roadmap.
The feature I would love, based on this design improvement is that each input plugin can explicitly indicate which pipeline id for the filter chain.
This is particularly interesting when you have numerous sources in input and we use single thread filter plugin like multiline or globally when we want to be assured that all logs from a same source are processed sequentially (for example when the output is 'file').
In this case, you can have one pipeline per source and each pipeline has one thread.

[purbon]

@jsvd @ph about the multiple pipelines design, I was thinking about the execution model of them. Have we thought about it? do we want them to be completely independent? to interact some way? pulling from the same config or from different ones? what do you think?

[ph]

@purbon I think the initial goal is to run them in isolation, but I agree this could benefit from resource sharing.

[suyograo]

Merged to master