Templatable output data_format

[gazpachoking]

Use Case

I want to output metrics to nagios using NSCA. There is a client which takes input on stdin, but I need to be able to customize the format. Having a data_format that allowed defining a go template would allow for this kind of use case using output.execd without having to write a wrapper script. The problem with the wrapper script is that most of the script consists of parsing the measurement, tags, and fields back out of the influx line protocol (whatever data_format is used.)

Expected behavior

A new data_format for outputting, which allows defining what goes to stdout based on a go template.

Actual behavior

A special script needs to be written which parses one of the existing data_formats, then rewrites to appropriate format that an external application is expecting.

Additional info

No response

[powersj]

Hi @gazpachoking,

This sounds potentially interesting. Would you be willing to provide an example of how you would use this, both a template + destination format, to give us a test case?

[gazpachoking]

Sure. This is super specific to my use case, but here's how I imagined it. The send_nsca program takes data via stdin, and I already have the data needed contained within my tags and fields.

[[outputs.exec]]
  fieldpass = ["CAT"]
  command = ["send_nsca"]
  data_format = "template"
  template = 'my-nagios-host\t{{ .Tag "site" }}-{{ .Tag "device" }}\t0\tOk: {{ .Field "CAT" }}'

EDIT: This is the format send_nsca is expecting host[tab]service[tab]status[tab]message[newline]

Right now I have it working with a wrapper script that parses the influx line protocol then just does the above templating itself and pipes the data to the send_nsca program.

I imagine a templated data format could also be useful if you were writing your own custom script for exec or execd output, in that you could make your parsing of stdin much simpler by only including the exact info you want in whatever format you want.

[powersj]

Thanks for that. I was looking at the template processor earlier to see how we set this up there as well. This could be rather powerful for folks who may want even more customization or power over for JSON output.

My concern is around getting the template definition right. For example, you have the line to define a template that I assume would apply to each metric the serializer receives. Let me chat with Sven when he is back in to see if he has any concerns.

Is this something you are interested in contributing?

[gazpachoking]

I can have a go at it if I get some time, but I'm not too versed in go, so I'm not fully confident if I'll come up with a satisfactory solution yet.

EDIT: Yeah, I might be able to handle it. I've hacked together something that seems to be working. Still needs quite a bit of cleanup and testing though.

[gazpachoking]

So, we have a couple options for how to handle batch mode:

1. Make the template serializer work the same as e.g. the influx one, where batch mode is just the same as individual mode.
2. Make it work more like the json serializer, where the user writes a template for the whole batch rather than for one metric.

I think 2 is the nicer answer, but I noticed that the 'exec' output always serializes in batches, and the 'execd' output always serializes individual metrics. I think that should probably be changed so that there is the option for either batched or non-batched mode in both of those plugins, with the default being individual metrics, so the user doesn't have to unexpectedly change their template when switching between them. The docs don't currently mention this at all, so I think this is already confusing if you are using exec/execd with the json serializer, and already warrants its own issue.

[gazpachoking]

If batch format is separate, is it preferred to just emit the array of metrics as dot for the template? Or is it better to be in a namespace.
i.e. {{ range . }}{{ .Name }}{{ end }} vs {{ range .Metrics }}{{ .Name }}{{ end }}

[gazpachoking]

Here's the work in progress:
master...gazpachoking:template_serializer

[powersj]

Thanks for working on this!

I think that should probably be changed so that there is the option for either batched or non-batched mode in both of those plugins, with the default being individual metrics, so the user doesn't have to unexpectedly change their template when switching between them.

#11902 already exists for execd to support batches versus single metrics. This is not something we would change by default, but an option could be possible given the unexpected behavior.

I think 2 is the nicer answer

Think I agree

Here's the work in progress:

Feel free to put up a PR, but we will absolutely want tests :)

[gazpachoking]

Feel free to put up a PR, but we will absolutely want tests :)

Absolutely. I was waiting to open up a PR until I got some in there.

[gazpachoking]

#11902 already exists for execd to support batches versus single metrics. This is not something we would change by default, but an option could be possible given the unexpected behavior.

This seems a bit different, no? Isn't that ticket to do with how the shim works in an external plugin, rather than how the internal execd plugin works? My understanding is that my proposal wouldn't affect how the shim/external plugin works at all. (I could be misunderstanding though, as I haven't fully grokked the telegraf internals, or how the shims work for external plugins.)

[powersj]

ok I see your point, they are slightly different so we should keep them separate. I've commented on the new issue.