Update json_v2 parser to allow excluding/including specific array elements #9381
Comments
|
Without being able to select an element within the array, you can only get the last element of the array, because the array elements all have the same time stamp and the same tags, and the same field value name, so InfluxDB overwrites the series each time it gets a new entry with the same time stamp, tags and field name. |
|
Actually, I was able to make this work. Here's the telegraf configuration I ended up with, and it only selects the first entry in the array for each station: [[inputs.http]]
urls = [
"http://api.bart.gov/api/etd.aspx?cmd=etd&orig=COLM&key=MW9S-E7SL-26DU-VV8V&dir=n&json=y",
"http://api.bart.gov/api/etd.aspx?cmd=etd&orig=POWL&key=MW9S-E7SL-26DU-VV8V&dir=s&json=y"
]
data_format = "json_v2"
tagexclude = ["host", "url"]
[[inputs.http.json_v2]]
[[inputs.http.json_v2.field]]
path = "root.station.#.etd.0.estimate.0.minutes"
[[inputs.http.json_v2.tag]]
path = "root.station.#.abbr"
rename = "from_station"
[[inputs.http.json_v2.tag]]
path = "root.station.#.etd.#.abbreviation"
rename = "to_station"
[[inputs.http.json_v2.tag]]
path = "root.station.#.etd.0.estimate.0.direction" |
|
Actually on further testing this doesn't completely work. It works in that you can select the first entry in the array, but when you independently select root.station.#.etd.0.estimate.0.minutes for the minutes, and root.station.#.abbr, you get inconsistent data. I am assuming this is because the # is expanded independently for these two paths, so you get every combination of station name and minutes to depart, rather than getting the station name corresponding to the number of minutes to depart. |
|
Hey, I'm having the same problem. Telegraf conf file: [[inputs.mqtt_consumer.json_v2.field]]
path = "data"
rename = "data"
type = "int"
According to GJSON i try it with path = "data". The only number i get is the last one in the array. It's possible to get the length of the array with |
|
@danberg13 Thank you for trying out the new parser, if you want to gather all elements in the data array then the path "data" should be correct and not return only the last one in the array. Here is an example from something I ran locally: Input json: {
"data": [
1,
2,
3
]
}Telegraf config: [[inputs.file]]
files = ["./testdata/fields_and_tags/input.json"]
data_format = "json_v2"
[[inputs.file.json_v2]]
[[inputs.file.json_v2.field]]
path = "data"
Expected output: Hopefully I understood your problem correctly? I also tried it in the gjson playground https://gjson.dev/ which you can use to try out the path syntax. |
|
Thank you @sspaink for testing my issue. Input: Telegraf config: Im testing the new JSON Parser in combination with InfluxDB and Grafana. How do you test it locally? Maybe there is the difference or there is another Problem with the configuration because i still get just the last number of the datainput. Any Ideas? The GJSON Path seems to be correct. Thanks for the help! |
|
@barbaranelson I believe I've implemented a change that addresses the issue your facing. If you'd like you can find the Telegraf artifacts with the changes in the link posted by the tiger bot here. I've updated the README.md in the change and added a unit test that uses bart data and you can find the config,expected output, and input json as reference here. With this change you can now define multiple |
|
@danberg13 Thank you for providing your config, I think the reason your only seeing the last number is because of how influxdata handles duplicate points. In the same measurement, it uses the tag and timestamp to distinguish between two separate line protocols and in your example there is no tag therefore it will just use the last one. I think to avoid this you could use the values in the data array as the tag, then each line protocol will be unique. I am testing out this parser locally by using the unit tests associated with the parser, with them I can add a config using the file input plugin,input json, and expected line protocol output. While working on this issue I added your config and json to a experimental change I am working on, and you can find the files here. If you'd like to try it out you can find the artifacts with the changes posted by the Telegraf tiger bot here. But I don't think you need to use this change to get it working with the json_v2 parser available in 1.19.0. If you cange your config to look like this, it should start working: [[inputs.mqtt_consumer]]
data_format = "json_v2"
[[inputs.mqtt_consumer.json_v2]]
[[inputs.mqtt_consumer.json_v2.object]]
path = "@this"
disable_prepend_keys = true
tags = ["data"]
[inputs.mqtt_consumer.json_v2.object.fields]
cnt = "int"
format = "int" |
|
I tried the new build, and unfortunately it crashed. Here's the stack trace: |
|
Oh no! Thank you for trying it out, I think I know why it happened the function |
|
Here is my config. |
|
Latest artifacts posted by the telegraf tiger no longer throws a panic, but instead produces this line protocol with the provided config: |
|
@sspaink Thank you for the explanation and the new config! The tag solution works but is there a possibility to get the values of the data array as field values, too? I think it would be nice to have the arrays in the JSON as field values. I think the old JSON Parser did it automatically. Thanks! |
|
@danberg13 I am not sure how you could get data array as field values without updating the JSON to include more information so you can use different tags. I don't think the old JSON parser can do it either, but I might be overlooking something. Do you have an example of using the old JSON parser that does what you want? Maybe I can use that to add the same functionality to json_v2. Because the problem we are discussing doesn't pertain to problem in this issue, would you mind creating a new issue describing your problem and tagging me in it? Then we can continue the discussion there. Thanks! |
|
I created the new issue @sspaink. Thanks! |
The json_v2 parser doesn't allow you to exclude/include a element inside an array. The parser should be updated to support index numbers in the JSON key's so that users can select individual array items. At the moment only an entire array can be excluded/included.
e.g.
The JSON key:
etd_0_estimate_0_minutesShould select:
24From the JSON:
{ "root": { "station": [ { "etd": [ { "estimate": [ { "minutes": "24", "platform": "2", "direction": "North", "length": "10", "color": "YELLOW", "hexcolor": "#ffff33", "bikeflag": "1", "delay": "0" } ] } ] } ] } }The text was updated successfully, but these errors were encountered: