Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lost functionality with jsonAta #13442

Closed
nicho2 opened this issue Jun 14, 2023 · 1 comment
Closed

lost functionality with jsonAta #13442

nicho2 opened this issue Jun 14, 2023 · 1 comment
Labels
bug unexpected problem or unintended behavior

Comments

@nicho2
Copy link

nicho2 commented Jun 14, 2023

Relevant telegraf.conf

[[outputs.file]]
  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout"]

  ## Data format to output.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
  data_format = "json"

	json_transformation = '''$merge([fields, {"tags": tags, "timestamp": $fromMillis(timestamp*1000)}])'''




# Read formatted metrics from one or more HTTP endpoints
[[inputs.http]]
  ## One or more URLs from which to read formatted metrics
  urls = [
	"https://api.weatherapi.com/v1/current.json?q=limoges&lang=fr&key=........&aqi=yes"
  ]


  ## Amount of time allowed to complete the HTTP request
  # timeout = "5s"

  ## List of success status codes
  # success_status_codes = [200]


  interval = "10s"
  
	# Exclude url and host items from tags
	tagexclude = ["url"]  
	
	
  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  # data_format = "influx"
  data_format = "json_v2"
	[[inputs.http.json_v2]]
			measurement_name_path = "current.condition"
        [[inputs.http.json_v2.object]]
			name_override = "currentWeatherCondition"
            path = "current.condition"
			excluded_keys = ["code"] # List of JSON keys (for a nested key, prepend the parent keys with underscores) that shouldn't be included in result

        [[inputs.http.json_v2.object]]
            path = "current.air_quality"

		[[inputs.http.json_v2.object]]
			path = "current"
			included_keys = ["is_day"]
			



# Process metrics using a Starlark script
[[processors.starlark]]
  ## The Starlark source can be set as a string in this configuration file, or
  ## by referencing a file containing the script.  Only one source or script
  ## should be set at once.
	#namepass = ["3ddlmlite_diag_neighbors"]
  ## Source of the Starlark script.
  source = '''
load("json.star", "json")
load("logging.star", "log")
	
def apply(metric):
	log.debug("debug: {}".format(metric.fields))
	if 'is_day' in metric.fields:
		metric.name = "is_day"
	elif "icon" in metric.fields:
		metric.name = "currentWeatherCondition"
	else:
		metric.name = "externalAirQuality"
	log.debug("debug_val: {}".format(metric))
	return metric
'''

Logs from Telegraf

time="2023-06-14T14:13:34+02:00" level=warning msg="The \"merge\" variable is not set. Defaulting to a blank string."
time="2023-06-14T14:13:34+02:00" level=warning msg="The \"fromMillis\" variable is not set. Defaulting to a blank string."
[{"icon":"//cdn.weatherapi.com/weather/64x64/day/113.png","text":"Ensoleillé"},{"tags":{"host":"FRL17040"},"timestamp":1686744825000}]
[{"co":176.89999389648438,"gb-defra-index":1,"no2":0.5,"o3":121.5999984741211,"pm10":6.099999904632568,"pm2_5":5.800000190734863,"so2":0.6000000238418579,"us-epa-index":1},{"tags":{"host":"FRL17040"},"timestamp":1686744825000}]
[{"is_day":1},{"tags":{"host":"FRL17040"},"timestamp":1686744825000}]
[{"icon":"//cdn.weatherapi.com/weather/64x64/day/113.png","text":"Ensoleillé"},{"tags":{"host":"FRL17040"},"timestamp":1686744833000}]
[{"co":176.89999389648438,"gb-defra-index":1,"no2":0.5,"o3":121.5999984741211,"pm10":6.099999904632568,"pm2_5":5.800000190734863,"so2":0.6000000238418579,"us-epa-index":1},{"tags":{"host":"FRL17040"},"timestamp":1686744833000}]
[{"is_day":1},{"tags":{"host":"FRL17040"},"timestamp":1686744833000}]

System info

Telegraf 1.27.0 (git: HEAD@dd48ee81)

Docker

No response

Steps to reproduce

  1. run
  2. without json_transformation data is:

{"fields":{"icon":"//cdn.weatherapi.com/weather/64x64/day/113.png","text":"Ensoleillé"},"name":"currentWeatherCondition","tags":{"host":"FRL17040"},"timestamp":1686745136}

try json transformation here:
https://try.jsonata.org/TuyDwaRsA
with jsonAta version 1.5.4

With Telegraf in version 1.26.2 the behaviour is correct. (it's regression!!)

Expected behavior

{
"icon": "//cdn.weatherapi.com/weather/64x64/day/113.png",
"text": "Ensoleillé",
"tags": {
"host": "FRL17040"
},
"timestamp": "2023-06-14T12:00:15.000Z"
}

Actual behavior

[{"icon":"//cdn.weatherapi.com/weather/64x64/day/113.png","text":"Ensoleillé"},{"tags":{"host":"FRL17040"},"timestamp":1686744833000}]

Additional info

No response

@nicho2 nicho2 added the bug unexpected problem or unintended behavior label Jun 14, 2023
@powersj
Copy link
Contributor

powersj commented Jun 14, 2023

Going to track this in #13432 as a regression due to the env variable parsing.

@powersj powersj closed this as not planned Won't fix, can't repro, duplicate, stale Jun 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

2 participants