FlattenJSON not working as expected with hashes/maps/objects (and sometimes with elements that doesn't have recursion)

[barongello]

I was searching for time series databases and found TICK. Started to play around and then I was in need of time series. I opened a LiteCoinPool account to mine LTC using just CPU to have a poor performance while I was using the computer to work, play games, etc.

The pool give me access to a JSON API, which I setted as the feeder

So I used httpjson as input and InfluxDB as output on Telegraf

I got results like the file response.json.txt

The pool, network and market keys were successfully parsed and stored into InfluxDB

But the user and workers keys were not parsed

Then I ran Telegraf in test and debug mode to see, which gave me the following results:

╭─username@hostname  ~telegraf  
╰─$ telegraf -test -config telegraf.conf
* Plugin: httpjson, Collection 1
* Plugin: httpjson, Collection 1
> httpjson_litecoinpool_stats,server=https://www.litecoinpool.org/api?api_key\=<<<REMOVED>>> market_btc_usd=419.005,market_ltc_btc=0.00766,market_ltc_cny=20.27211,market_ltc_eur=2.928,market_ltc_gbp=2.15797,market_ltc_rub=237.60188,market_ltc_usd=3.20481,network_block_number=942858,network_difficulty=54484.46947785,network_hash_rate=1594996605,network_next_difficulty=53051.67263839,network_retarget_time=92430,network_time_per_block=147,pool_active_users=796,pool_hash_rate=107900000,pool_pps_rate=0.000000007281,pool_pps_ratio=1.04,response_time=0.8673291980000001 1455937232314069638

╭─username@hostname  ~telegraf  
╰─$ telegraf -debug -config telegraf.conf
2016/02/20 01:10:04 Attempting connection to output: influxdb
2016/02/20 01:10:04 Successfully connected to output: influxdb
2016/02/20 01:10:04 Starting Telegraf (version 0.10.3)
2016/02/20 01:10:04 Loaded outputs: influxdb
2016/02/20 01:10:04 Loaded inputs: httpjson httpjson
2016/02/20 01:10:04 Tags enabled: host=hostname
2016/02/20 01:10:04 Agent Config: Interval:10s, Debug:true, Quiet:false, Hostname:"hostname", Flush Interval:10s 
> httpjson_litecoinpool_stats,host=hostname,server=https://www.litecoinpool.org/api?api_key\=<<<REMOVED>>> market_btc_usd=420.033,market_ltc_btc=0.00766,market_ltc_cny=20.27331,market_ltc_eur=2.917,market_ltc_gbp=2.1581,market_ltc_rub=239.59968,market_ltc_usd=3.205,network_block_number=942863,network_difficulty=54484.46947785,network_hash_rate=1582324350,network_next_difficulty=52959.62436856,network_retarget_time=92430,network_time_per_block=148,pool_active_users=804,pool_hash_rate=107100000,pool_pps_rate=0.000000007281,pool_pps_ratio=1.04,response_time=0.707231897 1455937810707738879
2016/02/20 01:10:10 Gathered metrics, (10s interval), from 2 inputs in 707.746698ms
2016/02/20 01:10:20 Wrote 1 metrics to output influxdb in 54.039878ms

So I came to the source code (the same code was also present on branch 0.3.1 when I created this issue, but now the branch 0.3.1 is gone) of FlattenJSON to see if I can figure out what was happening

I am not fluent in Go, but, when a field value is a hash/map/object, it goes through it recursively (as did on keys pool, network and market)

Is it a problem/issue/bug or am I missing something here?

Thank you in advance

[sparrc]

you should use a supported version, I can't reproduce on HEAD of master. Building from source on any branch is basically not supported. If you want to build a specific version from source, use tags.

[barongello]

@sparrc

Well, maybe I did not expressed myself very well.

• I have got this file: telegraf_0.10.3-1_amd64.deb from the README.md of this project
• I never built Telegraf from source code
• I am using the binary from this package
• This package is marked as latest
• The FlattenJSON source code from this version shows that the JSON is parsed recursively

So, your answer or closing the topic definetely didn't help me

What do you suggest? Build from source code and try the same input file? Or building from HEAD will give me the same binary than from the package marked as latest on README.md?

[sparrc]

If you can give me steps to reproduce I can try to debug further, but from my testing it's working fine. Seems as if maybe the API you are querying might not be returning the data you think it is?

[barongello]

The API is returning exactly what I have attached here (just change the values, of course)

I am building from source to check if the problem persists

---

EDIT:

@sparrc Didn't work

Here is my telegraf.conf (even with API key):

[agent]
  interval = "10s"
  round_interval = true
  metric_buffer_limit = 10000
  flush_buffer_when_full = true
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = false
  quiet = false
  hostname = ""

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf_litecoinpool"
  precision = "s"

[httpjson]
  [[inputs.httpjson]]
    name = "litecoinpool_stats"
    servers = [
      "https://www.litecoinpool.org/api?api_key=b057e1d6b13ce4ec3d9ac9673891db6e"
    ]
    method = "GET"

[sparrc]

@barongello got it, I'm investigating now...

[barongello]

@sparrc Thank you. I am not fluent in Go. I can understand most parts, but some things makes no sense to me yet

Also, I don't know how to debug and such, otherwise I could trace it and maybe do a PR

[sparrc]

okay, I was able to reproduce, re-opening the issue

[sparrc]

@barongello it looks like this PR: #712 broke the httpjson GET parameters, unfortunately 👎

This will fix it: #746

In the meantime, you should use Telegraf 0.10.2 and set your api key as a parameter of the http request, like this:

[agent]
  interval = "10s"
  round_interval = true
  metric_buffer_limit = 10000
  flush_buffer_when_full = true
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = false
  quiet = false
  hostname = ""

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "telegraf_litecoinpool"
  precision = "s"

[[inputs.httpjson]]
  name = ""
  servers = [
    "https://www.litecoinpool.org/api"
  ]
  method = "GET"
  [inputs.httpjson.parameters]
    api_key = "b057e1d6b13ce4ec3d9ac9673891db6e"

[barongello]

@sparrc Thank you

I don't even tryed passing the parameters outside the server URL

Tested now as you said and it is working in 0.10.2 (just needed to remove flush_buffer_when_full from the telegraf.conf because it was giving:

2016/02/23 16:15:21 line 5: field corresponding to `flush_buffer_when_full' is not defined in `*config.AgentConfig'

Will keep on this version as long as I need an API like that for my tests

Also I started studying Go too. Hope I can contribute to TICK soon, I really liked it

[sparrc]

closed by #746