Go HeapAlloc slowing increasing

[otoolep]

During 500K EPS write load of the TSM engine, internal metrics show the HeapAlloc profile slowly increasing. There was also query load on the system during this time, but not that significant.

See attached graph. This was with a6cdb52

[otoolep]

[otoolep]

cc @jwilder @pauldix @mjdesa

@mjdesa -- we need to get the Go runtime stats into Grafana for our test beds. Telegraf has an outstanding PR by @mark-rushakoff that will help.

[otoolep]

37K queries had been executed in total on this node.

[jwilder]

Would be good to run a test w/ b1 or bz1 to see if we see the same profile.

[desa]

Added a task to track go runtime stats into Grafana to Tracker.

[pauldix]

Any chance the heap growth is tied to the growing number of tsm files? All of those indexes are kept on the heap.

Would be good to run a test at this scale (5M unique series posted every 10s) without queries and see if the heap growth changes.

[otoolep]

Yeah, @pauldix -- that sounds feasible. Heap usage growth was pretty slow, we just noticed it and so I thought we should capture it. It the write load was taken off, perhaps it would flatline.

[mark-rushakoff]

The PR that @otoolep mentioned is influxdata/telegraf#449. Not merged yet but I think we're close. It should work fine now if you need it and you're willing to build telegraf from that branch.

[jwilder]

Tests run #4977 indicate this issue is resolved.