Windows Performance Counters input plugin doesn't gather data from multiple instances

[vlastahajek]

Relevant telegraf.conf:

[[inputs.win_perf_counters]]
  ## By default this plugin returns basic CPU and Disk statistics.
  ## See the README file for more examples.
  ## Uncomment examples below or write your own as you see fit. If the system
  ## being polled for data does not have the Object at startup of the Telegraf
  ## agent, it will not be gathered.
  ## Settings:
  # PrintValid = false # Print All matching performance counters
  PreVistaSupport=false
  UseWildcardsExpansion=false

  [[inputs.win_perf_counters.object]]
    ObjectName = "Process"
    #Instances = ["chrome", "chrome#1", "chrome#2"]
    Instances = ["*"]
    Counters = [
      "% Processor Time"
    ]
    Measurement = "win_proc"

System info:

Telegraf 1.7.0
Window 10 64bit

Expected behavior:

> win_proc,host=T480,instance=aesm_service,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=AppleMobileDeviceService,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=ApplicationFrameHost#1,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=ApplicationFrameHost,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=ApsInsSvc,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=audiodg,objectname=Process Percent_Processor_Time=1.547394037246704 1528893106000000000
> win_proc,host=T480,instance=Calculator,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=cmd#1,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=cmd#2,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=cmd,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#1,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#2,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#3,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#4,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#5,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#6,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#7,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost#8,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=conhost,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=csrss#1,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=csrss#2,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=csrss,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=ctfmon#1,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=ctfmon,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=dasHost,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=DbxSvc,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=dllhost#1,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=dllhost#2,objectname=Process Percent_Processor_Time=0 1528893106000000000
> win_proc,host=T480,instance=dllhost,objectname=Process Percent_Processor_Time=0 1528893106000000000

Actual behavior:

> win_proc,host=T480,instance=aesm_service,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=AppleMobileDeviceService,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=ApplicationFrameHost,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=ApsInsSvc,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=audiodg,objectname=Process Percent_Processor_Time=1.5502903461456299 1528892485000000000
> win_proc,host=T480,instance=Calculator,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=cmd,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=conhost,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=csrss,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=ctfmon,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=dasHost,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=DbxSvc,objectname=Process Percent_Processor_Time=0 1528892485000000000
> win_proc,host=T480,instance=dllhost,objectname=Process Percent_Processor_Time=0 1528892485000000000

Additional info:

Problem is that PdhGetFormattedCounterArray returns instance name without an instance index.
And in this case values grouping causes that each instance overwrites value of previous.

[vlastahajek]

As PdhGetFormattedCounterArray doesn't return counter handle neither it is impossible to determine instance index using available data.
A workaround could be at the point of adding value determine if there is already a value and in case true create temporary index for current instance and assign new value.

The flaw here is that there is no guarantee that the instances will be returned in the same order next time, so the same temporary index could be used for different instances.

Of course, the best solution is to use wildcards expansion 🥇

[danielnelson]

Can you create a unittest that shows this behavior?

[vlastahajek]

Sure, np.

[vlastahajek]

The test is here
Note, mocked GetFormatterCounterArrayDouble emulates PdhGetFormattedCounterArray flaw

[danielnelson]

I know we have had problems with multiple instances for a long time now, but has this issue gotten worse since 1.6 or is it essentially the same behavior as before?

[vlastahajek]

PR #4036 caused this..

[PeterKelecom]

I'm running into this issue. Any idea when / if it'll be resolved?

[vlastahajek]

@PeterKelecom What version of Telegraf are you using? If 1.7.x, try setting UseWildcardsExpansion = true

[PeterKelecom]

@vlastahajek That works, thanks!

[andryua]

@PeterKelecom What version of Telegraf are you using? If 1.7.x, try setting UseWildcardsExpansion = true

version 1.8.2 doesn't work for Windows 10 x64 - telegraf doesn't send data
for Windows 7 x32 - doesn't show multiple processes

[danielnelson]

@andryua Can you attach the config you are using?

[andryua]

@andryua Can you attach the config you are using?
yes
telegraf.txt

[natejgardner]

Setting UseWildcardsExpansion to true solved this for me. I was having the same issue where Telegraf wouldn't write anything for win_perf_counters at all until I added that.

[glinton]

@vlastahajek does setting UseWildcardsExpansion work for you?

[andryua]

So, how I can get summary data from multiple instances?

[danielnelson]

@andryua This would be something that can be done using an aggregator plugin or, I think most flexibly, at query time.

[danielnelson]

I feel like we are getting off topic here, this issue is documenting a known limitation/quirk of the current plugin that came up during development. Most people should use UseWildcardsExpansion = true if they can instead, but this option will cause counters to be localized.

If anyone is experiencing additional issues please check for other open issues and if none is found open a new issue.

[andryua]

@andryua This would be something that can be done using an aggregator plugin or, I think most flexibly, at query time.

can, You help me - where I can get this aggrefator plugin and how I can apply it in telegraf for Windows. Thanks!

[vlastahajek]

@vlastahajek does setting UseWildcardsExpansion work for you?

@glinton, everything seems to be working as expected with attached config in master and in 1.8.2. Both, with UseWildcardsExpansion=true and UseWildcardsExpansion=false.

[danielnelson]

@andryua Here are the docs, if you need more help can you create a new topic on the InfluxData Community site.

https://github.com/influxdata/telegraf/tree/master/plugins/aggregators/basicstats

[andryua]

@andryua Here are the docs, if you need more help can you create a new topic on the InfluxData Community site.

https://github.com/influxdata/telegraf/tree/master/plugins/aggregators/basicstats

Thanks! But only on Win 7 x64 work correctly. On Win 10 x64 - show min-max only in process section

[barbarajoost]

With Telegraf 1.8.3 on Server 2008 R2 x64 and UseWildcardsExpansion = true I get the expected results for the process I want to monitor.

[sspaink]

Closing as it seems UseWildcardsExpansion = true is the solution. Please comment/re-open if that isn't the case for you.