Filecount Input on windows 32bit

[dennisbleul]

Relevant telegraf.conf

[global_tags]

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 10000
  metric_buffer_limit = 100000
  collection_jitter = "1s"
  flush_interval = "10s"
  flush_jitter = "1s"
  precision = "1ms"
  debug = true
  quiet = false
  logtarget = "stderr"
  logfile_rotation_interval = 0
  logfile_rotation_max_size = "5MB"
  logfile_rotation_max_archives = 10
  hostname = ""
  omit_hostname = false

###############################################################################
#                            INPUT PLUGINS                                   #
###############################################################################


[[inputs.filecount]]
  interval = "10s"
  collection_jitter = "1s"
  directories = ["C:/Tools"]
   name = "*"
   recursive = true
  regular_only = true
  follow_symlinks = false
   size = "0B"
  mtime = "0s"


###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################


[[outputs.file]]
  files = ["stdout", "C://Users//ProduLocalAdmin//Desktop//metrics.out"]
  data_format = "influx"

Logs from Telegraf

2022-09-16T07:40:19Z I! Starting Telegraf 1.24.0
2022-09-16T07:40:19Z I! Available plugins: 205 inputs, 9 aggregators, 26 processors, 20 parsers, 57 outputs
2022-09-16T07:40:19Z I! Loaded inputs: filecount
2022-09-16T07:40:19Z I! Loaded aggregators:
2022-09-16T07:40:19Z I! Loaded processors:
2022-09-16T07:40:19Z I! Loaded outputs: file
2022-09-16T07:40:19Z I! Tags enabled: host=FWBOX-PC-A3PT
2022-09-16T07:40:19Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"FWBOX-PC-A3PT", Flush Interval:10s
2022-09-16T07:40:19Z D! [agent] Initializing plugins
2022-09-16T07:40:19Z D! [agent] Connecting outputs
2022-09-16T07:40:19Z D! [agent] Attempting connection to [outputs.file]
2022-09-16T07:40:19Z D! [agent] Successfully connected to outputs.file
2022-09-16T07:40:19Z D! [agent] Starting service inputs
2022-09-16T07:40:20Z E! [inputs.filecount] Error in plugin: EOF
2022-09-16T07:40:29Z D! [outputs.file] Buffer fullness: 0 / 100000 metrics
2022-09-16T07:40:30Z E! [inputs.filecount] Error in plugin: EOF
2022-09-16T07:40:39Z D! [outputs.file] Buffer fullness: 0 / 100000 metrics
2022-09-16T07:40:40Z E! [inputs.filecount] Error in plugin: EOF
2022-09-16T07:40:50Z E! [inputs.filecount] Error in plugin: EOF
2022-09-16T07:40:50Z D! [outputs.file] Buffer fullness: 0 / 100000 metrics
2022-09-16T07:40:55Z D! [agent] Stopping service inputs
2022-09-16T07:40:55Z D! [agent] Input channel closed
2022-09-16T07:40:55Z I! [agent] Hang on, flushing any cached metrics before shutdown
2022-09-16T07:40:55Z D! [outputs.file] Buffer fullness: 0 / 100000 metrics
2022-09-16T07:40:55Z I! [agent] Stopping running outputs
2022-09-16T07:40:55Z D! [agent] Stopped Successfully

System info

Telegraf 1.24.0, Windows Embedded Standard 32 bit

Docker

No response

Steps to reproduce

1. Starting Telegraf Service. It does not matter how I start the telegraf software.

Expected behavior

It counts all files

Actual behavior

In the log file after each interval for each file in the folder the error message
2022-09-16T07:40:50Z E! [inputs.filecount] Error in plugin: EOF

Additional info

No response

[popey]

Thanks for the issue report. Is this something which worked on a previous release of telegraf, but broke in 1.24, or have you not tried a previous release, and this is a new install/setup?

Also, how many files are there in the c:\Tools folder on that machine? (dir /s /b /o:gn > \filelist.txt)

Does this error only occur with that folder or does it occur on any folder, even an empty or mostly empty one?

[dennisbleul]

Thank you for your reply.

it has one file inside.
filelist.txt:
C:\Tools\TransferProduPG_verbinden.bat

I have tested it in several folders and always get the same error message.

With version 1.20.3 it worked. We tested it with 1.24.0 and 1.23.4 and it does not work in both versions anymore.

There is no relation to the number of files in the folders.

[popey]

Thanks for the additional data, very helpful. 1.20.3 dates back nearly a year.

If you have some time and are able to bisect between 1.20.3 and now - perhaps initially via the major releases 1.21.0, 1.22.0, 1.23.0, we might be able to narrow this down more specifically where it broke.

Sadly nobody on the team here has a Windows Embedded 32-bit system we can test on, so we're somewhat relying on your feedback here, for which we're grateful.

[dennisbleul]

I made some tests.
1.21.0, 1.22.0, 1.23.0 and 1.23.3 is working.
1.23.4 is broken.

I made the same test also on an Windows 10 64bit Enterprise Version with telegraf 64bit version and have the same result: since 1.23.4 is broken.

The problem is present in the 32 and 64bit version.

[SamSkjord]

Also experiencing the same issue on Windows Server 2016 x64
win_perf_counter metrics are being recorded correctly

2022-09-22T15:13:00Z I! Starting Telegraf 1.24.1
2022-09-22T15:13:00Z I! Available plugins: 205 inputs, 9 aggregat
2022-09-22T15:13:00Z I! Loaded inputs: filecount win_perf_counter
2022-09-22T15:13:00Z I! Loaded aggregators:
2022-09-22T15:13:00Z I! Loaded processors:
2022-09-22T15:13:00Z I! Loaded outputs: influxdb_v2
2022-09-22T15:13:00Z I! Tags enabled: host=HCC-GSC
2022-09-22T15:13:00Z I! [agent] Config: Interval:20s, Quiet:false
2022-09-22T15:13:20Z E! [inputs.filecount] Error in plugin: EOF
2022-09-22T15:13:40Z E! [inputs.filecount] Error in plugin: EOF
2022-09-22T15:14:00Z E! [inputs.filecount] Error in plugin: EOF

[[inputs.filecount]]
  ## Directories to gather stats about.
  ## This accept standard unit glob matching rules, but with the addition of
  ## ** as a "super asterisk". ie:
  ##   /var/log/**    -> recursively find all directories in /var/log and count files in each directories
  ##   /var/log/*/*   -> find all directories with a parent dir in /var/log and count files in each directories
  ##   /var/log       -> count all files in /var/log and all of its subdirectories
  directories = ["D:/XMLexport"]

  ## Only count files that match the name pattern. Defaults to "*".
  name = "*"

  ## Count files in subdirectories. Defaults to true.
  recursive = true

  ## Only count regular files. Defaults to true.
  regular_only = true

  ## Follow all symlinks while walking the directory tree. Defaults to false.
  follow_symlinks = false

  ## Only count files that are at least this size. If size is
  ## a negative number, only count files that are smaller than the
  ## absolute value of size. Acceptable units are B, KiB, MiB, KB, ...
  ## Without quotes and units, interpreted as size in bytes.
  size = "0B"

  ## Only count files that have not been touched for at least this
  ## duration. If mtime is negative, only count files that have been
  ## touched in this duration. Defaults to "0s".
  mtime = "0s"

Folder contains ~8000 xml files with names like Read_000bab60-9f64-4968-adf5-2cbf7975baf8.xml

[MehrdadGohari]

hi,
I have the same problem.
In Windows 10 64-bit version
The output in version 1.22.4 is correct
But in version 1.24.0 it gives the following error
2022-10-08T07:48:40Z E! [inputs.filecount] Error in plugin: EOF

Is it possible to reach the appropriate output by editing the code with “Go” language?

[dennisbleul]

Is there any news on this topic?

Sorry that I can not actively help. I have no knowledge in Go.

[St3f1n]

We have the same problem, and actually it is blocking us from updating telegraf further. If this issue does not get solved we will need to exchange our filecount plugin implementations with other solutions to allow us to update telegraf and all its other plugins. Unfortunately i don't see an easy solution to replace our filecount implementations, which makes it important to get a solution here.

Are there any plans to solve this issue during the next months?

[SamSkjord]

As an interim solution I've been using exec with a python script:

[[inputs.exec]]
  commands = [
    "python.exe exec_count.py --d D:/XMLexport",
  ]
  timeout = "10s"
  data_format = "influx"

exec_count.py

import os
import sys
import argparse

parser = argparse.ArgumentParser(description="filecount")
parser.add_argument("--d")
args = parser.parse_args()
dir_path = args.d


def count_files(dir_path): 
    if os.path.exists(dir_path) and os.path.isdir(
        dir_path
    ): 
        try:
            return sum(len(files) for _, _, files in os.walk(dir_path))
        except:
            pass
    else:
        pass


def print_to_stdout(*a):
    print(*a, file=sys.stdout)


print_to_stdout(
    "filecount " + dir_path + "=" + str(count_files(dir_path)) + "i"
)

Quick and dirty but it works

[St3f1n]

As an interim solution I've been using exec with a python script:

Thank you for this interim solution proposal.

However, i'd really prefer to get a fix of this issue to avoid the effort of such a workaround and a rough timeline for this would be appreciated.

[powersj]

Hi,

1.21.0, 1.22.0, 1.23.0 and 1.23.3 is working.
1.23.4 is broken.
The problem is present in the 32 and 64bit version.

Thank you for taking the time to try the different versions and help us narrow down the source of the change.

There were no changes to the filecount plugin itself between v1.23.3 and v1.23.4. Looking at the other relevant changes it appears that #11574 updated the version of godirwalk from 1.16.1 to 1.17.0.

The diff between those versions, shows primarily around changes to the Scan usage. In Telegraf, we use the Walk function. It appears that godirwalk's Walk does use Scan internally.

To help determine if the library change was in fact part of the issue, I have put up #12295 that reverts the versions back. Could someone who has run into this issue try the artifacts from that PR and let me know if it works?

Thanks!

[Tuxdiver]

I can confirm, that the filecount plugin on our "Windows Server 2019 Standard":

• 1.24.2 does not work
• 1.23.3 works
• your version from fix(inputs.filecount): revert library version #12295 works

Thanks for your investigations!

[powersj]

I have got a reproducer outside of Telegraf using code from the godirwalk own examples and filed karrick/godirwalk#70

I will wait to see if I get a response before we either land the revert

[eazywig]

The bug of filecount is still existing in 1.25.3, so workaround is to stay on telegraf version < 1.24.x.
As far as I understand the postings root cause has already been identified and fixed.
Nevertheless current version 1.25.3 still contain failing filecount, so it seems that fixed code is maybe not approved yet and even not merded into release branch.
May I kindly ask what is necessary to get the fix into release?

[powersj]

May I kindly ask what is necessary to get the fix into release?

Hmm it seems we reverted the library version and then dependabot turned around and updated the library :( I'll re-open, downgrade the library for v1.26.0. I'm really hoping my fix karrick/godirwalk#71 gets merged, but it has been a month.