CSV parsing issue

[miurlit]

Since version 1.21 there seems to be an issue with CSV files, I am getting a parsing error:

Jan 3 02:14:20 raspberrypi telegraf[426]: 2022-01-03T01:14:20Z E! [inputs.file] Error in plugin: parsing time "dd/MM/yyyy HH:mm:ss z" as "02/01/2006 15:04:05 -0700": cannot parse "dd/MM/yyyy HH:mm:ss z" as "02"

If revert back to 1.20.4 no error occurs with the same CSV file.

Thanks

[jonilala796]

I have the same problem. Going down the version to 1.20.4 also fixed it.
Here is the config for the input I use:

[[inputs.file]]

files = ["/path/to/file.csv"]
name_override = "custom"
data_format = "csv"
csv_header_row_count = 1
csv_delimiter = ";"
csv_timestamp_column = "dd/MM/yyyy HH:mm:ss"
csv_timestamp_format = "02/01/2006 15:04:05"
csv_timezone = "Europe/Berlin"

[gwlqt]

I have the same problem with version 1.21.2. Fixed by downgrading to 1.20.4.

Logs:
E! [inputs.http::alias] Error in plugin: [url=http://]: strconv.ParseInt: parsing "TimeInt": invalid syntax

Config:

[[inputs.http]]
  urls = ["http://"]
  data_format = "csv"
  csv_delimiter = ";"
  csv_header_row_count = 1
  csv_tag_columns = [""]
  csv_timestamp_column = "TimeInt"
  csv_timestamp_format = "unix"

[sspaink]

Can you provide a CSV file your trying to parse so we can reproduce this issue? There were changes made to the parser from v1.20->v1.21, but it isn't obvious what is causing this issue at first glance.

[gwlqt]

Here is a CSV file: example.csv.
Thanks for looking into this!

[sspaink]

I think I found the problem in this PR: #10484, it only happens when you run Gather twice the second time the code accidentally changed the configured csv_header_row_count so it would try to parse the header as data. There is still some issue with some of the test expecting the configured value to change that I need to work through.

[miurlit]

Sorry, but it seems the issue still persists in 1.21.3?!
I still geht the same error as in my original post...

[jonilala796]

Yeah the PR was closed without any actions taken.
Here is the sample data from me: https://pastebin.com/VD4r5U8x

[powersj]

@miurlit @jonilala796,

Thanks, we know it is still an open issue. The PR discovered it will require a different approach to fix so it was closed.

[miurlit]

@miurlit @jonilala796,

Thanks, we know it is still an open issue. The PR discovered it will require a different approach to fix so it was closed.

Ok, thanks for clarifying.

[srebhan]

Hey @miurlit, @jonilala796 @gwlqt, I cooked up an approach in #10575 and would be happy to get feedback if that solves your issues!?!? Could you please add a comment to the PR on your results? Thanks!

[miurlit]

@srebhan sorry, was not actively following this conversation, but I am afraid that the issue still persists with the latest release (1.21.4)