Logparser : Epoch convertion impossible with SYSLOGTIMESTAMP Grok pattern

[wolfyzvf]

Bug report

Relevant telegraf.conf:

[[inputs.logparser]]
files = ["/root/samples/*mail.log"]
from_beginning = true

[inputs.logparser.grok]
patterns = ["%{SYSLOGTIMESTAMP:timestamp:ts-“Mon Jan 2 15:04:05”} %{SYSLOGHOST:hostname} mta-in/milter[%{MILTER_PROCESSID}]: %{POSTFIX_QUEUEID:postfix_queue_id}: from=<%{DATA:postfix_from}>, firstto=<%{DATA:postfix_firstto}>, nrcpt=%{DATA:postfix_nrcpt}, size=%{DATA:postfix_msg_size}, score=%{DATA:score}, state=%{DATA:state:int}, status=%{DATA:status:tag}, level=%{DATA:level}, actions=%{DATA:actions}, subject="%{DATA:subject}", cause="%{DATA:cause}""]

Name of the outputted measurement name.

measurement = “test”
…
custom_patterns = ‘’‘
POSTFIX_QUEUEID ([0-9A-F]{6,}|[0-9a-zA-Z]{15,})
MILTER_PROCESSID ([0-9]{5,})
’’’

System info:

• Telegraf v1.4.2 (git: release-1.4 0cc5fc0)
• InfluxDB 1.3.6-1
• Docker official image

Steps to reproduce:

1. Deploy Telegraf using docker with the
2. Inject an existing log in SYSLOGTIMESTAMP format
   For ex. Sep 25 08:27:29 mailhost postfix/cleanup[17659]: ABFF15C..
3. The time is not correctly calculated.

Expected behavior:

The syslog_timestamp should be converted into time in Influxdb

Actual behavior:

As the year is missing, As Telegraf can't find the year he set it to 0 which generate a wrong time in influxdb

Additional info:

[Include gist of relevant config, logs, etc.]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Like the timezone option we should be able to set manually the year.
Assuming automatically the year could cause some trouble when we change of it.
if a person has more than 2-3 years of logs to inject into influxdb he should pass this option in each logparser.

Current behavior:

Telegraf assume automatically 0 has year if it's not provided

Desired behavior:

By setting manually the date, Telegraf will be able to build the correct time in influxdb.

Use case: [Why is this important (helps with prioritizing requests)]

This is a standard output for the syslog, this pattern should be recognize per default as many solution use it.

Forum Discussion

https://community.influxdata.com/t/custom-timestamp-for-syslogtimestamp-and-telegraf/2798/

[EduardoOliveira]

Any news or workaround?
Thanks

[danielnelson]

I added a new timestamp format for parsing syslog timestamps. You can use it with%{SYSLOGTIMESTAMP:timestamp:ts-syslog} on the nightly builds (give them a day to incorporate the change).