The newest td-agent 4.1.0, Fluentd 1.12.1 does not detect log rotation

[S3ky]

Describe the bug

After the upgrade td-agent to the latest version 4.1.0 (Fluentd 1.12.1, CentOS 7) we found a bug, that Fluentd did not detect log rotation. We manually confirmed that it was working in the td-agent v.4.0.1.

To Reproduce

We reproduced it by making an mv on the log file and then restarting our service, which is producing log files. After we restarted our service, the service started writing to another file. During this process, I checked the logs of the td-agent, that there was no message, and I looked in the pos file and it was clear that there was still inode from the file we moved.

Expected behavior

FluentD will catch a rotated log file.

Your Environment

• Fluentd or td-agent version: td-agent 4.1.0, Fluentd 1.12.1
• Operating system: CentOS 7
• Kernel version: 3.10.0-1160.21.1.el7.x86_64

Your Configuration

<source>
    @type tail
    path /data/haproxy/log/haproxy.log
    pos_file /var/log/td-agent/haproxy.pos
    tag "redacted-haproxy@#{Socket.gethostname}"
    <parse>
        @type kv
        time_key timestamp
        time_type float
        keep_time_key true
    </parse>
</source>
<filter redacted-haproxy@**>
    @type record_transformer
    <record>
        PROGRAM haproxy
    </record>
</filter>
<source>
    @type systemd
    tag "redacted@#{Socket.gethostname}"
    path /var/log/journal
    read_from_head true
    <storage>
        @type local
        persistent true
        path /var/log/td-agent/systemd.pos
    </storage>
    <entry>
        fields_strip_underscores true
        fields_lowercase true
    </entry>
</source>
<filter redacted@**>
    @type record_transformer
    enable_ruby true
    <record>
        PROGRAM ${record["syslog_identifier"]}
    </record>
</filter>
<source>
    @type tail
    path /var/log/audit/audit.log
    pos_file /var/log/td-agent/audit.log.pos
    tag "redacted-audit@#{Socket.gethostname}"
    <parse>
        @type kvp
        time_key msg
        keep_time_key true
        audit_time_log true
    </parse>
</source>
<filter redacted-audit@**>
    @type record_transformer
    <record>
        PROGRAM audit
    </record>
</filter>
<filter **>
    @type record_transformer
    <record>
        ENVIRONMENT redacted
        COMPONENT redacted
    </record>
</filter>

<source>
@type monitor_agent
bind 127.0.0.1
port 24220
</source>

<match redacted**>
    @type copy
    <store>
        @type kinesis_streams
        stream_name "redacted"
        region eu-west-1
        <assume_role_credentials>
            role_arn "redacted"
            role_session_name "fluentd-#{Socket.gethostname}"
        </assume_role_credentials>
        <buffer>
            @type file_single
            path /var/log/td-agent/kinesis.buffer
            chunk_limit_size 5MB
            queued_chunks_limit_size 4096
            total_limit_size 16GB
            flush_mode default
            flush_thread_interval 0.1
            flush_thread_count 4
            flush_interval 5s
            flush_at_shutdown false
            overflow_action drop_oldest_chunk
        </buffer>
        <inject>
            hostname_key HOSTNAME
            tag_key TAG
            time_key UNIXTIME
            time_type float
        </inject>
    </store>
	<store>
        @type cloudwatch_logs
        @id out_cloudwatch_logs
        log_group_name /aws/ec2/redacted
        auto_create_stream true
        use_tag_as_stream true
        json_handler yajl
        region eu-west-1
        <buffer>
            @type file
            chunk_limit_size 256K
            flush_thread_interval 0.1
            flush_thread_count 4
            queued_chunks_limit_size 2048
            flush_interval 30s
            path /var/log/td-agent/cloudwatch.*.buffer
        </buffer>
    </store>
</match>

Your Error Log

No errors presented

[ashie]

in_tail of fluentd 1.12.0 and 1.12.1 has some bugs caused by #3182 , probably you met the same issue:

• 1.12.0: tail plugin crashes worker upon ENOENT (cannot stat() file) #3274
• Log forwarding stopped after "No such file or directory" error in in_tail #3224
• in_tail may not send rotated logs when mv is used to rotate #3292 probably it is the same case with you

They are fixed by the following pull requests:

• in_tail: expect ENOENT during stat (try to fix #3274 and #3224) #3275
• in_tail: Handle to send rotated logs when mv is used for rotating #3294

fluentd 1.12.2 ships them.
But td-agent isn't updated yet, we are preparing it now: fluent/fluent-package-builder#288

[ashie]

We've published td-agent 4.1.1 which bundles fluentd v1.12.3: https://www.fluentd.org/download
Please use it.