Log rotation on Windows

[ashie]

From fluent/fluentd#3538

Describe the bug

The log file (C:\opt\td-agent\td-agent.log) is increasing continuously, how to put a limit ??

There is some configuration like file rotate and there is a command however we have a fluentd running as windows service, so if there is any configuration could you please suggest either in conf file or while running the fluentd service from powershell.

To Reproduce

Keep fluentd service running and process different messages the file size keeps increasing.

Expected behavior

The old contents in the file should be deleted after a certain limit is reached.

Your Environment

- Fluentd version:
- TD Agent version:
- Operating system:
- Kernel version:

Your Configuration

Normal configuration

Your Error Log

No Errors it's physical size issue.

Additional context

No response

[ashie]

refs:

• Enable system.log.rotate_... system configuration fluentd#3352
• Windows Service logs don't rotate, no way to set rotation for the service fluentd-docs-gitbook#246
• 1.0: Document rotate_age and rotate_size in system config section fluentd-docs-gitbook#316
• MSI: add log rotation setting #284

[ashie]

Should we add a default setting?

[kenhys]

Should we add a default setting?

It may be better to do so.

[harsh288]

Hello @ashie @kenhys

I tested with the below configuration, it's not working, 1048 means 1kb right ?? After happens if that limit is reached, my file size is more than 2Mb and it's still growing.

<system>
   <log>
     rotate_age 1
     rotate_size 1048
   </log>
</system>

I was confused where to the above settings so I have tried in 2 locations:

1. C:\opt\td-agent\lib\ruby\gems\2.7.0\bundler\gems\fluentd-b6e76c09d60e\Fluent.conf
2. C:\opt\td-agent\etc\td-agent\td-agent.conf

Below is my configuration.

• Fluentd version: 4.1.1
• TD Agent version: 4.1.1
• Operating system: Windows Server 2019

[kenhys]

@harsh288

To understand the situation (the reason why it's not working) correctly, could you show us the result of td-agent.bat --version?
It will show the actual version of TD Agent and Fluentd respectively.

Something like this:

[harsh288]

Hi @kenhys

After doing a fluentd update it started working, I kept rotate size as 104800 and rotate age 10, so, after the size is reached 103kb, the logs are not seen, I failed to understand how the whole process works, I assumed a new file is created once the file size limit is reached and based on the rotate age only the required number of logs will be kept and rest will be deleted, could you please help me to understand.

Below is the version detail.
td-agent 4.1.1 fluentd 1.12.3

[kenhys]

@harsh288

No, It's not working.

As I noted before, fluentd 1.12.3 will not accept rotate settings.

You can ensure by launch fluentd manually: (instead of launching as a service)

rotate.conf

<system>
  <log>
    rotate_age 10
    </log>
</system>

You can see a warning message.

> fluentd -c rotate.conf
2021-11-01 16:41:58 +0900 [info]: starting fluentd-1.12.3 pid=14780 ruby="2.7.3"
2021-11-01 16:41:58 +0900 [info]: spawn command to main:  cmdline=["C:/Ruby27-x64/bin/ruby.exe", "-Eutf-8", "C:/Ruby27-x64/bin/fluentd", "-c", ".\\rotate.conf", "--under-supervisor"]
2021-11-01 16:42:00 +0900 [warn]: parameter 'rotate_age' in <log>
  rotate_age 10
</log> is not used.

One thing you need to do is: upgrade Fluentd.

[tivanov-qb]

@kenhys this is old PR, but running td-agent 4.5.1 with fluentd 1.16.2 does not automatically rotate logs on my end

<system>
  <log>
    rotate_age 2
    rotate_size 57344 # Rotate logs at 56Kb
  </log>
</system>

here is screenshot for ref

[daipom]

@tivanov-qb I see rotate_age 0 setting on the image. Is it intended?

[tivanov-qb]

@daipom i was trying to get the rotation working and appearently this is what I had in the file when I took the screenshot. You can also ignore the comment for 50MB, as you can see it is set to Kbs, but does not work :(

[daipom]

@tivanov-qb Could you try removing rotate_age 0 setting?