in_tail: Handle records in the correct order on file rotation

[fujimotos]

This is the fix for the issue reported by the bug report #1055.

Problem

It turned out that this bug was a matter of the calling order of functions in TailWatcher.on_rotate().

In particular, when TailWatcher detects a log rotation and immediately finds a new file, it performs the following steps:

1. Create a new TailWatcher for the new file.
   • In this process, it reads the new file and emit its contents.
   • This all happens before proceeding to (2).
2. Dispatch the on_notify() for the archived file.
   • In this process, it reads the content of the old file and emit its contents.

So naturally, newer log records (written to the new log file) will be emitted before older records in the archived log file.

Solution

This patch fixes it by enforcing flush of older log records before performing the step (1).

[fujimotos]

Why doesn't this bug occur on every rotation?

Note that this bug occurs only when TailInput is slow to notice the file rotation.

In most cases, it does not occur because a log rotation is detected immediately after
the file rename (= before the new log file is created). In such a case, a file rotation
will be handled as follows.

1. Detect the rename of the log file and set up a new TailWatcher.
   • In this step, the new file is NOT created yet, so nothing is emitted.
2. Dispatch the on_notify() for the archived log file.
3. Detect the creation of the new log file.
   • In this step, the content of the new file gets processed.

As you can see, TailInpout handles the content of the old file in the step 2 and the
content of the new file in the step 3, so the output order of records is preserved.

[repeatedly]

Thanks for the investigation and patch. Just merged :)

[fujimotos]

Thanks for merging it!