Skip to content

Commit

Permalink
Consider timezone when calculate timekey
Browse files Browse the repository at this point in the history
Because the timestamp of the midnight cannot be divisible by 86400 in
localtime (JST). The timestamp of the midnight is divisible by 86400
only in UTC. Therefore we must consider offset from UTC in localtime.

For example, at `2018-07-04 01:23:23 +0900`:

If timekey is 86400 and path template is `/log/%Y%m%d.log`.
In previous version, extract path template to `/log/20180703.log`.
In this version, extract path template to `/log/20180704.log`.

Fix fluent#1986

Signed-off-by: Kenji Okimoto <okimoto@clear-code.com>
  • Loading branch information
okkez committed Jul 5, 2018
1 parent 84c2194 commit 6921b47
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 6 deletions.
19 changes: 13 additions & 6 deletions lib/fluent/plugin/output.rb
Original file line number Diff line number Diff line change
Expand Up @@ -803,20 +803,17 @@ def metadata(tag, time, record)
if !@chunk_key_time && !@chunk_key_tag
@buffer.metadata()
elsif @chunk_key_time && @chunk_key_tag
time_int = time.to_i
timekey = (time_int - (time_int % @buffer_config.timekey)).to_i
timekey = calculate_timekey(time)
@buffer.metadata(timekey: timekey, tag: tag)
elsif @chunk_key_time
time_int = time.to_i
timekey = (time_int - (time_int % @buffer_config.timekey)).to_i
timekey = calculate_timekey(time)
@buffer.metadata(timekey: timekey)
else
@buffer.metadata(tag: tag)
end
else
timekey = if @chunk_key_time
time_int = time.to_i
(time_int - (time_int % @buffer_config.timekey)).to_i
calculate_timekey(time)
else
nil
end
Expand All @@ -825,6 +822,16 @@ def metadata(tag, time, record)
end
end

def calculate_timekey(time)
time_int = time.to_i
if @buffer_config.timekey_use_utc
(time_int - (time_int % @buffer_config.timekey)).to_i
else
offset = Time.at(time_int).utc_offset
(time_int - ((time_int + offset)% @buffer_config.timekey)).to_i
end
end

def chunk_for_test(tag, time, record)
require 'fluent/plugin/buffer/memory_chunk'

Expand Down
33 changes: 33 additions & 0 deletions test/plugin/test_out_file.rb
Original file line number Diff line number Diff line change
Expand Up @@ -557,6 +557,39 @@ def parse_system(text)
check_gzipped_result(path, formatted_lines * 3)
end

test 'append when JST' do
time = event_time("2011-01-02 03:14:15+09:00")
formatted_lines = %[2011-01-02T03:14:15+09:00\ttest\t{"a":1}\n] + %[2011-01-02T03:14:15+09:00\ttest\t{"a":2}\n]

write_once = ->(){
d = create_driver %[
path #{TMP_DIR}/out_file_test
compress gz
append true
<buffer>
timekey_use_utc false
</buffer>
]
d.run(default_tag: 'test'){
d.feed(time, {"a"=>1})
d.feed(time, {"a"=>2})
}
d.instance.last_written_path
}

path = write_once.call
assert_equal "#{TMP_DIR}/out_file_test.20110102.log.gz", path
check_gzipped_result(path, formatted_lines)

path = write_once.call
assert_equal "#{TMP_DIR}/out_file_test.20110102.log.gz", path
check_gzipped_result(path, formatted_lines * 2)

path = write_once.call
assert_equal "#{TMP_DIR}/out_file_test.20110102.log.gz", path
check_gzipped_result(path, formatted_lines * 3)
end

test '${chunk_id}' do
time = event_time("2011-01-02 13:14:15 UTC")
formatted_lines = %[2011-01-02T13:14:15Z\ttest\t{"a":1}\n] + %[2011-01-02T13:14:15Z\ttest\t{"a":2}\n]
Expand Down

0 comments on commit 6921b47

Please sign in to comment.