Parser ignoring timezone #2941
Comments
|
This is due to Windows' While We'll need to add a special logic to retain timezone info on Windows. |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
Will there be a fix? It is more or less unusable on Windows. I found a workaround using a lua filter but that is not a very good solution. |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
This issue was closed because it has been stalled for 5 days with no activity. |
|
This issue was closed because it has been stalled for 5 days with no activity. |
|
@fujimotos Any update on this? |
|
Any update on this |
|
help |
|
Looks like this remains an issue. Based on some investigation done by @franciscovalentecastro, it affects Windows users that uses a parser with |
|
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the |
|
This issue was closed because it has been stalled for 5 days with no activity. |
|
This is still a problem. Perhaps a maintainer could add |
@julianklose Can you give an example of the workaround? |
|
@ThomasBeavers My workaround: filter.lua: function correct_timezone(tag, timestamp, record)
local utcdate = os.date("!*t", timestamp)
local localdate = os.date("*t", timestamp)
localdate.isdst = false
local offset = os.difftime(os.time(localdate), os.time(utcdate))
return 1, timestamp - offset, record
end |
|
I am having this issue as well. I am running some test scenarios before starting to deploy to our environments. The two first lines should be parsed, as if in my local time zone Here is my config files: I am running tests using bats-core. My test: ### time_prefixed_entries.bats
#!/usr/bin/env bats
load test_helper.bash
@test "time prefixed log entries should be parsed correctly" {
setup_input "time-prefixed-entries.log"
run fluent-bit -c charts/fluent-bit/build/conf/fluent-bit.conf
[ "$status" -eq 0 ]
assert_output_lines 6
IFS=$'\n' # iterate lines instead of words
output_lines="$(cat "$output_file")"
for line in $output_lines
do
json_line="$(echo "$line" | cut -d ' ' -f 2-)"
#NOTE: gdate is gnu-date app for mac
TZ="Europe/Stockholm" gdate -d @"$(echo "$json_line" | jq '.[0]')" +%Y-%m-%dT%H:%M:%S.%N
echo "$json_line" | jq '.[0]'
echo "$json_line" | jq '.[1].time'
echo "$json_line" | jq '.[1].log'
done
exit 1
}
### test_helper.bash
# global constants
git_root="$(git rev-parse --show-toplevel)"
output_file="$git_root/test/output.log"
input_file="$git_root/test/data/current_input.log"
readonly git_root
readonly output_file
readonly input_file
# setup & teardown
setup_input() { # - must be run manually
cp "$git_root/test/data/$1" "$input_file"
}
# setup() {} - runs before each test
teardown() { # - runs after each test
rm -f "$input_file"
rm -f "$output_file"
}
# setup_file() {} - runs before each bats file
# teardown_file() {} - runs after each bats file
# setup_suite() {} - runs once before starting testing
# teardown_suite() {} - runs once after ending testing
# helper functions
error() {
echo "$1"
exit 1
}
assert_output_lines() {
local expected_lines
expected_lines="$1"
local num_lines
num_lines="$(wc -l < "$output_file" | xargs )"
[ "$num_lines" -eq "$expected_lines" ] || error "expected 1 output lines, but got $num_lines"
}
assert_equal() {
local expected
local actual
expected="$1"
actual="$2"
[ "$1" == "$2" ] || error "expected value $expected, but got $actual"
}Here is the output, which shows the unexpected results: With this setup, using fluent-bit version 1.9.7 This should be fully reproducible with the data I've provided here. |
My answer above is from macos which also seems to be affected by this. EDIT: macos does support GMTOFF |
|
@cosmo0920 do you know if there was a workaround for this on Windows when implementing |
|
On Windows, we can use our usage: https://github.com/fluent/fluent-bit/blob/master/plugins/in_winevtlog/pack.c#L197 |
|
Thanks for taking a look @cosmo0920 . Do I understand it correctly that if we use the winevtlog input plugin instead of in_winlog, the problem is solved? Is |
No, they are still supported like as fluent-plugin-windows-eventlog.
Presumably, Yes. in_winevtlog uses localtime to emit its gathered EventLogs. |
|
Hello, I did a ton of test, and the result is not great. We are using the LUA from #2941 (comment) And even with this, the sent timestamp is wrong, sometimes : From the Fluentbit output : Capture from the destination Rsyslog : The timestamp received is not the same as the one seen in the stdout sometime the right one is sent, sometimes not. |
|
We use Because We identified a partial workaround for this, which is to set the environment variable We also expose JSON and regex parsers to our users who are free to configure time formats including We really need a proper fix for this issue. Fundamentally it is a bug with the parsers. Switching to |
|
We should be able to wrap We can then use either |
|
I applied @igorpeshansky's idea and came up with a fix here. I'll clean up my commits and then open some PRs. |
I'm using windows release td-agent-bit-1.6.10-win32. The parser is ignoring the timezone set in the logs.
Here a simple example using the default apache parser:
My logs:
The result:
As you can see, every entry was treated as if it was in "+0000".
The text was updated successfully, but these errors were encountered: