Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture unmatched lines #1421

Merged
merged 4 commits into from
Jan 25, 2017
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add a new option for in_tail called 'enable_catch_all'. When set to t…
…rue, all singular and multiline log lines that don't match the parser regex will be stored in an attribute called parse_fail, and eventually saved in the output destination.
  • Loading branch information
jitran authored and tranj3 committed Jan 17, 2017
commit 49ff636b15cf88bd0aa10f8f2eef2819c829332f
10 changes: 10 additions & 0 deletions lib/fluent/plugin/in_tail.rb
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ def initialize
config_param :read_lines_limit, :integer, default: 1000
desc 'The interval of flushing the buffer for multiline format'
config_param :multiline_flush_interval, :time, default: nil
desc 'Enable the option to capture unmatched lines.'
config_param :enable_catch_all, :bool, default: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

emit_unmatched_lines or emit_parse_failed_lines is more better.
enable_catch_all is unclear name for me.

desc 'Enable the additional watch timer.'
config_param :enable_watch_timer, :bool, default: true
desc 'The encoding after conversion of the input.'
Expand Down Expand Up @@ -357,6 +359,11 @@ def convert_line_to_event(line, es, tail_watcher)
record[@path_key] ||= tail_watcher.path unless @path_key.nil?
es.add(time, record)
else
if @enable_catch_all
record = {'parse_fail' => line}
record[@path_key] ||= tail_watcher.path unless @path_key.nil?
es.add(Fluent::EventTime.now, record)
end
log.warn "pattern not match: #{line.inspect}"
end
}
Expand Down Expand Up @@ -387,6 +394,9 @@ def parse_multilines(lines, tail_watcher)
lb = line
else
if lb.nil?
if @enable_catch_all
convert_line_to_event(line, es, tail_watcher)
end
log.warn "got incomplete line before first line from #{tail_watcher.path}: #{line.inspect}"
else
lb << line
Expand Down