Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi worker processing #1386

Merged
merged 34 commits into from
Jan 10, 2017
Merged
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
674fb9c
add "workers" configuration parameter in system config
tagomoris Dec 21, 2016
68065f1
add method #multi_workers_ready? to show that plugin is available wit…
tagomoris Dec 21, 2016
06c9be0
fix to raise error if non-multi-worker-ready plugins are configured u…
tagomoris Dec 21, 2016
3d8924f
add test helper method to set configurations for multi workers
tagomoris Dec 22, 2016
72f89f7
support multi workers in file buffer
tagomoris Dec 22, 2016
05c32c2
fix storage_local to support multi workers
tagomoris Dec 22, 2016
34d6204
set worker id during tests in default
tagomoris Dec 22, 2016
813ae40
fix to raise error for invalid worker id
tagomoris Dec 22, 2016
fa0242c
add a feature to inject worker id to let users know about it
tagomoris Dec 22, 2016
3b3bd9d
enable "workers"
tagomoris Dec 22, 2016
5f989b5
mark multi-workers-ready plugins
tagomoris Dec 22, 2016
9277271
fix to show worker id when operating worker processes
tagomoris Dec 26, 2016
7436df7
worker ids are shown as integer in logs
tagomoris Dec 26, 2016
b71cd9e
add "log type" and "process type" to control whether logs should be s…
tagomoris Dec 27, 2016
d170177
it is too much to show all before_shutdown/close messages about all p…
tagomoris Dec 27, 2016
aa65ca2
reduce # of log messages in loop/timers
tagomoris Dec 27, 2016
15c51a2
fix in_forward to work with multi workers
tagomoris Dec 27, 2016
2a17ed9
refactor out_file write operatoin for further fix (worker synchroniza…
tagomoris Dec 27, 2016
d3264bb
fix out_file implementation to lock path during #write operation for …
tagomoris Dec 27, 2016
f825845
fix bug not to show "shutting down" log message
tagomoris Dec 27, 2016
3314d7e
add tests to execute 2 workers
tagomoris Dec 27, 2016
fcd8cc1
remove the features #detach_process and #detach_multi_process
tagomoris Dec 27, 2016
ab7e59e
remove warnings
tagomoris Dec 28, 2016
97c535a
freeze non-modified string
tagomoris Jan 4, 2017
abaccf9
make error message more descriptive
tagomoris Jan 4, 2017
44c06af
make code readable
tagomoris Jan 4, 2017
8b98086
remove useless comment
tagomoris Jan 4, 2017
9b7170a
fix to read updated parameter values via command line options
tagomoris Jan 5, 2017
21fb0e9
update tests to use command line options correctly
tagomoris Jan 5, 2017
1e026a4
log level should be configured via params (updated params via command…
tagomoris Jan 5, 2017
ec6a29a
enable shared socket only when multi workers configured
tagomoris Jan 10, 2017
c25d14d
remove invalid code which are duplicated through rebase
tagomoris Jan 10, 2017
615d3a1
remove unused variable
tagomoris Jan 10, 2017
7eb3177
fix to warn about logging configuration only on worker0, and fix test…
tagomoris Jan 10, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove warnings
  • Loading branch information
tagomoris committed Jan 10, 2017
commit ab7e59e3248073e2e2393d685bdba9d79568790a
4 changes: 2 additions & 2 deletions test/test_plugin.rb
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ def emit_error_event(tag, time, record, error); end
filter: ['plugin_test_dummy', DummyFilter, :new_filter], # with DummyParser and DummyFormatter
output2: ['plugin_test_dummy2', Dummy2Output, :new_output], # with Dummy1Buffer
)
test '#configure does not raise any errors if plugins and its owned plugins are ready for multi workers' do |(type, klass, new_method)|
test '#configure does not raise any errors if plugins and its owned plugins are ready for multi workers' do |(type, _klass, new_method)|
conf = config_element()
instance = Fluent::Plugin.__send__(new_method, type)
if instance.respond_to?(:context_router=)
Expand All @@ -236,7 +236,7 @@ def emit_error_event(tag, time, record, error); end
input2: ['plugin_test_dummy2', Dummy2Input, :new_input, 'storage', 'plugin_test_dummy2', Dummy2Storage],
output2: ['plugin_test_dummy2', Dummy2Output, :new_output, 'buffer', 'plugin_test_dummy2', Dummy2Buffer],
)
test '#configure raise configuration error if configured owned plugins are not ready for multi workers' do |(type, klass, new_method, subsection, subsection_type, problematic)|
test '#configure raise configuration error if configured owned plugins are not ready for multi workers' do |(type, _klass, new_method, subsection, subsection_type, problematic)|
conf = config_element('root', '', {}, [config_element(subsection, '', {'@type' => subsection_type})])
instance = Fluent::Plugin.__send__(new_method, type)
if instance.respond_to?(:context_router=)
Expand Down