fluent-plugin-webhdfs fails test with fluentd v0.14.0.rc.3

[okkez]

How to reproduce

$ git clone git@github.com:fluent/fluent-plugin-webhdfs.git
$ cd fluent-plugin-webhdfs
$ echo 'gem "fluentd", "=0.14.0.rc.3"' >> Gemfile
$ bundle install
$ bundle exec rake test

Result

Some test fails as below.

Error: test_default(WebHDFSOutputTest::ConfigureTest::PathFormatTest): Fluent::ConfigError: 'path' parameter is required
fluentd/lib/fluent/config/section.rb:140:in `block in generate'
fluentd/lib/fluent/config/section.rb:127:in `each_pair'
fluentd/lib/fluent/config/section.rb:127:in `generate'
fluentd/lib/fluent/configurable.rb:68:in `configure'
fluentd/lib/fluent/plugin/base.rb:39:in `configure'
fluentd/lib/fluent/plugin/buffer.rb:86:in `configure'
fluentd/lib/fluent/plugin/buf_file.rb:57:in `configure'
fluentd/lib/fluent/plugin/output.rb:267:in `configure'
fluentd/lib/fluent/plugin_helper/event_emitter.rb:60:in `configure'
fluentd/lib/fluent/compat/output.rb:578:in `configure'
2.3.0/lib/ruby/gems/2.3.0/gems/fluent-mixin-config-placeholders-0.3.1/lib/fluent/mixin/config_placeholders.rb:106:in `configure'
2.3.0/lib/ruby/gems/2.3.0/gems/fluent-mixin-plaintextformatter-0.2.6/lib/fluent/mixin/plaintextformatter.rb:21:in `configure'
fluent-plugin-webhdfs/lib/fluent/plugin/out_webhdfs.rb:126:in `configure'
fluentd/lib/fluent/test/base.rb:71:in `configure'
fluent-plugin-webhdfs/test/plugin/test_out_webhdfs.rb:13:in `create_driver'
fluent-plugin-webhdfs/test/plugin/test_out_webhdfs.rb:100:in `test_default'
      97:
      98:     class PathFormatTest < self
      99:       def test_default
  => 100:         d = create_driver
     101:         assert_equal '/hdfs/path/file.%Y%m%d.log', d.instance.path
     102:         assert_equal '%Y%m%d', d.instance.time_slice_format
     103:         assert_equal '/hdfs/path/file.20120718.log', d.instance.path_format('20120718')

path parameter is not required in plugin definition.

I think something wrong around https://github.com/fluent/fluentd/blob/master/lib/fluent/config/section.rb#L140

[tagomoris]

path is required parameter for webhdfs output plugin.
This problem seems that path is provided as CONF, but #configure raises error.

[tagomoris]

Ah, What raises error is Buffer, and it raises path not found in <buffer> subsection which is automatically generated in #configure of Fluent::Compat::BufferedOutput.

[okkez]

Do we need fix fluent-plugin-webhdfs test adding <buffer> subsection to test CONFIG?

[tagomoris]

path in <buffer> sections should be set from buffer_path (if exists) for v0.12 plugins. It is required parameter for file buffers.
But webhdfs output uses memory buffer in default, so buffer_path should not be required. Weird.

[tagomoris]

I found that:

• webhdfs output overrides default value of buffer_type to memory
• compat layer's TimeSlicedOutput defines default values of:
  • buffer_type as file
  • <buffer> @type </buffer> as file

If buffer_path is missing in configuration, Fluent::Plugin::Output uses default value of <buffer> @type, not buffer_path. So it will be instantiated as file, and file buffer plugin requires path parameter.