Conditionally check S3 artifacts #175
Conversation
|
+1 |
|
Please add tests. |
| When it is false, | ||
| s3_object_key_format will be %{path}/events/%{date_slice}/%{time_slice}.%{file_extension} | ||
| where, time_slice will be in hhmmss format, so that each object will be unique. | ||
|
|
| When it is false, | ||
| s3_object_key_format will be %{path}/events/%{date_slice}/%{time_slice}.%{file_extension} | ||
| where, time_slice will be in hhmmss format, so that each object will be unique. | ||
|
|
There was a problem hiding this comment.
And adding s3 policy example is better for this mode.
|
After I added the test cases, build got failed. So I delete those test case. Even it is failing. Can you please give some light on this? |
|
Maybe v0.14.9 changes affect it. I will check it later. |
|
Thanks Masahiro |
|
Hi Masahiro, did you got the time to look into the issue? |
|
I found the cause and I have already opened an PR on fluentd: fluent/fluentd#1319 |
|
The test failure are not related your test. |
|
Hi Masahiro, I have added test cases and introduced the mock function to work. Please check |
| @@ -137,6 +147,14 @@ def test_path_slicing_utc | |||
| slice = path_slicer.call(path) | |||
| assert_equal slice, Time.now.utc.strftime("log/%Y/%m/%d") | |||
| end | |||
|
|
|||
| def test_hms_slicing | |||
There was a problem hiding this comment.
This tests Ruby's class, not plugin feature. Not needed.
| assert_equal '131415', Time.parse("2011-01-02 13:14:15").strftime("%H%M%S") | ||
| end | ||
|
|
||
| def test_hms_slicing_utc |
| previous_path = s3path | ||
| end while @bucket.object(s3path).exists? | ||
| else | ||
| @s3_object_key_format = "%{path}/%{date_slice}_%{hms_slice}.%{file_extension}" |
There was a problem hiding this comment.
Don't update instance variable in write.
Update should be in configure or start.
| @@ -142,10 +146,12 @@ def configure(conf) | |||
| @path_slicer = Proc.new {|path| | |||
| Time.now.strftime(path) | |||
| } | |||
| @hms_slicer = Time.now.strftime("%H%M%S") | |||
There was a problem hiding this comment.
This value is never updated in write, so all stored objects share same @hms_slicer value in the path.
Is this expected?
There was a problem hiding this comment.
That's my fault. To refactor I missed that portion. Corrected
| else | ||
| @path_slicer = Proc.new {|path| | ||
| Time.now.utc.strftime(path) | ||
| } | ||
| @hms_slicer = Time.now.utc.strftime("%H%M%S") |
| @@ -118,6 +120,14 @@ def test_configure_with_hex_random_length | |||
| create_driver(conf + "\nhex_random_length 16\n") | |||
| end | |||
| end | |||
|
|
|||
| def test_configure_with_no_check_on_s3 | |||
| conf = CONFIG.clone | |||
There was a problem hiding this comment.
|
After updated tests, please squash |
|
Could you merge "fix" commits into one? |
|
Hi Masahiro, The previous fixes is what when I try to fix the test cases. And all those changes are in the current commit itself. Do I still need to merge previous "fix" commits into one? |
|
Yes because if merge current commits into master, master branch has lots of useless |
…licy Condition to skip S3 bucket and object existence during runtime, so that more tight and granular AWS policy can be provided on the instance where fluentd is running. With existing one, we need minimum S3:ListBucket, S3:GetObjectVersion, and S3:PutObject policy. But if we apply this two configure parameter, fluentd will work with only S3:PutObject policy. For this to happen the below mentioned config parameters must be set: check_bucket => false check_object => false
|
Done |
|
Where is the changes of plugin? |
Condition to skip S3 bucket and object existence during runtime, so that more tight and granular AWS policy can be provided on the instance where fluentd is running. With existing one, we need minimum S3:ListBucket, S3:GetObjectVersion, and S3:PutObject policy. But if we apply this two configure parameter, fluentd will work with only S3:PutObject policy. For this to happen the below mentioned config parameters must be set: check_bucket => false check_object => false
Condition to skip S3 bucket and object existence during runtime, so that more tight and granular AWS policy can be provided on the instance where fluentd is running.
With existing one, we need minimum S3:ListBucket, S3:GetObjectVersion, and S3:PutObject policy. But if we apply this two configure parameter, fluentd will work with only S3:PutObject policy.
For this to happen the below mentioned config parameters must be set:
check_bucket => false
check_object => false