From 6f54acd806e382acf1baae011c40d0562dc1476f Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Tue, 30 Apr 2024 18:30:14 +0900 Subject: [PATCH 1/8] blog: Add Fluentd v1.17.0 release announcement (WIP) Signed-off-by: Daijiro Fukuda --- ...40430_fluentd-v1.17.0-has-been-released.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 content/blog/20240430_fluentd-v1.17.0-has-been-released.md diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md new file mode 100644 index 00000000..1a1207cb --- /dev/null +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -0,0 +1,49 @@ +# Fluentd v1.17.0 has been released + +Hi users! + +We have released v1.17.0 on 2024-04-30. ChangeLog is [here](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430). + +This release is a new release of v1.17 series. +In this release, we added some new features for some plugins, and fixed bugs of Parser. + +## Enhancement + +### `in_tail`: Add `glob_policy` option for expanding glob capability of `path` and `exclude_path` + +In this release, we added a new option [glob_policy](https://docs.fluentd.org/input/tail#glob_policy) for [in_tail](https://docs.fluentd.org/input/tail) plugin. + +In previous versions, we can use only `*` in glob patterns for [path](https://docs.fluentd.org/input/tail#path) and [exclude_path](https://docs.fluentd.org/input/tail#exclude_path) option. + +Example: + +``` +path /path/to/* +exclude_path ["/path/to/*.gz", "/path/to/*.zip"] +``` + +From this version, we can also use `[]`, `?` and `{}` in glob patterns depending on the `glob_policy` option. + +Example: + +``` +path "[0-1][2-3].log" +glob_policy extended +``` + +Please see [the document](https://docs.fluentd.org/input/tail#glob_policy) and [#4401](https://github.com/fluent/fluentd/pull/4401) for more information. + +### `out_http`: Support AWS Signature Version 4 authentication + +### `out_http`: Add option to reuse connections + +### `in_http`: Recognize CSP reports as JSON data + +## Bug Fixes + +### Make sure parser returns hash + +Enjoy logging! + +TAG: Fluentd Announcement +AUTHOR: clearcode From 60522ff5cfeed193187e64f2963c6f37e121a768 Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Tue, 14 May 2024 17:44:22 +0900 Subject: [PATCH 2/8] Apply suggestions from code review Signed-off-by: Daijiro Fukuda Co-authored-by: Hiroshi Hatake --- content/blog/20240430_fluentd-v1.17.0-has-been-released.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index 1a1207cb..7d5d80c7 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -5,7 +5,7 @@ Hi users! We have released v1.17.0 on 2024-04-30. ChangeLog is [here](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430). This release is a new release of v1.17 series. -In this release, we added some new features for some plugins, and fixed bugs of Parser. +In this release, we added some new features for some plugins and fixed bugs of Parser. ## Enhancement @@ -22,7 +22,7 @@ path /path/to/* exclude_path ["/path/to/*.gz", "/path/to/*.zip"] ``` -From this version, we can also use `[]`, `?` and `{}` in glob patterns depending on the `glob_policy` option. +From this version, we can also use `[]`, `?`, and `{}` in glob patterns depending on the `glob_policy` option. Example: From 18343904132b2beb9f8fd95de5be96fabd0ab76e Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Wed, 15 May 2024 18:42:14 +0900 Subject: [PATCH 3/8] Add description for enhancements Signed-off-by: Daijiro Fukuda --- ...40430_fluentd-v1.17.0-has-been-released.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index 7d5d80c7..61887d0c 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -35,14 +35,36 @@ Please see [the document](https://docs.fluentd.org/input/tail#glob_policy) and [ ### `out_http`: Support AWS Signature Version 4 authentication +In this release, we added a new option `aws_sigv4` for the [method](https://docs.fluentd.org/output/http#method) setting of [out_http](https://docs.fluentd.org/output/http) plugin. + +By using this option, `out_http` can use [AWS Signature Version 4](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). + +For example, this allows `out_http` to write to [Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html). + +Please see [the document](https://docs.fluentd.org/output/http#method) and [#4459](https://github.com/fluent/fluentd/pull/4459) for more information. + ### `out_http`: Add option to reuse connections +In this release, we add a new option [reuse_connections](https://docs.fluentd.org/output/http#reuse_connections) for [out_http](https://docs.fluentd.org/output/http) plugin. + +This option will improve performance of `out_http`. + +Please see [the document](https://docs.fluentd.org/output/http#reuse_connections) and [#4330](https://github.com/fluent/fluentd/pull/4330) for more information. + ### `in_http`: Recognize CSP reports as JSON data +In this release, we make the data of the request where the `Content-Type` is `application/csp-report` be considered JSON by default. + +Now, `in_http` can receive [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)'s report by default. + +Please see [#4282](https://github.com/fluent/fluentd/pull/4282) for more information. + ## Bug Fixes ### Make sure parser returns hash +TODO + Enjoy logging! TAG: Fluentd Announcement From b2aab89e463ab002b05a2cd5623ac6a9c30e0b35 Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Thu, 16 May 2024 10:05:06 +0900 Subject: [PATCH 4/8] Improve expression Signed-off-by: Daijiro Fukuda Co-authored-by: Hiroshi Hatake --- content/blog/20240430_fluentd-v1.17.0-has-been-released.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index 61887d0c..5865c180 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -47,7 +47,7 @@ Please see [the document](https://docs.fluentd.org/output/http#method) and [#445 In this release, we add a new option [reuse_connections](https://docs.fluentd.org/output/http#reuse_connections) for [out_http](https://docs.fluentd.org/output/http) plugin. -This option will improve performance of `out_http`. +This option will improve throughput of `out_http`. Please see [the document](https://docs.fluentd.org/output/http#reuse_connections) and [#4330](https://github.com/fluent/fluentd/pull/4330) for more information. From fb901338e28349c8f9c766365744d9f720ec6b6f Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Thu, 16 May 2024 15:19:03 +0900 Subject: [PATCH 5/8] Improve expression Signed-off-by: Daijiro Fukuda Co-authored-by: Hiroshi Hatake --- content/blog/20240430_fluentd-v1.17.0-has-been-released.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index 5865c180..faf8521a 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -53,7 +53,7 @@ Please see [the document](https://docs.fluentd.org/output/http#reuse_connections ### `in_http`: Recognize CSP reports as JSON data -In this release, we make the data of the request where the `Content-Type` is `application/csp-report` be considered JSON by default. +In this release, we make the data type of the request where the `Content-Type` is `application/csp-report` be considered JSON by default. Now, `in_http` can receive [Content Security Policy](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP)'s report by default. From 3515e9a6b51f9b0f3de5e7e59accc44f2d0d6ff3 Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Wed, 5 Jun 2024 17:32:54 +0900 Subject: [PATCH 6/8] Add description about parser fix Signed-off-by: Daijiro Fukuda --- ...40430_fluentd-v1.17.0-has-been-released.md | 137 +++++++++++++++++- 1 file changed, 136 insertions(+), 1 deletion(-) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index faf8521a..178a3952 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -63,7 +63,142 @@ Please see [#4282](https://github.com/fluent/fluentd/pull/4282) for more informa ### Make sure parser returns hash -TODO +The record data in a Fluentd event must be a hash object. + +* [Life of a Fluentd event](https://docs.fluentd.org/quickstart/life-of-a-fluentd-event) + +However, in the previous versions, some parser plugins could return a non-hash object, such as an array. +It could cause errors in subsequent processing. + +In this release, the following parser plugins have been fixed. + +* [parser_json](https://docs.fluentd.org/parser/json) +* [parser_msgpack](https://docs.fluentd.org/parser/msgpack) + +The changes are as follows. + +* Make sure to return a hash record. +* Make sure to accept only a hash or an array of hash. + +Here are the details for each case. + +#### Example of changed behavior + +Config: + +``` + + @type tcp + tag test.tcp + + @type json + + + + + @type stdout + +``` + +Send an array data: + +``` +$ netcat 0.0.0.0 5170 +[{"k":"v"}, {"k2":"v2"}] +``` + +The result before this version: + +``` +{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}] +``` + +The result after this version: + +``` +{datetime} test.tcp: {"k":"v"} +{datetime} test.tcp: {"k2":"v2"} +``` + +#### Example of resolved error + +Config: + +``` + + @type tcp + tag test.tcp + + @type json + null_empty_string + + + + + @type stdout + +``` + +Send an array data: + +``` +$ netcat 0.0.0.0 5170 +[{"k":"v"}, {"k2":"v2"}] +``` + +The result before this version: + +``` +{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array" +``` + +The result after this version: + +``` +{datetime} test.tcp: {"k":"v"} +{datetime} test.tcp: {"k2":"v2"} +``` + +#### Remaining problem: filter_parser + +In the previous versions, `filter_parser` could return an array record based on this wrong behavior. +From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. +This should be improved in the future. + +Here is an example. + +``` + + @type sample + tag test.array + sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"} + + + + @type parser + key_name message + + @type json + + + + + @type stdout + +``` + +The result before this version: + +``` +{datetime} test.array: [{"k":"v"},{"k2":"v2"}] +``` + +The result after this version: + +``` +{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"} +{datetime} test.array: {"k":"v"} +``` Enjoy logging! From 51643f4d9a4cd035407cdd5b502868e7043eed25 Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Thu, 6 Jun 2024 09:00:31 +0900 Subject: [PATCH 7/8] Add closing statement Signed-off-by: Daijiro Fukuda --- content/blog/20240430_fluentd-v1.17.0-has-been-released.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index 178a3952..792df82f 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -200,6 +200,11 @@ The result after this version: {datetime} test.array: {"k":"v"} ``` +These are the major changes for this release. + +In addition, some performance improvements have been included! +Please see [ChangeLog](https://github.com/fluent/fluentd/blob/master/CHANGELOG.md#release-v1170---20240430) for details! + Enjoy logging! TAG: Fluentd Announcement From 3e5fc8a53dd515f7b4fce8fa927e5fac8ec430b5 Mon Sep 17 00:00:00 2001 From: Daijiro Fukuda Date: Thu, 6 Jun 2024 13:13:15 +0900 Subject: [PATCH 8/8] Improve expressions Signed-off-by: Daijiro Fukuda Co-authored-by: Hiroshi Hatake --- content/blog/20240430_fluentd-v1.17.0-has-been-released.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md index 792df82f..45c9e22d 100644 --- a/content/blog/20240430_fluentd-v1.17.0-has-been-released.md +++ b/content/blog/20240430_fluentd-v1.17.0-has-been-released.md @@ -63,7 +63,7 @@ Please see [#4282](https://github.com/fluent/fluentd/pull/4282) for more informa ### Make sure parser returns hash -The record data in a Fluentd event must be a hash object. +The record data in an event of Fluentd must be a hash object. * [Life of a Fluentd event](https://docs.fluentd.org/quickstart/life-of-a-fluentd-event) @@ -75,7 +75,7 @@ In this release, the following parser plugins have been fixed. * [parser_json](https://docs.fluentd.org/parser/json) * [parser_msgpack](https://docs.fluentd.org/parser/msgpack) -The changes are as follows. +The changes are as follows: * Make sure to return a hash record. * Make sure to accept only a hash or an array of hash. @@ -163,7 +163,7 @@ The result after this version: In the previous versions, `filter_parser` could return an array record based on this wrong behavior. From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case. -This should be improved in the future. +This behavior should improve in the future. Here is an example.