loki_out: add stuctured_metadata_map_keys #9530
Conversation
7f1db76 to
a9cdcad
Compare
a9cdcad to
e9fee7d
Compare
| @@ -0,0 +1,41 @@ | |||
| services: | |||
| fluentbit: | |||
| image: fluent/fluent-bit:latest | |||
There was a problem hiding this comment.
I'd probably add a build section as well and comment out one or the other just to show we can either compile or use the latest.
There was a problem hiding this comment.
like this?:
build:
context: ../../
dockerfile: dockerfiles/Dockerfile
pull_policy: buildThere was a problem hiding this comment.
no need for pull_policy and I'd just leave it commented out once you've tested it - main thing is it allows people to see how to build vs pull
There was a problem hiding this comment.
This seems to be working locally now
| networks: | ||
| - loki-network | ||
| volumes: | ||
| - ./config/fluent-bit_loki_out-structured_metadata_map.yaml:/etc/fluent-bit_loki_out-structured_metadata_map.yaml |
There was a problem hiding this comment.
If you mount this to the default location then you can also remove the command override - although that only is true currently if it is the legacy TOML format config.
e.g. this would mean no need to have a command:
- ./config/fluent-bit_loki_out-structured_metadata_map.conf:/fluent-bit/etc/fluent-bit.confThere was a problem hiding this comment.
Is this preferred? I find the yaml easier to read.
I think I read somewhere that you're going to make the yaml config the default? Would this end up diverging between the 3.x branch and 4.x?
Happy to change this though if that's what you want 😄
There was a problem hiding this comment.
No, this is fine and yeah we will be moving to YAML by default for 4.0.
| pack_label_key(mp_pck, (char*) accessed_map_kv.key.via.str.ptr, | ||
| accessed_map_kv.key.via.str.size); | ||
| /* | ||
| * Does this need optimising? For example, to handle |
There was a problem hiding this comment.
Simple is better I think unless it causes an impact. I'd be tempted to keep a naive initial implementation that is simpler and optimise later with specific test cases then.
| {"structured_metadata_map_and_explicit", | ||
| flb_test_structured_metadata_map_and_explicit}, | ||
| {"structured_metadata_map_single_missing_map", | ||
| flb_test_structured_metadata_map_single_missing_map}, |
There was a problem hiding this comment.
Probably but we will know when we find them!
|
FYI I'm seeing the K8S event test fail in other PRs so I think it is a flake: https://github.com/fluent/fluent-bit/actions/runs/12834843167/job/35870345004?pr=9530 |
|
@0x006EA1E5 could you clean up the commits and/or squash them? |
560e444 to
8387b9a
Compare
| # | ||
| # If you would like to disable reporting, uncomment the following lines: | ||
| #analytics: | ||
| # reporting_enabled: false |
There was a problem hiding this comment.
Not unless required I'd say
8387b9a to
99caf96
Compare
* Adds stuctured_metadata_map_keys config to dynamically populate stuctured_metadata from a map * Add docker-compose to test loki backend Signed-off-by: Greg Eales <0x006EA1E5@gmail.com>
99caf96 to
ca46caf
Compare
| if (kv->ra_key != NULL && kv->ra_val == NULL) { | ||
|
|
||
| /* try to get the value for the record accessor */ | ||
| if (flb_ra_get_kv_pair(kv->ra_key, *map, &start_key, &out_key, &out_val) |
There was a problem hiding this comment.
flb_ra_* API do not use msgpack-c return codes like MSGPACK_UNPACK_CONTINUE, need to check the return values and adjust it
Resolves #9463
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#1527
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.