Elastic-Agent does not seem to support all of Beats processor features

[belimawr]

While I was working on #6730, I tried to create the following processor logic for the filebeat-monitoring (the processors there are placeholders):

        - if:
          has_fields:
              - "data_stream.dataset"
          then:
            - add_fields:
                fields:
                  then_run: true
          else:
            - add_fields:
                fields:
                  then_run: false
                  else_run: true

It can be translated to the Go code to generate the monitoring configuration:

				map[string]interface{}{
					"if": map[string]interface{}{
						"has_fields": []interface{}{
							"component.dataset",
						},
					}, // condition
					"then": []map[string]interface{}{
						{
							"copy_fields": map[string]interface{}{
								"fields": []interface{}{
									map[string]interface{}{
										"from": "component.dataset",
										"to":   "data_stream.dataset",
									},
								},
							},
						},
					},
					"else": []map[string]interface{}{
						{
							"copy_fields": map[string]interface{}{
								"fields": []interface{}{
									map[string]interface{}{
										"from": "data_stream.dataset_original",
										"to":   "data_stream.dataset",
									},
								},
								"ignore_missing": true,
							},
						},
					},
				},

This code can be added somewhere in the v1_monitoring.go:

elastic-agent/internal/pkg/agent/application/monitoring/v1_monitor.go

Lines 437 to 461 in 45e3abf

	// copy component.dataset as the real dataset
	map[string]interface{}{
	"copy_fields": map[string]interface{}{
	"fields": []interface{}{
	map[string]interface{}{
	"from": "component.dataset",
	"to": "data_stream.dataset",
	},
	},
	"fail_on_error": false,
	"ignore_missing": true,
	},
	},
	// possible it's a log message from agent itself (doesn't have component.dataset)
	map[string]interface{}{
	"copy_fields": map[string]interface{}{
	"fields": []interface{}{
	map[string]interface{}{
	"from": "data_stream.dataset_original",
	"to": "data_stream.dataset",
	},
	},
	"fail_on_error": false,
	},
	},

This generates the following error:

   ├─ filestream-monitoring
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '4169'
   │  ├─ filestream-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ filestream-monitoring-filestream-monitoring-agent
   │     ├─ status: (FAILED) proto: invalid type: []map[string]interface {}
   │     └─ type: INPUT

{"log.level":"error","@timestamp":"2025-02-11T20:29:57.198Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":671},"message":"Spawned new unit filestream-monitoring-filestream-monitoring-agent: proto: invalid type: []map[string]interface {}","log":{"source":"elastic-agent"},"component":{"id":"filestream-monitoring","state":"STARTING"},"unit":{"id":"filestream-monitoring-filestream-monitoring-agent","type":"input","state":"FAILED"},"ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Repeated maps can't be represented in protobuf which is what the error is telling you. https://protobuf.dev/programming-guides/proto3/#maps-features

You have to write this without the []map[string]interface{}.

[belimawr]

Repeated maps can't be represented in protobuf which is what the error is telling you. https://protobuf.dev/programming-guides/proto3/#maps-features

You have to write this without the []map[string]interface{}.

Thanks Craig! I didn't know that, nor did I expect it 🤦‍♂