[Fleet][APM] APM Server via Elastic Agent Fleet is missing permissions to create data stream

[simitt]

Description of the problem including expected versus actual behavior:

When starting APM Server via Elastic Agent integration with Fleet documents ending up in log or metric data streams can be written successfully, but the APM Server raises an error when trying to index span and transaction documents. For these docs a trace dataset should be created, but the API Key received from Fleets is missing privileges to create the data stream.

Logged Error:

 {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [xxxx] of user [fleet_enroll] on indices [traces-apm-default], this action is granted by the privileges [auto_configure,create_index,manage,all]\"}

Steps to reproduce:

1. Check out @graphaelli 's branch adding elastic-agent support to the integration testing
2. Run ./scripts/compose.py start master --with-elastic-agent --no-apm-server
3. Navigate to the Kibana UI and manually install APM Server integration
4. log into the docker container docker exec -it <Container-ID> /bin/bash and send an example request to the APM Server that contains all kinds of events. (For now the APM Server port is not exposed, therefore sending the request from inside the container). For the span and transaction docs the mentioned error is logged. The logs can be found under data/elastic-agent-<ID>/logs/default/apm-server-json.log*.
   This is all a bit manual for now, until we have finished integration APM Server under Elastic Agent into the integration testing repo.

cc @ruflin

[ruflin]

The problem is that these permissions were not given yet to the user in Kibana: https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/setup.ts#L142 @nchaulet Can you take a look?

[simitt]

Sure thing, thanks @ruflin .

I also identified that the _index_template is missing the "allow_auto_create" : true, compared to the shipped minimal templates for logs and metrics.

[nchaulet]

Yes adding the permission in the fleet setup and here https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/api_keys/index.ts/#L22-L32 should fix the missing permission issue

We do not have a way right now to update the existing API keys so the migration for existing Fleet user from 7.10 to 7.11 will not work for APM out of the box.

[ruflin]

@nchaulet Can you update an issue around the "update API keys permissions" problem? Would be nice if we would have at least a way to "recreate" API Keys for a set of Elastic Agents or similar to upgrade them.

[simitt]

Thanks @ruflin and @nchaulet for the pointers - I added the changes and tested it through - APM Server ingestion works perfectly now with #85802

[nchaulet]

For 7.11 we are probably not going to support the api key upgrade so we should probably document what users will have to do:

• to recreate the fleet user POST /fleet/agents/setup {forceRecreate: true} should we expose this in the UI?
• re-enroll existing agents.

For 7.12 we should probably have a better plan #85777

[jalvz]

We do not have a way right now to update the existing API keys so the migration for existing Fleet user from 7.10 to 7.11 will not work for APM out of the box.

There is no APM integration in 7.10, so 7.10->7.11 migration for APM should not be a problem, right?
Or did I misunderstand?

[nchaulet]

@jalvz We already have users using Fleet in 7.10 so that mean the user (fleet_enroll) that create agent API keys already exists with missing permission, and eventually a user can add the APM integration to an existing agent (where the existing API key will miss permission)

[jalvz]

got it, thanks!

[simitt]

to recreate the fleet user POST /fleet/agents/setup {forceRecreate: true} should we expose this in the UI?

I would not expose this in the UI but only cover by documentation for the experimental APM integration, and it sounds like in the longer term #85777 should handle this in a better way anyways.