[Fleet]: Status not updated for Remote Elasticsearch cluster when set to default and agent is connected.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.13.0 BC3
BUILD: 71857
COMMIT: 82f46148c91eec93ac7382147936028db2eb8883

Host OS: All

Preconditions:

1. 8.13.0-BC3 Kibana cloud environment should be available.
2. An agent should be installed.

Steps to reproduce:

1. Create Remote ES and set it to default.
2. Now install an agent with agent policy outputs set to default.
3. Observe inconsistently-> agent never sends data to the Remote Elasticsearch.
4. Observe Status not updated for default remote cluster.
5. Navigate to Settings and set default Elasticsearch as default output.
6. Under Agent policy settings> set output to Remote Elasticsearch.
7. Observe agent is now sending the data to the remote elasticsearch and status is also updated.

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2024-03-04.17-33-15.mp4

Expected Result:
Status should get updated for Remote Elasticsearch cluster when set to default and agent should send data to the remote cluster when set to Default.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[juliaElastic]

I looked into this, and tried to reproduce locally:

• enrolled an agent to an agent policy with default remote es output
• I'm seeing the data being sent to remote es correctly (3. not reproduced)
• I can reproduce that the output health is not showing up on the UI. I found that this is because the remote es output health is reported under the name "default" instead of the output name. This comes from the fact that the agent policy refers to the remote es output as "default".

One way to solve this would be to improve the UI, to find the output health based on the name "default", if the default output is remote es. Alternatively fleet-server could look up the output id to report output health with the real id instead of "default".

I'm going to decrease the impact to medium, as this is an edge case of using default remote output, and there is an easy workaround (to set an elasticsearch output as default).

[amolnater-qasource]

Hi @juliaElastic

We have revalidated this issue on latest 8.13.0 BC7 kibana cloud environment and had below observations:

Observations:

• Status doesn't get updated and no data for agent for Remote Elasticsearch cluster when set to default.

Steps:

1. Create two fresh environments.
2. Install an agent with new Agent policy with default output set as elasticsearch(the default one available.).
3. Create new Remote Elasticsearch output and set it to Default.
4. Observe under Agent policy too, the default output is now Remote Elasticsearch.
5. Observe no data under Discover tab of Remote Elasticsearch.

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2024-03-26.13-12-33.mp4

Logs:

elastic-agent-diagnostics-2024-03-26T07-53-36Z-00.zip

Build details:
VERSION: 8.13.0 BC7
BUILD: 72069
COMMIT: 2e3a5cd

Hence, we are reopening this issue.

Further we have shared the build details with you over slack.

Thanks!!

[juliaElastic]

Sorry, the fix was not backported to 8.13, so that's why it is reproducible. I'll backport it now, but it won't make it to 8.13.0 as the last BC is built. We can include in 8.13.1.

[juliaElastic]

Looked at the clusters as well, and noticed that the issue is only partially that the backport was not done. The other issue is that the data is not being sent to the remote elasticsearch.
I found in agent logs that the agent can't access the remote ES because the API key is invalid. Did you create the service token in the remote ES? I'm not seeing any fleet-server-remote API keys in the second cluster.

{"log.level":"error","@timestamp":"2024-03-26T07:53:36.741Z","message":"Failed to connect to backoff(elasticsearch(https://f2f004010bde45539e6a68e6bacaef4f.us-west2.gcp.elastic-cloud.com:443)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id P4ioeY4BR0MsIeJOyZ8w\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id P4ioeY4BR0MsIeJOyZ8w\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

The remote service token should have been created manually with the API request in the remote output instructions:

[amolnater-qasource]

Thank you for looking into this @juliaElastic

I found in agent logs that the agent can't access the remote ES because the API key is invalid. Did you create the service token in the remote ES? I'm not seeing any fleet-server-remote API keys in the second cluster.

We have created the Token in remote cluster only

However, the issue is only reproducible for the first time(inconsistent) and whenever we retry after that, it always sends the data.

Screenshot:

Further, thanks for backporting, we will retest the status issue once 8.13.1 is available.

Thanks!

[amolnater-qasource]

Hi Team,

We have revalidated this issue on latest 8.14.0 SNAPSHOT and found this issue fixed now.

Observations:

• Status is updated for Remote Elasticsearch cluster when set to default and agent is connected.

Screenshot:

Build details:
VERSION: 8.14.0 SNAPSHOT
BUILD: 73332
COMMIT: 3f75f6a

Hence, we are closing and marking this issue as QA:Validated.
Thanks!