[testing/integration]: fix TestLogIngestionFleetManaged

[VihasMakwana]

Looking at diagnostics of #5631 failure, it was clear that the monitoring units were not healthy. This is a known issue and it takes some time for them to be healthy.

This PR aims to fix that by making sure the components are healthy before proceeding with the test case.

• Closes [Flaky Test]: TestLogIngestionFleetManaged – Condition never satisfied #5631

[mergify]

This pull request does not have a backport label. Could you fix it @VihasMakwana? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[VihasMakwana]

/test

[VihasMakwana]

Will run this 4-5 times and observer the CI.

[VihasMakwana]

/test

[VihasMakwana]

succeeded 2/2 times. Triggered 3rd time

[VihasMakwana]

TestLogIngestionFleetManaged succeeded 3/3

[VihasMakwana]

/test

[cmacknz]

I think we should also decrease the metrics interval used in the test to 1s, this will effectively be speeding up and allowing for retries if we hit this failure on the first attempt to collect metrics.

#5631 (comment)

[VihasMakwana]

I think we should also decrease the metrics interval used in the test to 1s, this will effectively be speeding up and allowing for retries if we hit this failure on the first attempt to collect metrics.

#5631 (comment)

This particular test creates policy in fleet and the policy schema doesn't really have any metrics_period field.

EDIT: I think we can do that. I'll try that and update the PR.

EDIT2: We can override elastic-agent.yml, but doing so would contradict the purpose of this fleet-managed test.

[VihasMakwana]

I think we should also decrease the metrics interval used in the test to 1s, this will effectively be speeding up and allowing for retries if we hit this failure on the first attempt to collect metrics.

#5631 (comment)

This was the first thing I wanted to do, as we've done the same with Monitoring* integration test cases by overriding the elastic-agent.yml

[blakerouse]

Looks good.

[cmacknz]

EDIT2: We can override elastic-agent.yml, but doing so would contradict the purpose of this fleet-managed test.

This is fine, if you want to properly do it through Fleet you can use the policy overrides API. See https://support.elastic.dev/knowledge/view/06b69893 for an example.

[cmacknz]

(I don't think it matters for the purpose of what we are trying to test here whether you change the initial local agent configuration or use the overrides API to do this).

Checking for health at startup makes sure we identify the right error, but it roughly equivalent to just waiting longer for us to get healthy. If we increase the metrics interval the whole test should execute faster.

[VihasMakwana]

I'll create a minor release once the CI is green on this commit.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-log-ingestion-tests upstream/fix-log-ingestion-tests
git merge upstream/main
git push upstream fix-log-ingestion-tests

[VihasMakwana]

CI is red.
BLOCKED on #5542

[AndersonQ]

CI is red. BLOCKED on #5542

I'm fixing it with #5741
the issue is with the test, not with github.com/elastic/elastic-agent-libs v0.12.0

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube