[8.15](backport #5345) Improve watcher and TestWatcher_AgentErrorQuick logs

[mergify]

What does this PR do?

It improves the watcher logs and prints the logs if TestWatcher_AgentErrorQuick fails.
If the test would fail, the logs would be like the following:

❯ go test -run TestWatcher_AgentErrorQuick ./internal/pkg/agent/application/upgrade/
--- FAIL: TestWatcher_AgentErrorQuick (1.00s)
    watcher_test.go:287: [info] Agent watcher started
    watcher_test.go:287: [info] Trying to connect to agent
    watcher_test.go:287: [info] Connected to agent
    watcher_test.go:287: [debug] received state: FAILED:force failure
    watcher_test.go:287: [info] Communicating with PID 0
    watcher_test.go:287: [debug] received state: HEALTHY:healthy
    watcher_test.go:287: [error] Agent reported failure (starting failed timer): agent reported failed state: force failure
    watcher_test.go:287: [info] Agent reported healthy (failed timer stopped)
    watcher_test.go:287: [debug] received state: error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
    watcher_test.go:287: [error] Lost connection: failed reading next state: rpc error: code = DeadlineExceeded desc = context deadline exceeded
FAIL
FAIL	github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade	1.351s
FAIL

Why is it important?

TestWatcher_AgentErrorQuick was flaky before, but it hasn't happened again on CI. Even running it for 12 hours didn't reproduce the problem.

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

None

How to test this PR locally

make TestWatcher_AgentErrorQuick to fail by adding t.Fail() at the end of the test and then run the test.

❯ go test -count 43200 -run TestWatcher_AgentErrorQuick -timeout=0
PASS
ok      github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade 43320.474s

Related issues

• Closes Flaky Test TestWatcher_AgentErrorQuick #3983

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #5345 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @AndersonQ? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @AndersonQ? 🙏

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
86.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube