What do the 'Non-zero metrics' logs mean when displayed under elastic-agent.filebeat in (Agent Beats) logs.

[dikshachauhan-qasource]

Kibana version:
Kibana:8.0.0 snapshot Cloud environment

Elasticsearch version:
Elasticsearch: 8.0.0 snapshot Cloud environment

Host OS and Browser version:
Windows 10, All

Original install method (e.g. download page, yum, from source, etc.):
8.0.0 snapshot Cloud environment

Description
[Ingest Manager]: Non-zero metrics logs are getting displayed under elastic-agent.filebeat dataset on Agents log page.

Pre-condition:

1. Agent should be deplayed to 8.0.0 Kibana cloud environment. Build details are as follows:

BUILD 38781
COMMIT 0f408041b48a03fa46492f62d08fe5d8025390ea
Artifact link: https://snapshots.elastic.co/8.0.0-59591b4b/downloads/beats/elastic-agent/elastic-agent-8.0.0-SNAPSHOT-windows-x86_64.zip

Steps to reproduce:

1. Login to Kibana cloud environment.
2. Go to Agents tab. Observe, logs are generated successfully.
3. Click on Agent logging level and select option 'error'.
4. Apply changes. And observe confirmation messgae is displayed.
5. Refresh the browser. Observe, changes are updated on agent details page at logging level field.
6. Navigate back to Logs UI. Select 'elastic-agent.filebeat' filter at 'Dataset'
7. Observe, no-specific level logs are displayed.

We have observed same behaviour on 7.11 snapshot Kibana cloud build as well. Build details are as folows:
BUILD 37292
COMMIT 9b0ec30e60ba98f663fbe46b54d27b273f18fd5e
Artifact link: https://snapshots.elastic.co/7.11.0-83fccc3d/downloads/beats/elastic-agent/elastic-agent-7.11.0-SNAPSHOT-windows-x86_64.zip

Actual result:
Non-zero metrics logs are getting displayed under elastic-agent.filebeat dataset on Agents log page.

Expected Result:
Non-zero metrics logs should not be displayed under elastic-agent.filebeat dataset on Agents log page.

Screenshot:

[dikshachauhan-qasource]

Please review @manishgupta-qasource

[manishgupta-qasource]

Reviewed & assigned to @EricDavisX

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

hi @dikshachauhan-qasource @rahulgupta-qasource to confirm, I'll ask:

is the concern here that when user selects 'Error' level that the more verbose 'Info' level logs are still displayed?

@nchaulet @ph @michalpristas thoughts?

[nchaulet]

I will look a little more to that but looks like by default the log level of the agent is debug (I think it should be info so there is probably a bug) I misread the screenshot the log here is info

Also I think it's normal that these logs are visible changing the log level affect logs after you set the log level but will not remove existing logs here the log level was changed at 16:50 and the latest debug log is at 16:49

[EricDavisX]

I couldn't tell well, but I agree that 16:50 and 16:49 are close and that the logging is ingested into Elastic... and so once it is there it is expected to remain. So, I'm unsure of what the bug is. We can get info from team when they reply back - thanks for the poke.

[dikshachauhan-qasource]

Hi @EricDavisX

The issue reported is related to non metrics logs, for which we are unable to understand what these logs are for.

As these logs are not explanatory, so we reported issue to track out if these are expected one or not.

Please confirm if these logs are expected.

Thanks
QA

[botelastic]

This issue doesn't have a Team:<team> label.

[EricDavisX]

Thank you @dikshachauhan-qasource for clarifying. I will pass this on to the Beats team then... I'm not sure who owns it. @andresrc @ph can you help? To confirm, I assess it as a low priority question now that we've triaged it. But any info we can pass would help the team log better bugs and test better.

[ph]

@EricDavisX Ok, there seems to be some noise here, The non-zero log statement is an expected log warning, we could remove it from beats but I think I would just close the issue and once we refactor beats into a single beats we can drop unnecessary log.