Metricbeat Doesn't Get ILM Policy via Auto ILM

[dsanders11]

• Version: 7.0.1
• Operating System: CentOS 7.4

Logs

idxmgmt/std.go:165	Set output.elasticsearch.index to 'metricbeat-7.0.1' as ILM is enabled.
idxmgmt/std.go:361	Set setup.template.name to '{metricbeat-7.0.1 {now/d}-000001}' as ILM is enabled.
idxmgmt/std.go:366	Set setup.template.pattern to 'metricbeat-7.0.1-*' as ILM is enabled.

Can anyone spot the problem? The template pattern won't match the index name, so the index gets created without the template applying, so ILM doesn't actually apply.

[ph]

@dsanders11 Can you share your metricbeat configuration?

[dsanders11]

@ph, it was the standard configuration on CentOS 7.4. All I changed was the cloud.id and cloud.auth settings.

[webjunkie]

I figured out I have the same problem. Default config, no special index names set. ILM seems to be enabled automatically and the template seems not to be used because of the pattern mismatch.

[CodeAdminDe]

The problem seems to exist.
At a fresh cluster (kibana 7.8.0, elasticsearch 7.8.0 with metricbeat 7.8.0) after inital import of index_template, ilm-policy, index-pattern and dashboards via metricbeat setup command.

At first run of metricbeat to collect data, an index will be created (index: metricbeat-7.8.0) but the template (index patterns: metricbeat-7.8.0-*) won't be applied.
It was noticed because no rollover index was created and because of this, the ILM Policy was not applied to the index.
Manual application of ILM Policy to the metricbeat-7.8.0 index failed, of course.

Adjusting the template index pattern after creation (before first data collection) seems to be a workaround. But is that a good idea, or does it trigger not yet known problems?

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.

[abraxxa]

@elastic: please check if this problem still exists, for me 7.13 seems to have finally fixed this issue but I also have changed metricbeat_writer rights which might also have been a problem.

[jsoriano]

idxmgmt/std.go:165	Set output.elasticsearch.index to 'metricbeat-7.0.1' as ILM is enabled.
idxmgmt/std.go:361	Set setup.template.name to '{metricbeat-7.0.1 {now/d}-000001}' as ILM is enabled.
idxmgmt/std.go:366	Set setup.template.pattern to 'metricbeat-7.0.1-*' as ILM is enabled.

Can anyone spot the problem? The template pattern won't match the index name, so the index gets created without the template applying, so ILM doesn't actually apply.

At first run of metricbeat to collect data, an index will be created (index: metricbeat-7.8.0) but the template (index patterns: metricbeat-7.8.0-*) won't be applied.

This is expected, but it is important to notice that metricbeat-7.0.1/metricbeat-7.8.0 are not indexes, but aliases that point to the actual indexes whose names match with the template pattern. These indexes are managed by ILM, what also updates the alias when a new write index is created.
On a new installation, or after an upgrade, you will see that you have an alias metricbeat-<version>, and indexes named metricbeat-<version>-<date>-<sequence>, that match the index pattern.

A common problem with this is that if the alias is manually removed while beats are running, beats will continue trying to write there, and Elasticsearch will create a new index without any mapping. Beats have some checks to detect this kind of situations on startup, and try to solve it or explicitly fail, but they cannot detect them when they are already running, or if ilm is disabled. This alias shouldn't be removed when Beats are running.

please check if this problem still exists, for me 7.13 seems to have finally fixed this issue but I also have changed metricbeat_writer rights which might also have been a problem.

7.13.2 includes a fix (#26146) that solved an issue creating the write index the alias points to, so this may actually solve some of the issues caused by manually deleting the alias or the write index.