Field formats specified in package not being applied in data view

[ebeahan]

Kibana version:

8.1.1

Describe the bug:

Data views created by Fleet at package install do not apply the format option, if format is included on a field definition in the package config.

Example: the system package defines the field system.memory.total:

https://github.com/elastic/package-storage/blob/production/packages/system/1.6.4/data_stream/memory/fields/fields.yml#L6

The field includes format: bytes, but the data view created in Kibana is not setting the Bytes format.

Expected behavior:

The format specified in the package's field definitions would be applied in the Kibana data view:

Any additional context:

Many ECS and non-ECS fields index as an Elasticsearch numeric type. However, when viewed in Kibana are better displayed as a String or a Byte value.

Examples:

• Process IDs (pids) and TCP/UDP port numbers. These values should display as 12345 not 12,345
• Some metric values, like memory totals and usage, can use the Bytes format: 5GB vs. 5,123,456,789

The Beats docs also discuss the format field: https://github.com/elastic/beats/blob/main/docs/devguide/newdashboards.asciidoc#generate-index-pattern

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

From elastic/kibana#119527, Fleet no longer declares a list of fields on Kibana data views, therefore we do not prescriptively apply a format to package fields:

However, as of 7.12, index patterns generate their fields dynamically at runtime, ignoring the fields key on the saved object, meaning these dashboard errors are visible. This has allowed me to remove all field generation code and move the index pattern creation to before the package is installed.

I believe the Beats documentation is old and does not take into account the above changes to how Kibana handles index pattern/data view fields now. Actually, the package spec does not even support format property for field definitions: https://github.com/elastic/package-spec/blob/02e9a7fd88fec29302517e89d49879da2f6a3050/versions/1/data_stream/fields/fields.spec.yml

In this case system.memory.total field type could just be changed to bytes and Kibana will apply the default bytes formatting.

For fields that declared as integer or long type, Kibana automatically applies numeric formatting.

I will move this to integrations repo for further discussion about cleaning up system field definitions (and maybe other packages too?).

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ebeahan]

Thanks for the background, @jen-huang.

Some fields use a numeric type but look unconventional with the Kibana default numeric formatting, like PIDs or port numbers. Users are likely to note the change after migrating from Beats to Agent. Migrating existing fields using the 64-bit long to an 8-bit byte type isn't likely an easy option.

The past solution was to set the format, which would override the index pattern/data view field format. If that's no longer feasible because of the referenced Fleet change, integrations should still have some sort of mechanism to override a field's format.

Related, I did open a separate issue proposing to add format to the spec: elastic/package-spec#302.

[andrewkroh]

I will move this to integrations repo for further discussion about cleaning up system field definitions (and maybe other packages too?).

Doing this cleanup would be great. Currently there is no strict validation of the keys used "fields.yml" files. I think this stems from an incomplete definition of the keys in the spec. I noted this in elastic/package-spec#271 (comment).

@jen-huang Perhaps we could get some help from your team telling us what fields are actually honored by Fleet and I can update the spec to reflect this. Then we can turn on strict validation and remove invalid keys from packages.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!