[Fleet] Mapping does not reflect the fields definition in package

[tetianakravchenko]

Kibana version:
8.8.0
Elasticsearch version:
8.8.0
Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):
tested in elastic-package and cloud

Description of the problem including expected versus actual behavior:
package field definitions are not respected and mapping does not include expected configuration. Example: in package field defined as a of type scaled_float , in mapping this field appear as long (seems like dynamic mapping applies here). Some other metadata is not respected as well (like unit and metric_type) and are missing

Steps to reproduce:

1. Install system package - specifically diskio data_stream
2. Check the mapping of the host.disk.read.bytes field:

mapping:
"host": {
       "properties": {
         "architecture": {
           "type": "keyword",
           "ignore_above": 1024
         },
         "containerized": {
           "type": "boolean"
         },
         "disk": {
           "properties": {
             "read": {
               "properties": {
                 "bytes": {
                   "type": "long"
                 }
               }
             },
             "write": {
               "properties": {
                 "bytes": {
                   "type": "long"
                 }
               }
             }
           }
         },

3. compare to the package definition - https://github.com/elastic/integrations/blob/main/packages/system/data_stream/diskio/fields/fields.yml#L123-L137

- name: host
  type: group
  fields:
    - name: disk.read.bytes
      type: scaled_float
      unit: byte
      metric_type: gauge
      description: |
        The total number of bytes read successfully in a given period of time.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Describe the feature:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[tetianakravchenko]

Hey @juliaElastic could you please provide any information for this? any plans for this issue?

[mlunadia]

@jlind23 would you be able to help @tetianakravchenko with visibility on this one? Thanks :)

[jlind23]

Thanks for the ping, i'll add this to one of our upcoming sprint to sprint what the problem might be.

[juliaElastic]

@tetianakravchenko sorry for the late reply, missed the ping.

Investigated this and it seems that the problem is that the disk.read.bytes field is defined 2 times in the diskio data stream, first as scaled_float, second as long here.
The fleet logic has a dedupFields function which removes duplicate fields from the same datastream, that is why only the long type remains in the mapping. I tested by commenting out the long field, and then seeing the mapping with type:scaled_float as expected.
Do you know why this field is defined 2 times in system package? Can we change that? It seem like not a bug in Fleet, as the dedupe logic was added intentionally.

[tetianakravchenko]

Hey @juliaElastic thank you for the investigation! I've already did some work on removing duplicated fields in system package, so I thought it should be a different issue, but seems I missed those 2 fields. Sorry for that.

Do you know why this field is defined 2 times in system package? Can we change that?

yes, it should be fixed, I will check which fields should be removed.
I will close this issue.