[Docs] Fix positioning of ECS fields in the documents

[nehaduggal]

Currently the ECS fields documentation shows up along side other field documentation that can give the impression that APM fields are not ECS compliant.

Lets change the structure of this page to emphasize APM fields that are/are not ECS compliant.

[cyrille-leclerc]

FYI the SIEM team has documented the ECS fields it is using: https://www.elastic.co/guide/en/siem/guide/current/siem-field-reference.html

[webmat]

One thing that I find confusing in how the APM field docs are laid out is that the ECS fields section lists all ECS fields, whether they're relevant to APM or not. If some ECS fields are not relevant to (nor populated by) APM, they don't need to be mentioned in the docs, IMO.

Perhaps a better way to approach this would be to simply list all APM fields as you currently do, and have a checkmark or some sort of mention per field, that indicates whether it's an ECS field or not.

The page referenced by @cyrille-leclerc above has a slightly different purpose. SIEM was initially one of the main drivers of ECS, both as a producer and a consumer of the data. However some users aren't using Beats, but still want to have their events populate the SIEM. So the page linked above serves the need of explaining what's the minimum users need to do, in order to have their data appear in SIEM.

If APM has a desire to instruct users on how can inject custom events in APM, then a page like the SIEM page may be interesting for APM. Otherwise it may not be relevant.

[bmorelli25]

Thanks, everyone. I agree the ECS field documentation is less than ideal.

It's worth noting too that Andrew opened a bunch of ECS alignment issues for APM Server yesterday: https://github.com/elastic/apm-server/issues?q=is%3Aopen+is%3Aissue+label%3Aecs.

Perhaps a better way to approach this would be to simply list all APM fields as you currently do, and have a checkmark or some sort of mention per field, that indicates whether it's an ECS field or not.

I like this idea a lot. I'll look at the script we currently use to build these docs and see how difficult that would be to implement.

The page referenced by @cyrille-leclerc above has a slightly different purpose. SIEM was initially one of the main drivers of ECS, both as a producer and a consumer of the data. However some users aren't using Beats, but still want to have their events populate the SIEM. So the page linked above serves the need of explaining what's the minimum users need to do, in order to have their data appear in SIEM.

If APM has a desire to instruct users on how can inject custom events in APM, then a page like the SIEM page may be interesting for APM. Otherwise it may not be relevant.

I think this is useful documentation as well. We have a longstanding issue open to address this: https://github.com/elastic/observability-dev/issues/223.

[webmat]

Thanks for pointing out these issues @bmorelli25 !

[bmorelli25]

Here's what I've learned about this documentation over the past couple of days. Bear with me, some of this is basic, but I'm learning in-depth about this for the first time. Thanks to DeDe / Gil / Andrew for sharing this knowledge.

• make update runs generate_fields_docs.py which (among other things) updates the field documentation. The script reads field descriptions from YAML files in _meta directories and creates Asciidoc files in the docs directory under each Beat.
• If a field is defined twice, you get an error. Gil added the ability to bypass this with an overwrite field in allow/merge fields.yml overrides beats#9188. This allows APM to "own" certain fields (i.e., add our own description) as long as the type remains the same.
• You can edit the Beat's build script by checking out beats and modifying APM's go.mod file to use your local clone: go mod edit -replace=github.com/elastic/beats/v7=<path/to/beats>. Then run make update to test your changes.

Possible solution:

• Alter generate_fields_docs.py to mark APM fields that include overwrite: true as ECS fields. I've only spot-checked this--but in my quick check, all overwrite: true fields are also ECS fields. The downside of this solution is that it is only for APM, and not for Beats.

We should also clarify this:

• Regardless of whether APM uses a field or not, all ECS fields are listed in the APM Server Reference: https://www.elastic.co/guide/en/apm/server/master/exported-fields-ecs.html. This isn't necessarily a bad thing, as there's nothing stopping our users from adding an ingest node processor and putting data in the fields. However, we should clarify this on this page.

[bmorelli25]

I've opened elastic/beats#21359 to address this.

[bmorelli25]

Related issue opened in Beats: elastic/beats#20729.