Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[system] - Cleanup fields.yml files #8100

Merged
merged 8 commits into from
Oct 5, 2023
Merged

[system] - Cleanup fields.yml files #8100

merged 8 commits into from
Oct 5, 2023

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Oct 5, 2023 •
edited
Loading

Proposed commit message

This cleans up the field definitions for the package.

Remove unnecessary attributes like required, level,format, etc.

Use external: ecs for any field that exists in ECS.

Remove description from field groups. The description does not get used by anything (not in docs, mappings, fleet).

Remove duplicate field declarations.

Replace dotted YAML key kibana.version in manifest.yml.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

@andrewkroh
system - use ecs, remove unused attributes
9c47b05 
[git-generate]
go run github.com/andrewkroh/fydler@eddf66dada2 -fix packages/system/**/fields/*yml
cd packages/system
elastic-package format
@andrewkroh
system - remove ECS fields that shouldn't exist
7ab5e6d 
packages/system/data_stream/core/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/cpu/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/diskio/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/fsstat/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/load/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/memory/fields/ecs.yml:1:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/network/fields/ecs.yml:5:3 group is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/network/fields/ecs.yml:11:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/network/fields/ecs.yml:15:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/network/fields/ecs.yml:21:3 source is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/network/fields/ecs.yml:39:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process/fields/ecs.yml:1:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process/fields/ecs.yml:14:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process/fields/ecs.yml:18:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process_summary/fields/ecs.yml:5:3 group is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process_summary/fields/ecs.yml:11:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process_summary/fields/ecs.yml:15:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process_summary/fields/ecs.yml:21:3 source is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/process_summary/fields/ecs.yml:39:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/socket_summary/fields/ecs.yml:5:3 group is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/socket_summary/fields/ecs.yml:11:3 host is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/socket_summary/fields/ecs.yml:15:3 process is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/socket_summary/fields/ecs.yml:21:3 source is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
packages/system/data_stream/socket_summary/fields/ecs.yml:39:3 user is declared with 'external: ecs' but this field does not exist in ECS version "v8.0.0" (ecsdefinitionfact)
@andrewkroh
system - remove duplicate field definitions
8e0b974
@andrewkroh
system - generate readme
f7877bf 
[git-generate]
cd packages/system
elastic-package format
@andrewkroh
system - remove dotted YAML key
63c3e3b
@andrewkroh
system - add changelog
44e6fcf 
[git-generate]
cd packages/system
elastic-package changelog add --link '#8100' --next minor --type enhancement --description 'Modified the field definitions to reference ECS where possible and remove invalid field attributes.'
@andrewkroh andrewkroh added Team:Security-External Integrations Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Oct 5, 2023
@elasticmachine
Copy link

elasticmachine commented Oct 5, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-05T09:25:30.968+0000

  • Duration: 16 min 0 sec

Test stats 🧪

Test Results
Failed 0
Passed 151
Skipped 0
Total 151

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@andrewkroh
system - workaround ecs with metric_type issue
a57d4ad 
Add 'type' to workaround validation issues.
See elastic/package-spec#634
@elasticmachine
Copy link

elasticmachine commented Oct 5, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (3/3) 💚
Files 100.0% (4/4) 💚
Classes 100.0% (4/4) 💚
Methods 63.415% (52/82) 👎 -36.585
Lines 99.863% (2924/2928) 👍 7.581
Conditionals 100.0% (0/0) 💚

@andrewkroh andrewkroh marked this pull request as ready for review October 5, 2023 03:08
@andrewkroh andrewkroh requested review from a team as code owners October 5, 2023 03:08
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@muthu-mps
Copy link
Contributor

@andrewkroh - As these changes are common across integrations, Do you recommend creating separate issues to make this change for the rest of the integrations by keeping this PR as a base ?

@efd6
Copy link
Contributor

efd6 commented Oct 5, 2023

@muthu-mps What do you mean by "keeping this PR as a base"?

@muthu-mps
Copy link
Contributor

@muthu-mps What do you mean by "keeping this PR as a base"?

I mean applying this changes to the rest of the integrations which is not referencing the ECS fields and having descriptions in agent.yml group fields.

@efd6
Copy link
Contributor

efd6 commented Oct 5, 2023

The change is for the most part entirely mechanical, so it should be easy to do the same to other packages.

@andrewkroh
address review comment
44284b6 
Move @timestamp to base-fields.yml
@muthu-mps muthu-mps self-requested a review October 5, 2023 09:42
muthu-mps
muthu-mps approved these changes Oct 5, 2023
View reviewed changes
Copy link
Contributor

@muthu-mps muthu-mps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@andrewkroh andrewkroh merged commit bb4fbfa into elastic:main Oct 5, 2023
@elasticmachine
Copy link

Package system - 1.41.0 containing this change is available at https://epr.elastic.co/search?package=system

@ishleenk17 ishleenk17 mentioned this pull request Oct 20, 2023
Closed
This was referenced Oct 24, 2023
Merged
Merged
@constanca-m constanca-m mentioned this pull request Dec 12, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 approved these changes

@muthu-mps muthu-mps muthu-mps approved these changes

Assignees
No one assigned
Labels
Integration:system System Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@andrewkroh @elasticmachine @muthu-mps @efd6