[PROPOSAL] Highlight REST API Changes in PRs

[dblock]

Is your feature request related to a problem? Please describe

We do not have enough eyes or gates on REST API changes.

For example, in #13306 we introduced a REST API change that could have used more discussion. It's quite likely that we would have concluded that it was not the best way for users to configure ingest processor batch size, and would have opted for a server-side setting per processor, rather than a REST API option that puts the burden on every user to get right. This was discussed in #12457.

Related, we also have been working on https://github.com/opensearch-project/opensearch-api-specification that is used to auto-generate API clients, and these API changes are not always reflected in the spec as they are being introduced before the server ships.

Describe the solution you'd like

1. A gradle plugin that can be used in a CI workflow for OpenSearch core and all plugins to highlight API changes and comment on pull requests with the details about these (to make them more visible).
2. Prevent the PR from being merged without a corresponding PR in https://github.com/opensearch-project/opensearch-api-specification that describes and documents the API change(s).
3. Additional requirements (e.g. bake time) for API changes to ensure more maintainers weigh on every API change.
4. Review of API conventions, rules, and documentation around how to author new REST APIs and the additional scrutiny required.

Related component

Other

Describe alternatives you've considered

No response

Additional context

We're only talking about REST APIs called by users, not Java/core APIs used by plugins.
This issue was opened collaboratively with @andrross.

[vamshin]

Thanks @dblock. Liked the idea of gatekeeping at the PR level to ensure API changes are added to Spec.

[reta]

All in for it, thanks @dblock !

[peternied]

[Triage - attendees 1 2 3]
@dblock Thanks for creating this discussion, look forward to seeing how this resolves