[Feature] Improve compressed request handling

[peternied]

Is your feature request related to a problem? Please describe.

In some cases, a request is considered invalid from its headers and its unnecessary to inspect the body of the request. For compressed requests, decompressed bodies can sometimes be quite large so its a benefit to skip decompression if the body is only being discarded.

Describe the solution you'd like
There should be some mechanism that is accessible by plugins to provide guidance to continue processing a request before more resources are used.

Describe alternatives you've considered

[peternied]

We been exploring two approaches to address how compressed requests are handled within our system. Here's a overview of both:

Header Verifier with early request exit [1]

Craig is looking into adjusting the timing and manner in which certain components, specifically the "rest controller components", are triggered within the core of OpenSearch. In essence, this approach brings existing request functionalities into broader usage.

Pros:

• Minimizes the changes needed in plugins that are connected downstream.

Cons:

• It blurs the distinctiveness of the "request" feature's design, potentially introducing unexpected behaviors.

PRs

• [1] Allow customization of netty channel handles before and during decompression #10261
• [1] Add early rejection from RestHandler for unauthorized requests security#3418

Minimal Header Verifier [2]

Peter is focusing on the security side of things. Specifically, he's rethinking how the security plugin responds to verification (or "authentication") requests.

Pros:

• The core system remains mostly untouched, preserving its stability.
• Tidies up sections of the security plugin that were previously a bit convoluted.

Cons:

• Requires a significant overhaul of the security plugin, which could be time-consuming and might introduce temporary instability.

PRs

• [2] Add header verification netty channel handler #10290
• [2] Expanding Authentication with SecurityRequest Abstraction security#3430

As we delve deeper, we'll keep everyone updated on our progress and the potential implications of each approach. Feedback and thoughts are always welcome!

[reta]

In some cases, a request is considered invalid from its headers and its unnecessary to inspect the body of the request.

@peternied could you please give an example of invalid request? I am thinking about wrong auth credentials fe, is that the case?

I think if we need to inspect headers and reject the request even before consuming the content, there are two options:

• HttpRequestDecoder that decodes the request, including headers
• HttpObjectAggregator that aggregates the request from chunk with begin / aggregate / end lifecycle

Something to be aware of, with #9672 the request body is consumed lazily (as a stream), so the headers could be consulted before processing the request body.

[peternied]

@reta Here is an example of what a request could look likehttps://github.com/opensearch-project/security/pull/3417/files#r1343153212

POST /*/_search
{
"items": [
    {"a": 123, "b":123, ...},
    {"a": 123, "b":123, ...},
    ... repeated many times ...
]

I think having validation processes that only use headers vs a latter part of the pipeline get the request body (whole vs stream) seems like a good way to separate these concerns of validation vs processing while lowers the resource usage on invalid / rejected requests

[peternied]

Updating this issue; @cwperks and I have been trying to figure out how to best approach this problem, earlier this was kind of called top-down vs bottoms-up, we we have landed on a design pattern that is the meeting place between both of these points.

By allowing a header verification step to the pipeline along with allowing the decompressor to be wrapped, plugins that are engaged in the network configuration can fine tune their behavior. The business logic of to process criteria for inclusion/exclusion solely are declared within the security plugin.

If there is any behavior that needs to pass information between these system states it is encapsulated wholly in the plugin codebase.

The following pull requests are part of the total delivery to address this feature:

• Allow customization of netty channel handles before and during decompression #10261
• Expanding Authentication with SecurityRequest Abstraction security#3430
• Add early rejection from RestHandler for unauthorized requests security#3418
• [Draft] Resource usage case test security#3417