[META] Explore usage of SIMD in OpenSearch

[sohami]

Describe the solution you'd like
With the Project Panama there is support for Vector API to utilize the available SSE and AVX instructions sets on hardware for vectorized processing. The API is currently in 6th incubation but we are seeing some on-demand usage of such APIs in Lucene already. I would think similar to lucene there will be potential to utilize these in OpenSearch codebase itself (may be in Aggregations or other places (TBD)). I wanted to start a discussion to see if any community member has explored it or has any thought around it.

Additional context

• Came across this issue which is specifically around enabling the vector APIs and use lucene KNN related improvements

[reta]

@sohami why do we need another issue? I think we could retarget #9423 instead

[sohami]

@reta That issue I was reading is specifically discussing around KNN improvements and probably also getting lucene optimizations. I wanted to keep this issue to discuss if there is potential to make improvements in OpenSearch codebase itself.

[reta]

@sohami we have some many issue on the same topic, it is very difficult to keep track of the discussions, let's close this one and update the description on #9423 to have broader context.

[sohami]

@reta Earlier based on my search on issue title I didn't came across any other issue for SIMD exploration. My thought was having a parent issue to discuss broadly on SIMD optimization which can keep track of different specific discussions like enabling the lucene optimizations can be sub issues/tracks (hence linked it here). That sub-issue seems like is also tracked for 2.10.0 whereas this is more open ended to figure out what other potential improvements we can do in OpenSearch itself. What do you think ?

[reta]

opensearch-project/security#3213

this is already enabled by default on 3.0.0 and we be backported to 2.x once we get JDK-21, would OpenSearch or any other plugin use it - this is different question.

That sub-issue seems like is also tracked for 2.10.0 whereas this is more open ended to figure out what other potential improvements we can do in OpenSearch itself. What do you think ?

Up to you, #9423 is general ticket which could repurposed as I mentioned (the one for 2.11.0 would be JDK-21 update that would provide Project Panama).

[sohami]

Closing this as I have repurposed the issue #9423

[heemin32]

opensearch-project/security#3213

this is already enabled by default on 3.0.0 and we be backported to 2.x once we get JDK-21, would OpenSearch or any other plugin use it - this is different question.

That sub-issue seems like is also tracked for 2.10.0 whereas this is more open ended to figure out what other potential improvements we can do in OpenSearch itself. What do you think ?

Up to you, #9423 is general ticket which could repurposed as I mentioned (the one for 2.11.0 would be JDK-21 update that would provide Project Panama).

Correcting the reference link for enabling incubating api. #8601