Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent segment search GA and API changes #6356

Merged
merged 16 commits into from
Feb 12, 2024
Merged
76 changes: 37 additions & 39 deletions _api-reference/index-apis/stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,41 @@ GET /testindex/_stats
```
{% include copy-curl.html %}

#### Example request: Comma-separated list of indexes

```json
GET /testindex1,testindex2/_stats
```
{% include copy-curl.html %}

#### Example request: Wildcard expression

```json
GET /testindex*/_stats
```
{% include copy-curl.html %}

#### Example request: Specific stats

```json
GET /testindex/_stats/refresh,flush
```
{% include copy-curl.html %}

#### Example request: Expand wildcards

```json
GET /testindex*/_stats?expand_wildcards=open,hidden
```
{% include copy-curl.html %}

#### Example request: Shard-level statistics

```json
GET /testindex/_stats?level=shards
```
{% include copy-curl.html %}

#### Example response

By default, the returned statistics are aggregated in the `primaries` and `total` aggregations. The `primaries` aggregation contains statistics for the primary shards. The `total` aggregation contains statistics for both primary and replica shards. The following is an example Index Stats API response:
Expand Down Expand Up @@ -773,46 +808,9 @@ By default, the returned statistics are aggregated in the `primaries` and `total
```
</details>

#### Example request: Comma-separated list of indexes

```json
GET /testindex1,testindex2/_stats
```
{% include copy-curl.html %}

#### Example request: Wildcard expression

```json
GET /testindex*/_stats
```
{% include copy-curl.html %}

#### Example request: Specific stats

```json
GET /testindex/_stats/refresh,flush
```
{% include copy-curl.html %}

#### Example request: Expand wildcards

```json
GET /testindex*/_stats?expand_wildcards=open,hidden
```
{% include copy-curl.html %}

#### Example request: Shard-level statistics

```json
GET /testindex/_stats?level=shards
```
{% include copy-curl.html %}

## Concurrent segment search

Starting in OpenSearch 2.10, [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/) allows each shard-level request to search segments in parallel during the query phase. If you [enable the experimental concurrent segment search feature flag]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search#enabling-the-feature-flag), the Index Stats API response will contain several additional fields with statistics about slices (units of work executed by a thread). These fields will be provided whether or not the cluster and index settings for concurrent segment search are enabled. For more information about slices, see [Concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search#searching-segments-concurrently).
## Response fields

The following table provides information about the added response fields.
The following table provides information about the response fields.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only having these new response fields specific to concurrent search looks incomplete. Ideally, we should add details about all the fields in the table or just explicitly say these are concurrent search related fields in the response.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sohami @ticheng-aws Fully agreed. Could you provide all the info about other fields please so I can include it here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kolchfa-aws it looks like all the index stats are already documented in the node stats page here: https://opensearch.org/docs/latest/api-reference/nodes-apis/nodes-stats/#indices

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jed326 Thanks! So should I remove this section and add these to the nodes-stats table?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you already added these to the node stats table in this PR. I think it's good to have the search stats separately on this page too though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I'll add a link to the nodes stats API so users can reference descriptions for all fields.


|Response field | Description |
|:--- |:--- |
Expand Down
9 changes: 4 additions & 5 deletions _api-reference/nodes-apis/nodes-stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -831,6 +831,10 @@ get.missing_total | Integer | The number of failed get operations.
get.missing_time_in_millis | Integer | The total time for all failed get operations, in milliseconds.
get.current | Integer | The number of get operations that are currently running.
search | Object | Statistics about the search operations for the node.
search.concurrent_avg_slice_count | Integer | The average slice count of all search requests. This is computed as the total slice count divided by the total number of concurrent search requests.
search.concurrent_query_total |Integer | The total number of query operations that use concurrent segment search.
search.concurrent_query_time_in_millis | Integer | The total amount of time taken by all query operations that use concurrent segment search, in milliseconds.
search.concurrent_query_current |Integer | The number of currently running query operations that use concurrent segment search.
search.open_contexts | Integer | The number of open search contexts.
search.query_total | Integer | The total number of shard query operations.
search.query_time_in_millis | Integer | The total amount of time for all shard query operations, in milliseconds.
Expand Down Expand Up @@ -1259,11 +1263,6 @@ Field | Field type | Description
admission_control.global_cpu_usage.transport.rejection_count.search | Integer | The total number of search rejections in the transport layer when the node CPU usage limit was breached. In this case, additional search requests are rejected until the system recovers.
admission_control.global_cpu_usage.transport.rejection_count.indexing | Integer | The total number of indexing rejections in the transport layer when the node CPU usage limit was breached. In this case, additional indexing requests are rejected until the system recovers.


## Concurrent segment search

Starting in OpenSearch 2.10, [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/) allows each shard-level request to search segments in parallel during the query phase. If you [enable the experimental concurrent segment search feature flag]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search#enabling-the-feature-flag), the Nodes Stats API response will contain several additional fields with statistics about slices (units of work executed by a thread). For the descriptions of those fields, see [Index Stats API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/stats#concurrent-segment-search).

## Required permissions

If you use the Security plugin, make sure you have the appropriate permissions: `cluster:monitor/nodes/stats`.
Loading
Loading