[BUG] HTTP status code changed on 2.3.0 on querying for non-existing index

[Yury-Fridlyand]

How can one reproduce the bug?

curl -v -s -XPOST "http://localhost:9200/_plugins/_sql" -H 'Content-Type: application/json' -d "{\"query\": \"select * from non-existed\"}"

OpenSearch 2.2.1:

HTTP/1.1 400 Bad Request
{
  "error": {
    "reason": "Error occurred in OpenSearch engine: no such index [non-existed]",
    "details": "org.opensearch.index.IndexNotFoundException: no such index [non-existed]\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",
    "type": "IndexNotFoundException"
  },
  "status": 404
}

2.3.0:

HTTP/1.1 503 Service Unavailable
{
  "error": {
    "reason": "Error occurred in OpenSearch engine: no such index [non-existed]",
    "details": "org.opensearch.index.IndexNotFoundException: no such index [non-existed]\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",
    "type": "IndexNotFoundException"
  },
  "status": 404
}

Payloads are same, but status codes are different.

[dai-chen]

Thanks for reporting this issue! This is probably because the approach of getting index mapping is changed. Previously the code on each OpenSearch node just read status in ClusterState object locally. In this way, we can avoid blocking operation during parsing, analyzing and planning and make all running on NIO thread. However, this is changed to calling OpenSearch API due to some other issue.

Previously index not found exception is thrown and handled by https://github.com/opensearch-project/sql/blob/2.2/legacy/src/main/java/org/opensearch/sql/legacy/plugin/RestSqlAction.java#L165 in NIO thread. Now it's thrown from worker thread and handled by https://github.com/opensearch-project/sql/blob/2.2/legacy/src/main/java/org/opensearch/sql/legacy/plugin/RestSQLQueryAction.java#L177 without isClientError check. We may need to check here as we move everything to worker thread.

[acarbonetto]

Test skipped for change:

sql/sql-cli/tests/test_opensearch_connection.py

Line 42 in 63c5137

@pytest.mark.skip(reason="Test is not compatible with OpenSearch >= 2.3.0, it returns HTTP/503 instead of HTTP/400")

[dai-chen]

I found similar issue in metrics. In legacy engine, index not found is treated as client error. However, the issue reported here cause it treated as server error in v2 engine which may cause wrong alarm in monitor system from user side.