Warn if file path is invalid. (#793)

* Warn if file path is invalid. Signed-off-by: dblock <dblock@amazon.com> * Comply with invalid-path-detected. Signed-off-by: dblock <dblock@amazon.com> --------- Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
opensearch-project · Jan 20, 2025 · d8e4583 · d8e4583
1 parent d3bdef9
commit d8e4583
Show file tree

Hide file tree

Showing 101 changed files with 266 additions and 159 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -53,6 +53,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Added `GET /_plugins/_ml/stats/`, `stats/{stat}`, `{nodeId}/stats/`, `{nodeId}/stats/{stat}` ([#794](https://github.com/opensearch-project/opensearch-api-specification/pull/794))
 - Added `GET`, `POST /_plugins/_ml/tasks/_search`, `GET /_plugins/_ml/tools`, `tools/{tool_name}` ([#797](https://github.com/opensearch-project/opensearch-api-specification/pull/797))
 - Added `POST /_plugins/_ml/agents/{agent_id}/_execute`, `GET /_plugins/_ml/agents/{agent_id}`, `GET`, `POST /_plugins/_ml/agents/_search` ([#798](https://github.com/opensearch-project/opensearch-api-specification/pull/798))
+- Added a warning for test file names that don't match the API being tested ([#793](https://github.com/opensearch-project/opensearch-api-specification/pull/793))
 
 ### Removed
 - Removed unsupported `_common.mapping:SourceField`'s `mode` field and associated `_common.mapping:SourceFieldMode` enum ([#652](https://github.com/opensearch-project/opensearch-api-specification/pull/652))

diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md
@@ -20,6 +20,7 @@
     - [Waiting for Tasks](#waiting-for-tasks)
     - [Warnings](#warnings)
       - [multiple-paths-detected](#multiple-paths-detected)
+      - [invalid-path-detected](#invalid-path-detected)
       - [Suppressing Warnings](#suppressing-warnings)
   - [Collecting Test Coverage](#collecting-test-coverage)
     - [Coverage Summary](#coverage-summary)
@@ -358,7 +359,7 @@ For example, an ML task returns `CREATED` when created, and `COMPLETED` when it'
 
 #### multiple-paths-detected
 
-The test runner expects all tests in the same file to be variation of the same path in order to keep tests well-organized. Otherwise, a warning will be emitted.
+The test runner expects all tests in the same file to be variation of the same path in order to keep tests well-organized. Prerequisites to the API being tested should be moved to `prologues`, and any cleanup to `epilogues`. Otherwise, a warning will be emitted.
 
 ```
 WARNING Multiple paths detected, please group similar tests together and move paths not being tested to prologues or epilogues.
@@ -367,6 +368,15 @@ WARNING Multiple paths detected, please group similar tests together and move pa
   /{index}
 ```
 
+#### invalid-path-detected
+
+The test file names are expected to match the one API being tested in that file. Otherwise, a warning will be emitted.
+
+```
+PASSED  plugins/ml/ml/train_and_predict.yaml (tests/plugins/ml/ml/train_and_predict.yaml)
+WARNING Invalid path detected, please move /tests/plugins/ml/ml/train_and_predict.yaml to ml/train_predict.yaml.
+```
+
 #### Suppressing Warnings
 
 The test runner may generate warnings that can be suppressed with `warnings:` at story or chapter level. For example, to suppress the multiple paths detected warning.

diff --git a/json_schemas/test_story.schema.yaml b/json_schemas/test_story.schema.yaml
@@ -252,4 +252,8 @@ definitions:
         type: boolean
         default: true
         description: Enable/disable warnings about multiple paths being tested in the same story.
+      invalid-path-detected:
+        type: boolean
+        default: true
+        description: Enable/disable warnings about file paths that do not match paths tested in the story.
     additionalProperties: false
diff --git a/spec/namespaces/ml.yaml b/spec/namespaces/ml.yaml
@@ -781,7 +781,7 @@ paths:
       responses:
         '200':
           $ref: '#/components/responses/ml.get_stats@200'
-  /_plugins/_ml/{node_id}/stats/:
+  /_plugins/_ml/{node_id}/stats:
     get:
       operationId: ml.get_stats.2
       x-operation-group: ml.get_stats

diff --git a/tests/default/_core/analyze/filter/asciifolding.yaml b/tests/default/_core/analyze/filter/asciifolding.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test /_analyze with a filter.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.17'
 chapters:
   - synopsis: Apply a asciifolding filter.

diff --git a/tests/default/_core/analyze/filter/lowercase.yaml b/tests/default/_core/analyze/filter/lowercase.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test /_analyze with a filter.
+warnings:
+  invalid-path-detected: false
 chapters:
   - synopsis: Apply a lowercase character filter.
     path: /_analyze

diff --git a/tests/default/_core/analyze/filter/persian_stem.yaml b/tests/default/_core/analyze/filter/persian_stem.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test /_analyze with a filter.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.17'
 chapters:
   - synopsis: Apply a persian_stem filter.

diff --git a/tests/default/_core/analyze/filter/porterstem.yaml b/tests/default/_core/analyze/filter/porterstem.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test /_analyze with a filter.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.17'
 chapters:
   - synopsis: Apply a porter_stem filter.

diff --git a/tests/default/_core/analyze/filter/stop.yaml b/tests/default/_core/analyze/filter/stop.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test /_analyze with a filter.
+warnings:
+  invalid-path-detected: false
 chapters:
   - synopsis: Combine a lowercase translation with a stop filter.
     path: /_analyze

diff --git a/tests/default/_core/analyze/filter/uppercase.yaml b/tests/default/_core/analyze/filter/uppercase.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test /_analyze with a filter.
+warnings:
+  invalid-path-detected: false
 chapters:
   - synopsis: Apply an uppercase character filter.
     path: /_analyze

diff --git a/tests/default/_core/info.yaml b/tests/default/_core/info.yaml
@@ -2,6 +2,8 @@
 $schema: ../../../json_schemas/test_story.schema.yaml
 
 description: Test root endpoint.
+warnings:
+  invalid-path-detected: false
 distributions:
   excluded:
     - amazon-serverless

diff --git a/tests/default/_core/msearch/index.yaml → tests/default/_core/msearch.yaml b/tests/default/_core/msearch/index.yaml → tests/default/_core/msearch.yaml
@@ -1,4 +1,4 @@
-$schema: ../../../../json_schemas/test_story.schema.yaml
+$schema: ../../../json_schemas/test_story.schema.yaml
 
 description: Test document multi-search.
 epilogues:

diff --git a/tests/default/_core/reindex/pipeline.yaml b/tests/default/_core/reindex/pipeline.yaml
@@ -2,6 +2,7 @@ $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test reindex with a Search pipeline.
 warnings:
+  invalid-path-detected: false
   multiple-paths-detected: false
 epilogues:
   - path: /movies

diff --git a/tests/default/_core/scripts/context.yaml b/tests/default/_core/scripts/context.yaml
@@ -1,7 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test the _scripts contexts endpoint.
-
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_scripts/add_ratings
     method: DELETE

diff --git a/tests/default/_core/search/_source.yaml b/tests/default/_core/search/_source.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint (_source).
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies/_doc
     method: POST

diff --git a/tests/default/_core/search/aggregations/avg.yaml b/tests/default/_core/search/aggregations/avg.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test avg aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_bulk
     method: POST

diff --git a/tests/default/_core/search/aggregations/historgram.yaml b/tests/default/_core/search/aggregations/historgram.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test histogram aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_bulk
     method: POST

diff --git a/tests/default/_core/search/aggregations/max.yaml b/tests/default/_core/search/aggregations/max.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test max aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_bulk
     method: POST

diff --git a/tests/default/_core/search/aggregations/min.yaml b/tests/default/_core/search/aggregations/min.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test min aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_bulk
     method: POST

diff --git a/tests/default/_core/search/aggregations/nested.yaml b/tests/default/_core/search/aggregations/nested.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: books
     method: PUT

diff --git a/tests/default/_core/search/aggregations/range.yaml b/tests/default/_core/search/aggregations/range.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test range aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_bulk
     method: POST

diff --git a/tests/default/_core/search/aggregations/sum.yaml b/tests/default/_core/search/aggregations/sum.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test sum aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /_bulk
     method: POST

diff --git a/tests/default/_core/search/aggregations/terms.yaml b/tests/default/_core/search/aggregations/terms.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test terms aggregation.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies
     method: PUT

diff --git a/tests/default/_core/search/cancel_after_time_interval.yaml b/tests/default/_core/search/cancel_after_time_interval.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with cancel_after_time_interval.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies/_doc
     method: POST

diff --git a/tests/default/_core/search/knn/on_disk.yaml b/tests/default/_core/search/knn/on_disk.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with knn query.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.17'
 
 prologues:

diff --git a/tests/default/_core/search/match.yaml b/tests/default/_core/search/match.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with match query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies/_doc
     method: POST

diff --git a/tests/default/_core/search/multi_match.yaml b/tests/default/_core/search/multi_match.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with multi_match query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies/_doc
     method: POST

diff --git a/tests/default/_core/search/phase_took.yaml b/tests/default/_core/search/phase_took.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with phase_took.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies/_doc
     method: POST

diff --git a/tests/default/_core/search/pipeline.yaml b/tests/default/_core/search/pipeline.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test the creation of a search pipeline.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.9'
 epilogues:
   - path: /_search/pipeline/empty-pipeline

diff --git a/tests/default/_core/search/pipeline/request_processor/filter_query.yaml b/tests/default/_core/search/pipeline/request_processor/filter_query.yaml
@@ -2,6 +2,8 @@ $schema: ../../../../../../json_schemas/test_story.schema.yaml
 
 description: |-
   Test the creation of a search pipeline with a filter_query request processor.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.8'
 prologues:
   - path: /_bulk

diff --git a/tests/default/_core/search/pipeline/response_processor/rename_field.yaml b/tests/default/_core/search/pipeline/response_processor/rename_field.yaml
@@ -2,6 +2,8 @@ $schema: ../../../../../../json_schemas/test_story.schema.yaml
 
 description: |-
   Test the creation of a search pipeline with a response processor.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.8'
 prologues:
   - path: /movies/_doc/1

diff --git a/tests/default/_core/search/pipeline/response_processor/sort.yaml b/tests/default/_core/search/pipeline/response_processor/sort.yaml
@@ -2,6 +2,8 @@ $schema: ../../../../../../json_schemas/test_story.schema.yaml
 
 description: |-
   Test the creation of a search pipeline with a response processor.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.16'
 prologues:
   - path: /movies/_doc/1

diff --git a/tests/default/_core/search/pipeline/response_processor/split_and_sort.yaml b/tests/default/_core/search/pipeline/response_processor/split_and_sort.yaml
@@ -2,6 +2,8 @@ $schema: ../../../../../../json_schemas/test_story.schema.yaml
 
 description: |-
   Test the creation of a search pipeline with split and sort response processors.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.17'
 prologues:
   - path: /numbers/_doc/1

diff --git a/tests/default/_core/search/point_in_time.yaml b/tests/default/_core/search/point_in_time.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../json_schemas/test_story.schema.yaml
 
 description: Test point-in-time.
+warnings:
+  invalid-path-detected: false
 version: '>= 2.4'
 epilogues:
   - path: /_search/point_in_time/_all

diff --git a/tests/default/_core/point_in_time/all.yaml → ...fault/_core/search/point_in_time/all.yaml b/tests/default/_core/point_in_time/all.yaml → ...fault/_core/search/point_in_time/all.yaml
@@ -1,4 +1,4 @@
-$schema: ../../../../json_schemas/test_story.schema.yaml
+$schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test point-in-time.
 version: '>= 2.4'

diff --git a/tests/default/_core/search/query/fuzzy.yaml b/tests/default/_core/search/query/fuzzy.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with fuzzy query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /movies
     method: PUT

diff --git a/tests/default/_core/search/query/geo_bounding_box.yaml b/tests/default/_core/search/query/geo_bounding_box.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with geo_bounding_box query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /cinemas
     method: PUT

diff --git a/tests/default/_core/search/query/geo_distance.yaml b/tests/default/_core/search/query/geo_distance.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with geo_distance query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /cinemas
     method: PUT

diff --git a/tests/default/_core/search/query/geo_shape_point.yaml b/tests/default/_core/search/query/geo_shape_point.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with geo_shape query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /cinemas
     method: PUT

diff --git a/tests/default/_core/search/query/geo_shape_shape.yaml b/tests/default/_core/search/query/geo_shape_shape.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test search endpoint with geo_shape query.
+warnings:
+  invalid-path-detected: false
 prologues:
   - path: /cinemas
     method: PUT

diff --git a/tests/default/_core/search/query/script.yaml b/tests/default/_core/search/query/script.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../json_schemas/test_story.schema.yaml
 
 description: Test ScriptQuery functionality.
+warnings:
+  invalid-path-detected: false
 
 prologues:
   - path: /movies

diff --git a/tests/default/_core/search/query/terms/array_terms.yaml b/tests/default/_core/search/query/terms/array_terms.yaml
@@ -1,6 +1,8 @@
 $schema: ../../../../../../json_schemas/test_story.schema.yaml
 
 description: Comprehensive test suite for TermsQuery, including array of terms.
+warnings:
+  invalid-path-detected: false
 version: '>= 1.2'
 
 prologues: