Ensure template exists when creating data stream (elastic#56888)

Limit the creation of data streams only for namespaces that have a composable template with a data stream definition.

This way we ensure that mappings/settings have been specified and will be used at data stream creation and data stream rollover.

Also remove `timestamp_field` parameter from create data stream request and
let the create data stream api resolve the timestamp field
from the data stream definition snippet inside a composable template.

Relates to elastic#53100

docs/reference/indices/create-data-stream.asciidoc

@@ -23,18 +23,32 @@ addressed directly, data streams are integrated with the
 <<index-lifecycle-management, index lifecycle management (ILM)>> to facilitate
 the management of the time series data contained in their backing indices.
 
+A data stream can only be created if the namespace it targets has a component
+template exists with a `data_stream` definition.
+
 [source,console]
---------------------------------------------------
-PUT _data_stream/my-data-stream
+-----------------------------------
+PUT _index_template/template
 {
-  "timestamp_field": "@timestamp"
+  "index_patterns": ["my-data-stream*"],
+  "data_stream": {
+    "timestamp_field": "@timestamp"
+  }
 }
+-----------------------------------
+// TEST
+
+[source,console]
+--------------------------------------------------
+PUT _data_stream/my-data-stream
 --------------------------------------------------
+// TEST[continued]
 
 ////
 [source,console]
 -----------------------------------
 DELETE /_data_stream/my-data-stream
+DELETE /_index_template/template
 -----------------------------------
 // TEST[continued]
 ////
@@ -71,11 +85,3 @@ Data stream names must meet the following criteria:
 will count towards the 255 limit faster)
 --
 
-[[indices-create-data-stream-api-request-body]]
-==== {api-request-body-title}
-
-`timestamp_field`::
-(Required, string) The name of the timestamp field. This field must be present
-in all documents indexed into the data stream and must be of type
-<<date, `date`>> or <<date_nanos, `date_nanos`>>.
-

docs/reference/indices/delete-data-stream.asciidoc

@@ -9,10 +9,15 @@ Deletes an existing data stream along with its backing indices.
 ////
 [source,console]
 -----------------------------------
-PUT /_data_stream/my-data-stream
+PUT _index_template/template
 {
-  "timestamp_field" : "@timestamp"
+  "index_patterns": ["my-data-stream*"],
+  "data_stream": {
+    "timestamp_field": "@timestamp"
+  }
 }
+
+PUT /_data_stream/my-data-stream
 -----------------------------------
 // TESTSETUP
 ////
@@ -22,6 +27,13 @@ PUT /_data_stream/my-data-stream
 DELETE _data_stream/my-data-stream
 --------------------------------------------------
 
+////
+[source,console]
+-----------------------------------
+DELETE /_index_template/template
+-----------------------------------
+// TEST[continued]
+////
 
 [[delete-data-stream-api-request]]
 ==== {api-request-title}

docs/reference/indices/get-data-stream.asciidoc

@@ -9,10 +9,15 @@ Returns information about one or more data streams.
 ////
 [source,console]
 -----------------------------------
-PUT /_data_stream/my-data-stream
+PUT _index_template/template
 {
-  "timestamp_field" : "@timestamp"
+  "index_patterns": ["my-data-stream*"],
+  "data_stream": {
+    "timestamp_field": "@timestamp"
+  }
 }
+
+PUT /_data_stream/my-data-stream
 -----------------------------------
 // TESTSETUP
 ////
@@ -21,6 +26,7 @@ PUT /_data_stream/my-data-stream
 [source,console]
 -----------------------------------
 DELETE /_data_stream/my-data-stream
+DELETE /_index_template/template
 -----------------------------------
 // TEARDOWN
 ////

docs/reference/indices/index-templates.asciidoc

@@ -26,7 +26,7 @@ specify settings, mappings, and aliases.
 
 If a new index matches more than one index template, the index template with the highest priority is used.
 
-If an index is created with explicit settings and also matches an index template, 
+If an index is created with explicit settings and also matches an index template,
 the settings from the create index request take precedence over settings specified in the index template and its component templates.
 
 [source,console]
@@ -112,7 +112,7 @@ DELETE _component_template/*
 [[put-index-template-api-desc]]
 ==== {api-description-title}
 
-Creates or updates an index template. 
+Creates or updates an index template.
 
 // tag::index-template-def[]
 Index templates define <<index-modules-settings,settings>> and <<mapping,mappings>> that you can
@@ -555,3 +555,36 @@ PUT /_index_template/template_1
 --------------------------------------------------
 
 To check the `_meta`, you can use the <<indices-get-template, get index template>> API.
+
+[[data-stream-definition]]
+===== Data stream definition
+
+If a composable template should auto create a data stream instead of an index then
+a `data_stream` definition can be added to a composable template.
+
+[source,console]
+--------------------------------------------------
+PUT /_index_template/template_1
+{
+  "index_patterns": ["logs-*"],
+  "template": {
+    "mappings": {
+      "properties": {
+        "@timestamp": {
+          "type": "date"
+        }
+      }
+    }
+  },
+  "data_stream": {
+    "timestamp_field": "@timestamp"
+  }
+}
+--------------------------------------------------
+
+Required properties of a data stream definition:
+
+`timestamp_field`::
+(Required, string) The name of the timestamp field. This field must be present
+in all documents indexed into the data stream and must be of type
+<<date, `date`>> or <<date_nanos, `date_nanos`>>.

docs/reference/indices/rollover-index.asciidoc

@@ -226,11 +226,20 @@ The API returns the following response:
 ===== Roll over a data stream
 
 [source,console]
---------------------------------------------------
-PUT /_data_stream/my-data-stream <1>
+-----------------------------------
+PUT _index_template/template
 {
-  "timestamp_field": "date"
+  "index_patterns": ["my-data-stream*"],
+  "data_stream": {
+    "timestamp_field": "@timestamp"
+  }
 }
+-----------------------------------
+// TEST
+
+[source,console]
+--------------------------------------------------
+PUT /_data_stream/my-data-stream <1>
 
 # Add > 1000 documents to my-data-stream
 
@@ -243,6 +252,7 @@ POST /my-data-stream/_rollover <2>
   }
 }
 --------------------------------------------------
+// TEST[continued]
 // TEST[setup:huge_twitter]
 // TEST[s/# Add > 1000 documents to my-data-stream/POST _reindex?refresh\n{"source":{"index":"twitter"},"dest":{"index":"my-data-stream-000001"}}/]
 <1> Creates a data stream called `my-data-stream` with one initial backing index
@@ -286,6 +296,7 @@ The API returns the following response:
 [source,console]
 -----------------------------------
 DELETE /_data_stream/my-data-stream
+DELETE /_index_template/*
 -----------------------------------
 // TEST[continued]
 ////

rest-api-spec/src/main/resources/rest-api-spec/api/indices.create_data_stream.json

@@ -24,8 +24,7 @@
     "params":{
     },
     "body":{
-      "description":"The data stream definition",
-      "required":true
+      "description":"The data stream definition"
     }
   }
 }

rest-api-spec/src/main/resources/rest-api-spec/test/indices.data_stream/10_basic.yml

@@ -1,21 +1,39 @@
+setup:
+  - skip:
+      features: allowed_warnings
+  - do:
+      allowed_warnings:
+        - "index template [my-template1] has index patterns [simple-data-stream1*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [my-template1] will take precedence during new index creation"
+      indices.put_index_template:
+        name: my-template1
+        body:
+          index_patterns: [simple-data-stream1]
+          data_stream:
+            timestamp_field: '@timestamp'
+  - do:
+      allowed_warnings:
+        - "index template [my-template2] has index patterns [simple-data-stream2*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [my-template2] will take precedence during new index creation"
+      indices.put_index_template:
+        name: my-template2
+        body:
+          index_patterns: [simple-data-stream2]
+          data_stream:
+            timestamp_field: '@timestamp2'
+
 ---
 "Create data stream":
   - skip:
-      version: " - 7.8.99"
-      reason:  "data streams only supported in 7.9+"
+      version: " - 7.99.99"
+      reason:  "mute bwc until backported"
 
   - do:
       indices.create_data_stream:
         name: simple-data-stream1
-        body:
-          timestamp_field: "@timestamp"
   - is_true: acknowledged
 
   - do:
       indices.create_data_stream:
         name: simple-data-stream2
-        body:
-          timestamp_field: "@timestamp2"
   - is_true: acknowledged
 
   - do:
@@ -62,62 +80,57 @@
 ---
 "Create data stream with invalid name":
   - skip:
-      version: " - 7.8.99"
-      reason: "data streams only supported in 7.9+"
+      version: " - 7.99.99"
+      reason:  "mute bwc until backported"
 
   - do:
       catch: bad_request
       indices.create_data_stream:
         name: invalid-data-stream#-name
-        body:
-          timestamp_field: "@timestamp"
 
   - match: { status: 400 }
   - match: { error.root_cause.0.type: "illegal_argument_exception" }
+  - match: { error.root_cause.0.reason: "data_stream [invalid-data-stream#-name] must not contain '#'" }
 
 ---
 "Get data stream":
   - skip:
-      version: " - 7.8.99"
-      reason:  "data streams only supported in 7.9+"
+      version: " - 7.99.99"
+      reason:  "mute bwc until backported"
 
   - do:
       indices.create_data_stream:
-        name: get-data-stream1
-        body:
-          timestamp_field: "@timestamp"
+        name: simple-data-stream1
   - is_true: acknowledged
 
   - do:
       indices.create_data_stream:
-        name: get-data-stream2
-        body:
-          timestamp_field: "@timestamp2"
+        name: simple-data-stream2
   - is_true: acknowledged
 
   - do:
       indices.get_data_stream: {}
-  - match: { 0.name: get-data-stream1 }
+  - match: { 0.name: simple-data-stream1 }
   - match: { 0.timestamp_field: '@timestamp' }
   - match: { 0.generation: 1 }
-  - match: { 1.name: get-data-stream2 }
+  - match: { 1.name: simple-data-stream2 }
   - match: { 1.timestamp_field: '@timestamp2' }
   - match: { 1.generation: 1 }
 
   - do:
       indices.get_data_stream:
-        name: get-data-stream1
-  - match: { 0.name: get-data-stream1 }
+        name: simple-data-stream1
+  - match: { 0.name: simple-data-stream1 }
   - match: { 0.timestamp_field: '@timestamp' }
   - match: { 0.generation: 1 }
 
   - do:
       indices.get_data_stream:
-        name: get-data-*
-  - match: { 0.name: get-data-stream1 }
+        name: simple-data-stream*
+  - match: { 0.name: simple-data-stream1 }
   - match: { 0.timestamp_field: '@timestamp' }
   - match: { 0.generation: 1 }
-  - match: { 1.name: get-data-stream2 }
+  - match: { 1.name: simple-data-stream2 }
   - match: { 1.timestamp_field: '@timestamp2' }
   - match: { 1.generation: 1 }
 
@@ -136,25 +149,23 @@
 
   - do:
       indices.delete_data_stream:
-        name: get-data-stream1
+        name: simple-data-stream1
   - is_true: acknowledged
 
   - do:
       indices.delete_data_stream:
-        name: get-data-stream2
+        name: simple-data-stream2
   - is_true: acknowledged
 
 ---
 "Delete data stream with backing indices":
   - skip:
-      version: " - 7.8.99"
-      reason:  "data streams only supported in 7.9+"
+      version: " - 7.99.99"
+      reason:  "mute bwc until backported"
 
   - do:
       indices.create_data_stream:
-        name: delete-data-stream1
-        body:
-          timestamp_field: "@timestamp"
+        name: simple-data-stream1
   - is_true: acknowledged
 
   - do:
@@ -167,25 +178,25 @@
 
   - do:
       indices.get:
-        index: ['delete-data-stream1-000001', 'test_index']
+        index: ['simple-data-stream1-000001', 'test_index']
 
   - is_true: test_index.settings
-  - is_true: delete-data-stream1-000001.settings
+  - is_true: simple-data-stream1-000001.settings
 
   - do:
       indices.get_data_stream: {}
-  - match: { 0.name: delete-data-stream1 }
+  - match: { 0.name: simple-data-stream1 }
   - match: { 0.timestamp_field: '@timestamp' }
   - match: { 0.generation: 1 }
   - length: { 0.indices: 1 }
-  - match: { 0.indices.0.index_name: 'delete-data-stream1-000001' }
+  - match: { 0.indices.0.index_name: 'simple-data-stream1-000001' }
 
   - do:
       indices.delete_data_stream:
-        name: delete-data-stream1
+        name: simple-data-stream1
   - is_true: acknowledged
 
   - do:
       catch: missing
       indices.get:
-        index: "delete-data-stream1-000001"
+        index: "simple-data-stream1-000001"