[DOCS] Reformat data stream tutorial docs (elastic#57883) (elastic#57946

)

Creates a new page for a 'Set up a data stream' tutorial, based on
existing content in 'Data streams'.

Also adds tutorials for:

* Configuring an ILM policy for a data stream
* Indexing documents to a data stream
* Searching a data stream
* Manually rolling over a data stream

docs/reference/data-streams.asciidoc → docs/reference/data-streams/data-streams.asciidoc

@@ -1,7 +1,8 @@
-[chapter]
 [[data-streams]]
-= Data Streams
+= Data streams
 
+[partintro]
+--
 You can use data streams to index time-based data that's continuously generated.
 A data stream groups indices from the same time-based data source.
 A data stream tracks its indices, known as _backing indices_, using an ordered
@@ -116,3 +117,14 @@ The following APIs are available for managing data streams:
 * To get information about data streams, use the <<indices-get-data-stream, get data stream API>>.
 * To delete data streams, use the <<indices-delete-data-stream, delete data stream API>>.
 * To manually create a data stream, use the <<indices-create-data-stream, create data stream API>>.
+
+[discrete]
+[[data-streams-toc]]
+== In this section
+
+* <<set-up-a-data-stream>>
+* <<use-a-data-stream>>
+--
+
+include::set-up-a-data-stream.asciidoc[]
+include::use-a-data-stream.asciidoc[]

docs/reference/data-streams/set-up-a-data-stream.asciidoc

@@ -0,0 +1,243 @@
+[[set-up-a-data-stream]]
+== Set up a data stream
+
+To set up a data stream, follow these steps:
+
+. Check the <<data-stream-prereqs, prerequisites>>.
+. <<configure-a-data-stream-ilm-policy>>.
+. <<create-a-data-stream-template>>.
+. <<create-a-data-stream>>.
+
+After you set up a data stream, you can <<use-a-data-stream, use the data
+stream>> for indexing, searches, and other supported operations.
+
+[discrete]
+[[data-stream-prereqs]]
+=== Prerequisites
+
+* {es} data streams are intended for time-series data only. Each document
+indexed to a data stream must contain a shared timestamp field.
++
+TIP: Data streams work well with most common log formats. While no schema is
+required to use data streams, we recommend the {ecs-ref}[Elastic Common Schema
+(ECS)].
+
+* Data streams are designed to be append-only. While you can index new documents
+directly to a data stream, you cannot use a data stream to directly update or
+delete individual documents. To update or delete specific documents in a data
+stream, submit a <<docs-delete,delete>> or <<docs-update,update>> API request to
+the backing index containing the document.
+
+
+[discrete]
+[[configure-a-data-stream-ilm-policy]]
+=== Optional: Configure an {ilm-init} lifecycle policy for a data stream
+
+You can use <<index-lifecycle-management,{ilm} ({ilm-init})>> to automatically
+manage a data stream's backing indices. For example, you could use {ilm-init}
+to:
+
+* Spin up a new write index for the data stream when the current one reaches a
+  certain size or age.
+* Move older backing indices to slower, less expensive hardware.
+* Delete stale backing indices to enforce data retention standards.
+
+To use {ilm-init} with a data stream, you must
+<<set-up-lifecycle-policy,configure a lifecycle policy>>. This lifecycle policy
+should contain the automated actions to take on backing indices and the
+triggers for such actions.
+
+TIP: While optional, we recommend using {ilm-init} to scale data streams in
+production.
+
+.*Example*
+[%collapsible]
+====
+The following <<ilm-put-lifecycle,create lifecycle policy API>> request
+configures the `logs_policy` lifecycle policy.
+
+The `logs_policy` policy uses the <<ilm-rollover,`rollover` action>> to create a
+new write index for the data stream when the current one reaches 25GB in size.
+The policy also deletes backing indices 30 days after their rollover.
+
+[source,console]
+----
+PUT /_ilm/policy/logs_policy
+{
+  "policy": {
+    "phases": {
+      "hot": {
+        "actions": {
+          "rollover": {
+            "max_size": "25GB"
+          }
+        }
+      },
+      "delete": {
+        "min_age": "30d",
+        "actions": {
+          "delete": {}
+        }
+      }
+    }
+  }
+}
+----
+====
+
+
+[discrete]
+[[create-a-data-stream-template]]
+=== Create a composable template for a data stream
+
+Each data stream requires a <<indices-templates,composable template>>. The data
+stream uses this template to create its backing indices.
+
+Composable templates for data streams must contain:
+
+* A name or wildcard (`*`) pattern for the data stream in the `index_patterns`
+  property.
+
+* A `data_stream` definition containing the `timestamp_field` property.
+  This timestamp field must be included in every document indexed to the data
+  stream.
+
+* A <<date,`date`>> or <<date_nanos,`date_nanos`>> field mapping for the
+  timestamp field specified in the `timestamp_field` property.
+
+* If you intend to use {ilm-init}, you must specify the
+  <<configure-a-data-stream-ilm-policy,lifecycle policy>> in the 
+  `index.lifecycle.name` setting.
+
+You can also specify other mappings and settings you'd like to apply to the
+stream's backing indices.
+
+.*Example*
+[%collapsible]
+====
+The following <<indices-templates,put composable template API>> request
+configures the `logs_data_stream` template.
+
+[source,console]
+----
+PUT /_index_template/logs_data_stream
+{
+  "index_patterns": [ "logs*" ],
+  "data_stream": {
+    "timestamp_field": "@timestamp"
+  },
+  "template": {
+    "mappings": {
+      "properties": {
+        "@timestamp": {
+          "type": "date"
+        }
+      }
+    },
+    "settings": {
+      "index.lifecycle.name": "logs_policy"
+    }
+  }
+}
+----
+// TEST[continued]
+====
+
+[discrete]
+[[create-a-data-stream]]
+=== Create a data stream
+
+With a composable template, you can create a data stream using one of two
+methods:
+
+* Submit an <<add-documents-to-a-data-stream,indexing request>> to a target
+matching the name or wildcard pattern defined in the template's `index_patterns`
+property.
++
+--
+If the indexing request's target doesn't exist, {es} creates the data stream and
+uses the target name as the name for the stream.
+
+NOTE: Data streams support only specific types of indexing requests. See
+<<add-documents-to-a-data-stream>>.
+
+.*Example: Index documents to create a data stream*
+[%collapsible]
+====
+The following <<docs-index_,index API>> request targets `logs`, which matches
+the wildcard pattern for the `logs_data_stream` template. Because no existing
+index or data stream uses this name, this request creates the `logs` data stream
+and indexes the document to it.
+
+[source,console]
+----
+POST /logs/_doc/
+{
+  "@timestamp": "2020-12-06T11:04:05.000Z",
+  "user": {
+    "id": "vlb44hny"
+  },
+  "message": "Login attempt failed"
+}
+----
+// TEST[continued]
+
+The API returns the following response. Note the `_index` property contains
+`.ds-logs-000001`, indicating the document was indexed to the write index of the
+new `logs` data stream.
+
+[source,console-result]
+----
+{
+  "_index": ".ds-logs-000001",
+  "_id": "qecQmXIBT4jB8tq1nG0j",
+  "_type": "_doc",
+  "_version": 1,
+  "result": "created",
+  "_shards": {
+    "total": 2,
+    "successful": 1,
+    "failed": 0
+  },
+  "_seq_no": 0,
+  "_primary_term": 1
+}
+----
+// TESTRESPONSE[s/"_id": "qecQmXIBT4jB8tq1nG0j"/"_id": $body._id/]
+====
+--
+
+* Use the <<indices-create-data-stream,create data stream API>> to manually
+create a data stream. The name of the data stream must match the
+name or wildcard pattern defined in the template's `index_patterns` property.
++
+--
+.*Example: Manually create a data stream*
+[%collapsible]
+====
+The following <<indices-create-data-stream,create data stream API>> request
+targets `logs_alt`, which matches the wildcard pattern for the
+`logs_data_stream` template. Because no existing index or data stream uses this
+name, this request creates the `logs_alt` data stream.
+
+[source,console]
+----
+PUT /_data_stream/logs_alt
+----
+// TEST[continued]
+====
+--
+
+////
+[source,console]
+----
+DELETE /_data_stream/logs
+
+DELETE /_data_stream/logs_alt
+
+DELETE /_index_template/logs_data_stream
+
+DELETE /_ilm/policy/logs_policy
+----
+// TEST[continued]
+////