[7.x] [DOCS] Revamps Anomaly Detection section (#592)

* [7.x] [DOCS] Revamps Anomaly Detection section.

docs/en/stack/ml/anomaly-detection/create-jobs.asciidoc

@@ -1,6 +1,6 @@
 [role="xpack"]
 [[create-jobs]]
-== Creating {anomaly-jobs}
+=== Creating {anomaly-jobs}
 
 {anomaly-jobs-cap} contain the configuration information and metadata
 necessary to perform an analytics task.

docs/en/stack/ml/anomaly-detection/examples.asciidoc

@@ -0,0 +1,34 @@
+[role="xpack"]
+[testenv="platinum"]
+[[anomaly-examples]]
+== {anomaly-detect-cap} examples
+++++
+<titleabbrev>Examples</titleabbrev>
+++++
+
+Though it is quite simple to analyze your data and provide quick {ml} results,
+gaining deep insights might require some additional planning and configuration.
+The scenarios in this section describe some best practices for generating useful
+{ml} results and insights from your data.
+
+* <<ml-configuring-url>>
+* <<ml-configuring-aggregation>>
+* <<ml-configuring-categories>>
+* <<ml-configuring-detector-custom-rules>>
+* <<ml-configuring-pop>>
+* <<ml-configuring-transform>>
+* <<ml-delayed-data-detection>>
+
+include::{es-repo-dir}/ml/anomaly-detection/customurl.asciidoc[]
+
+include::{es-repo-dir}/ml/anomaly-detection/aggregations.asciidoc[]
+
+include::{es-repo-dir}/ml/anomaly-detection/detector-custom-rules.asciidoc[]
+
+include::{es-repo-dir}/ml/anomaly-detection/categories.asciidoc[]
+
+include::{es-repo-dir}/ml/anomaly-detection/populations.asciidoc[]
+
+include::{es-repo-dir}/ml/anomaly-detection/transforms.asciidoc[]
+
+include::{es-repo-dir}/ml/anomaly-detection/delayed-data-detection.asciidoc[]

docs/en/stack/ml/anomaly-detection/index.asciidoc

@@ -1,40 +1,40 @@
 [role="xpack"]
 [[xpack-ml]]
-= {ml-cap} anomaly detection
+= {ml-cap} {anomaly-detect}
 
 [partintro]
 --
-Machine learning is tightly integrated with the {stack}. Data is pulled
-from {es} for analysis and anomaly results are displayed in {kib} dashboards.
+{ml-cap} is tightly integrated with the {stack}. Data is pulled from {es} for 
+analysis and anomaly results are displayed in {kib} dashboards.
 
 * <<ml-overview>>
 * <<create-jobs>>
+* <<ml-concepts>>
+* <<ml-configuration>>
 //* <<ml-getting-started>>
-* <<ml-configuring>>
-* <<stopping-ml>>
-* <<ml-troubleshooting, Troubleshooting Machine Learning>>
 * <<ml-api-quickref>>
 * <<ml-functions>>
+* <<anomaly-examples>>
+* <<ml-limitations>>
+//* <<ml-troubleshooting>>
 
 
 --
 
 include::overview.asciidoc[]
 
-include::create-jobs.asciidoc[]
+include::ml-concepts.asciidoc[]
 
-include::job-tips.asciidoc[]
+include::ml-configuration.asciidoc[]
 
 //include::getting-started.asciidoc[]
 
-include::{es-repo-dir}/ml/anomaly-detection/configuring.asciidoc[]
-
-include::{es-repo-dir}/ml/anomaly-detection/stopping-ml.asciidoc[]
-
 include::api-quickref.asciidoc[]
 
 include::{es-repo-dir}/ml/anomaly-detection/functions.asciidoc[]
 
-include::troubleshooting.asciidoc[]
+include::examples.asciidoc[]
 
 include::limitations.asciidoc[]
+
+//include::troubleshooting.asciidoc[]

docs/en/stack/ml/anomaly-detection/job-tips.asciidoc

@@ -1,6 +1,6 @@
 [role="xpack"]
 [[job-tips]]
-=== Machine learning job tips
+==== Machine learning job tips
 ++++
 <titleabbrev>Job tips</titleabbrev>
 ++++
@@ -11,7 +11,7 @@ suggestions, you can create jobs that are more likely to produce insightful {ml}
 results.
 
 [[bucket-span]]
-==== Bucket span
+===== Bucket span
 
 The bucket span is the time interval that {ml} analytics use to summarize and
 model data for your job. When you create an {anomaly-job} in {kib}, you can
@@ -25,7 +25,7 @@ than the estimated value, you receive an informational message. For more
 information about choosing an appropriate bucket span, see <<ml-buckets>>.
 
 [[cardinality]]
-==== Cardinality
+===== Cardinality
 
 If there are logical groupings of related entities in your data, {ml} analytics
 can make data models and generate results that take these groupings into
@@ -42,7 +42,7 @@ Likewise if you are performing population analysis and the cardinality of the
 field to use. For more information, see <<ml-configuring-pop>>.
 
 [[detectors]]
-==== Detectors
+===== Detectors
 
 Each {anomaly-job} must have one or more _detectors_. A detector applies an
 analytical function to specific fields in your data. If your job does not
@@ -54,7 +54,7 @@ duplicates if they have the same `function`, `field_name`, `by_field_name`,
 `over_field_name` and `partition_field_name`. 
 
 [[influencers]]
-==== Influencers
+===== Influencers
 
 When you create an {anomaly-job}, you can specify _influencers_, which are also 
 sometimes referred to as _key fields_. Picking an influencer is strongly
@@ -75,7 +75,7 @@ overwhelming and there is a small overhead to the analysis.
 The job creation wizards in {kib} can suggest which fields to use as influencers.
 
 [[model-memory-limits]]
-==== Model memory limits
+===== Model memory limits
 
 For each {anomaly-job}, you can optionally specify a `model_memory_limit`, which
 is the approximate maximum amount of memory resources that are required for

docs/en/stack/ml/anomaly-detection/jobs.asciidoc

@@ -17,7 +17,7 @@ are considered anomalous. For example, you can specify whether entities are
 analyzed relative to their own previous behavior or relative to other entities
 in a population. There are also multiple options for splitting the data into
 categories and partitions. Some of these more advanced job configurations
-are described in the following section: <<ml-configuring>>.
+are described in the following section: <<anomaly-examples>>.
 
 For a description of all the job properties, see
 {ref}/ml-job-resource.html[{anomaly-jobs-cap} resources].

docs/en/stack/ml/anomaly-detection/ml-concepts.asciidoc

@@ -0,0 +1,25 @@
+[role="xpack"]
+[[ml-concepts]]
+== Concepts
+
+This section explains the fundamental concepts of the Elastic {ml} 
+{anomaly-detect} feature.
+
+* <<ml-jobs>>
+* <<ml-dfeeds>>
+* <<ml-buckets>>
+* <<ml-calendars>>
+* <<ml-rules>>
+* <<ml-nodes>>
+
+include::jobs.asciidoc[]
+
+include::datafeeds.asciidoc[]
+
+include::buckets.asciidoc[]
+
+include::calendars.asciidoc[]
+
+include::rules.asciidoc[]
+
+include::architecture.asciidoc[]

docs/en/stack/ml/anomaly-detection/ml-configuration.asciidoc

@@ -0,0 +1,37 @@
+[role="xpack"]
+[[ml-configuration]]
+== Configuring {anomaly-detect}
+
+If you want to use {ml-features}, there must be at least one {ml} node in
+your cluster and all master-eligible nodes must have {ml} enabled. By default,
+all nodes are {ml} nodes. For more information about these settings, see 
+{ref}/modules-node.html#ml-node[{ml} nodes].
+
+To use the {ml-features} to analyze your data, you can create an {anomaly-job}
+and send your data to that job.
+
+//* If your data is stored in {es}:
+
+//** You can create a {dfeed}, which retrieves data from {es} for analysis.
+//** You can use {kib} to expedite the creation of jobs and {dfeeds}.
+
+//* If your data is not stored in {es}, you can
+//{ref}/ml-post-data.html[POST data] from any source directly to an API.
+
+The results of {ml} analysis are stored in {es} and you can use {kib} to help
+you visualize and explore the results.
+
+//For a tutorial that walks you through these configuration steps,
+//see <<ml-getting-started>>.
+
+* <<create-jobs>>
+* <<stopping-ml>>
+
+After you learn how to create and stop {anomaly-detect} jobs, you can check the 
+<<anomaly-examples>> for more advanced settings and scenarios.
+
+include::create-jobs.asciidoc[]
+
+include::job-tips.asciidoc[]
+
+include::stopping-ml.asciidoc[]

docs/en/stack/ml/anomaly-detection/overview.asciidoc

@@ -3,11 +3,5 @@
 == Overview
 
 include::analyzing.asciidoc[]
-include::forecasting.asciidoc[]
-include::jobs.asciidoc[]
-include::datafeeds.asciidoc[]
-include::buckets.asciidoc[]
-include::calendars.asciidoc[]
-include::rules.asciidoc[]
-include::architecture.asciidoc[]
 
+include::forecasting.asciidoc[]

docs/en/stack/ml/anomaly-detection/stopping-ml.asciidoc

@@ -0,0 +1,88 @@
+[role="xpack"]
+[[stopping-ml]]
+=== Stopping {ml} {anomaly-detect}
+
+An orderly shutdown ensures that:
+
+* {dfeeds-cap} are stopped
+* Buffers are flushed
+* Model history is pruned
+* Final results are calculated
+* Model snapshots are saved
+* {anomaly-jobs-cap} are closed
+
+This process ensures that jobs are in a consistent state in case you want to
+subsequently re-open them.
+
+[float]
+[[stopping-ml-datafeeds]]
+=== Stopping {dfeeds}
+
+When you stop a {dfeed}, it ceases to retrieve data from {es}. You can stop a
+{dfeed} by using {kib} or the
+{ref}/ml-stop-datafeed.html[stop {dfeeds} API]. For example, the following
+request stops the `feed1` {dfeed}:
+
+[source,console]
+--------------------------------------------------
+POST _ml/datafeeds/feed1/_stop
+--------------------------------------------------
+// TEST[skip:setup:server_metrics_startdf]
+
+NOTE: You must have `manage_ml`, or `manage` cluster privileges to stop {dfeeds}.
+For more information, see <<security-privileges>>.
+
+A {dfeed} can be started and stopped multiple times throughout its lifecycle.
+
+//For examples of stopping {dfeeds} in {kib}, see <<ml-gs-job1-manage>>.
+
+[float]
+[[stopping-all-ml-datafeeds]]
+==== Stopping all {dfeeds}
+
+If you are upgrading your cluster, you can use the following request to stop all
+{dfeeds}:
+
+[source,console]
+----------------------------------
+POST _ml/datafeeds/_all/_stop
+----------------------------------
+// TEST[skip:needs-licence]
+
+[float]
+[[closing-ml-jobs]]
+=== Closing {anomaly-jobs}
+
+When you close an {anomaly-job}, it cannot receive data or perform analysis
+operations. If a job is associated with a {dfeed}, you must stop the {dfeed}
+before you can close the job. If the {dfeed} has an end date, the job closes
+automatically on that end date.
+
+You can close a job by using the
+{ref}/ml-close-job.html[close {anomaly-job} API]. For 
+example, the following request closes the `job1` job:
+
+[source,console]
+--------------------------------------------------
+POST _ml/anomaly_detectors/job1/_close
+--------------------------------------------------
+// TEST[skip:setup:server_metrics_openjob]
+
+NOTE: You must have `manage_ml`, or `manage` cluster privileges to stop {dfeeds}.
+For more information, see <<security-privileges>>.
+
+{anomaly-jobs-cap} can be opened and closed multiple times throughout their
+lifecycle.
+
+[float]
+[[closing-all-ml-datafeeds]]
+==== Closing all {anomaly-jobs}
+
+If you are upgrading your cluster, you can use the following request to close
+all open {anomaly-jobs} on the cluster:
+
+[source,console]
+----------------------------------
+POST _ml/anomaly_detectors/_all/_close
+----------------------------------
+// TEST[skip:needs-licence]

docs/en/stack/redirects.asciidoc

@@ -425,6 +425,17 @@ See {ref}/watcher-troubleshooting.html[Troubleshooting Watcher].
 This page has moved. 
 See {ref}/watcher-limitations.html[Watcher limitations].
 
+role="exclude",id="ml-dataframes"]
+=== Transforming data
+
+See {ref}/transforms.html[Transforming data].
+
+[role="exclude",id="ml-troubleshooting"]
+=== Troubleshooting {ml} {anomaly-detect}
+
+This page has moved.
+// See TBD
+
 [role="exclude",id="auditing"]
 === Auditing security events