Added documentation for supporting cluster metrics monitors that can …

…monitor remote clusters. Added documentation for configuring query and bucket monitors through the UI that can query remote indexes. These are experimental for v2.12.

Signed-off-by: AWSHurneyt <hurneyt@amazon.com>

_observing-your-data/alerting/per-cluster-metrics-monitors.md

@@ -68,7 +68,8 @@ Painless scripts define triggers for cluster metrics monitors, similar to per qu
 
 The cluster metrics monitor supports up to **ten** triggers.
 
-In the following example, a JSON object creates a trigger that sends an alert when the cluster health is yellow. `script` points the `source` to the Painless script `ctx.results[0].status == \"yellow\`.
+In the following example, the monitor is configured to call the cluster health API for two clusters, (`cluster-1`, and `cluster-2`). The trigger condition will create an alert when either of the clusters' `status` is not `green`. 
+`script` points the `source` to the Painless script `for (cluster in ctx.results[0].keySet()) if (ctx.results[0][cluster].status != \"green\") return true`. See [Trigger variables]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/triggers/#trigger-variables) for more `painless ctx` variable options.
 
 ```json
 {
@@ -88,7 +89,8 @@ In the following example, a JSON object creates a trigger that sends an alert wh
         "api_type": "CLUSTER_HEALTH",
         "path": "_cluster/health/",
         "path_params": "",
-        "url": "http://localhost:9200/_cluster/health/"
+        "url": "http://localhost:9200/_cluster/health/",
+        "cluster": ["cluster-1", "cluster-2"]
       }
     }
   ],
@@ -100,7 +102,7 @@ In the following example, a JSON object creates a trigger that sends an alert wh
         "severity": "1",
         "condition": {
           "script": {
-            "source": "ctx.results[0].status == \"yellow\"",
+            "source": "for (cluster in ctx.results[0].keySet()) if (ctx.results[0][cluster].status != \"green\") return true",
             "lang": "painless"
           }
         },
@@ -110,14 +112,14 @@ In the following example, a JSON object creates a trigger that sends an alert wh
   ]
 }
 ```
+The dashboards interface supports selecting clusters to monitor, and the desired API.
 
-See [Trigger variables]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/triggers/#trigger-variables) for more `painless ctx` variable options.
+<img src="{{site.url}}{{site.baseurl}}/images/alerting/cross-cluster-cluster-metrics-monitors.png" alt="Cluster metrics monitor" width="700"/>
 
 ### Limitations
 
 Per cluster metrics monitors have the following limitations:
 
-- You cannot create monitors for remote clusters.
 - The OpenSearch cluster must be in a state where an index's conditions can be monitored and actions can be executed against the index.
 - Removing resource permissions from a user will not prevent that user’s preexisting monitors for that resource from executing.
 - Users with permissions to create monitors are not blocked from creating monitors for resources for which they do not have permissions; however, those monitors will not run.

_observing-your-data/alerting/per-query-bucket-monitors.md

@@ -13,6 +13,10 @@ Per query monitors are a type of alert monitor that can be used to identify and
 
 Per bucket monitors are a type of alert monitor that can be used to identify and alert on specific buckets of data that are created by a query against an OpenSearch index.
 
+Both monitor types support querying remote indexes using the same `cluster-name:index-name` pattern used by [cross-cluster search](https://opensearch.org/docs/latest/security/access-control/cross-cluster-search/), or using the new dashboards interface.
+
+<img src="{{site.url}}{{site.baseurl}}/images/alerting/cross-cluster-per-query-per-bucket-monitors.png" alt="Cluster metrics monitor" width="700"/>
+
 ## Creating a per query or per bucket monitor
 
 To create a per query monitor, follow these steps:

_observing-your-data/alerting/settings.md

@@ -54,6 +54,7 @@ Setting | Default | Description
 `plugins.alerting.alert_history_retention_period` | 60d | The amount of time to keep history indexes before automatically deleting them.
 `plugins.alerting.destination.allow_list` | ["chime", "slack", "custom_webhook", "email", "test_action"] | The list of allowed destinations. If you don't want to allow users to a certain type of destination, you can remove it from this list, but we recommend leaving this setting as-is.
 `plugins.alerting.filter_by_backend_roles` | "false" | Restricts access to monitors by backend role. See [Alerting security]({{site.url}}{{site.baseurl}}/monitoring-plugins/alerting/security/).
+`plugins.alerting.remote_monitoring_enabled` | "false" | Toggles whether cluster metrics monitors support executing against remote clusters.
 `plugins.scheduled_jobs.sweeper.period` | 5m | The alerting feature uses its "job sweeper" component to periodically check for new or updated jobs. This setting is the rate at which the sweeper checks to see if any jobs (monitors) have changed and need to be rescheduled.
 `plugins.scheduled_jobs.sweeper.page_size` | 100 | The page size for the sweeper. You shouldn't need to change this value.
 `plugins.scheduled_jobs.sweeper.backoff_millis` | 50ms | The amount of time the sweeper waits between retries---increases exponentially after each failed retry.