[DOCS] Update remote cluster docs

[lockewritesdocs]

This PR seeks to pull out CCR and CCS information that relates to using those features and integrate it as part of configuring remote clusters. This PR includes the following changes (and more):

• Consolidate the remote cluster docs into a single location
• Incorporate roles needed for CCR and CCS into the remote cluster docs
• Integrate and clarify the security docs for CCS
• Add security docs for CCR that mirror the updated CCS docs
• Simplify the CCR tutorial to make it more self-contained
• Redirect moved or removed pages

Preview links

Remote clusters

• Connect to remote clusters
• Configure remote cluster privileges
• Configure remote clusters with security

Closes #40724

Relates to #72841

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[lockewritesdocs]

@elasticmachine update branch

[ywangd]

Thanks for working on this topic, Adam. Some (many?) of the comments are probably on existing texts which you just moved around. I am OK if you prefer to skip them for this PR and have them addressed in follow-ups.

I think one major thing we'd like to achieve with this PR is to make the help docs read more logically and connected for the remote cluster and features enabled by it. Hence the order of introducing each concept is important. It seems to me that we should start with "brief intro of remote cluster, what are they and relevant concept", then "configuring security for cluster connection", including both TLS setup and enabling xpack.security.enabled. This is then followed by "configuring remote clusters with Cluster Settings update API etc", then "configuring individual privileges" for CCR and CCS.

It's also worthwhile to have a few sentences to briefly talk about the main security limitation of remote clusters, i.e. they are effectively one security domain and you cannot restrict user's access purely from the remote cluster. Similar questions have been asked and answered many times previously. So I think it's helpful to clear user confusions.

[ywangd]

I raised #61308 as a bug for CCR privileges a while back. Not sure if it is still valid today. I can check it later.

[lockewritesdocs]

@elasticmachine update branch

[lockewritesdocs]

@elasticmachine update branch

[ywangd]

Just had another round of look. Thanks for your patience.

[lockewritesdocs]

@elasticmachine update branch

[ywangd]

LGTM

I'd appreciate if you could also address the following comments before merging. There is still room for improvements. But I think they can be iterative with follow-ups. This is a step forward already. Thanks!

[ywangd]

It still does not read straightforward to me because it seems to suggest "alter elasticsearch.yml" is a reason for having "potential access to private keys". But these two things are separate. Maybe it's just me? I'd suggest the following change:

Suggested change

	clusters. Furthermore, a local user with administrative privileges at the
	operating system level can alter `elasticsearch.yml`. A malicious user with
	these privileges has potential access to private keys that could give them the
	ability to take over a remote cluster. To use {ccr} and {ccs} safely,
	clusters. Furthermore, a local administrator at the operating system level with sufficient access
	to Elasticsearch config files and private keys can potentially take over a remote cluster. To use {ccr} and {ccs} safely,

Also, I wonder whether we should take this sentence out from here because Enabing security at the Elasticsearch level does not help mitigating this threat. I think we can move it to the end of this paragraph after we finishing talking about ES level security first, something like:

Enabling and configuring security is important on both local and remote clusters. When connecting a local cluster to remote clusters, an Elasticsearch superuser (such as the elastic user) gains total read access to the remote clusters. To use cross-cluster replication and cross-cluster search safely, enable security on all connected clusters and configure Transport Layer Security (TLS) on at least the transport level on every node. Furthermore, a local administrator at the operating system level with sufficient access to Elasticsearch config files and private keys can potentially take over a remote cluster. Therefore you should treat the operating system level security for both clusters with equal importance.

I added the sentence in bold as a very generic advice for OS level security. Otherwise, it feels that we raise an issue without offering any recommendation. Please feel free to paraphrase.

[lockewritesdocs]

I split this information into two paragraphs:

Enabling and configuring security is important on both local and remote
clusters. When connecting a local cluster to remote clusters, an {es} superuser
(such as the elastic user) on the local cluster gains total read access to the
remote clusters. To use {ccr} and {ccs} safely,
<<remote-clusters-security,enable security>> on all connected clusters
and configure Transport Layer Security (TLS) on at least the transport level on
every node.

Furthermore, a local administrator at the operating system level
with sufficient access to {es} configuration files and private keys can
potentially take over a remote cluster. Ensure that your security strategy
includes securing local and remote clusters at the operating system level.

[ywangd]

This still reads funny to me. I think what we want to say is:

Suggested change

	to add a remote cluster. You can also use this API to
	<<configure-remote-clusters-dynamic,apply and update>> settings across the
	entire remote cluster. To configure settings on individual nodes in a remote
	cluster, define
	<<configure-remote-clusters-static,static settings>> in `elasticsearch.yml`.
	to add a remote cluster. You use this API to dynamically
	<<configure-remote-clusters-dynamic,configure remote clusters>> for every node
	in the local cluster. To configure remote clusters for an individual node of the local cluster,
	define <<configure-remote-clusters-static,static settings>> in the node's `elasticsearch.yml`.

[lockewritesdocs]

I'll change this to read:

Alternatively, use the <<cluster-update-settings,cluster update settings API>>
to add a remote cluster. You use this API to dynamically
<<configure-remote-clusters-dynamic,configure remote clusters>> for every node
in the local cluster. To configure remote clusters for individual nodes in the
local cluster, define <<configure-remote-clusters-static,static settings>> in elasticsearch.yml for each node.

[lockewritesdocs]

@elasticmachine update branch

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts
❌	7.15	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 77043