Add mutual TLS docs (#1132)

* Add mutual TLS docs

* Address comments

* Add fixes for latest comments

(cherry picked from commit d5358d2)

docs/en/ingest-management/index.asciidoc

@@ -93,6 +93,8 @@ include::security/generate-certificates.asciidoc[leveloffset=+1]
 
 include::security/certificates.asciidoc[leveloffset=+2]
 
+include::security/mutual-tls.asciidoc[leveloffset=+2]
+
 include::security/logstash-certificates.asciidoc[leveloffset=+2]
 
 include::fleet/fleet.asciidoc[leveloffset=+1]

docs/en/ingest-management/security/certificates.asciidoc

@@ -5,6 +5,8 @@ If you're running a self-managed cluster, configure Transport Layer Security
 (TLS) to encrypt traffic between {agent}s, {fleet-server}, and other components
 in the {stack}.
 
+For the install settings specific to mutual TLS, as opposed to one-way TLS, refer to <<mutual-tls>>.
+
 TIP: Our {ess-product}[hosted {ess}] on {ecloud} provides secure, encrypted
 connections out of the box!
 

docs/en/ingest-management/security/generate-certificates.asciidoc

@@ -8,4 +8,5 @@
 Some connections may require you to generate certificates and configure SSL/TLS.
 
 * <<secure-connections>>
+* <<mutual-tls>>
 * <<secure-logstash-connections>>

docs/en/ingest-management/security/mutual-tls.asciidoc

@@ -0,0 +1,236 @@
+[[mutual-tls]]
+= {agent} deployment models with mutual TLS
+
+Mutual Transport Layer Security (mTLS) provides a higher level of security and trust compared to one-way TLS, where only the server presents a certificate. It ensures that not only the server is who it claims to be, but the client is also authenticated. This is particularly valuable in scenarios where both parties need to establish trust and validate each other's identities, such as in secure API communication, web services, or remote authentication.
+
+* <<mutual-tls-overview>>
+* <<mutual-tls-on-premise>>
+* <<mutual-tls-cloud>>
+* <<mutual-tls-cloud-proxy>>
+* <<mutual-tls-on-premise-hosted-es>>
+
+
+//[source,shell]
+//----
+//example
+//----
+
+//image::images/fleet-server-certs.png[Screen capture of a folder called fleet-server that contains two files: fleet-server.crt and fleet-server.key]
+
+[discrete]
+[[mutual-tls-overview]]
+== Overview
+
+With mutual TLS the following authentication and certification verification occurs:
+
+* **Client Authentication**: The client presents its digital certificate to the server during the TLS handshake. This certificate is issued by a trusted Certificate Authority (CA) and contains the client's public key.
+* **Server Authentication**: The server also presents its digital certificate to the client, proving its identity and sharing its public key. The server's certificate is also issued by a trusted CA.
+* **Certificate Verification**: Both the client and server verify each other's certificates by checking the digital signatures against the CAs' public key (note that the client and server need not use the same CA).
+
+{fleet}-managed {agent} has two main connections to ensure correct operations:
+
+* Connectivity to {fleet-server} (the control plane, to check in, download policies, and similar).
+* Connectivity to an Output (the data plane, such as {es} or {ls}).
+
+In order to bootstrap, {agent} initially must establish a secure connection to the {fleet-server}, which can reside on-premises or in {ecloud}. This connectivity verification process ensures the agent's authenticity. Once verified, the agent receives the policy configuration. This policy download equips the agent with the knowledge of the other components it needs to engage with. For instance, it gains insights into the output destinations it should write data to.
+
+//If mutual TLS (mTLS) is a requirement, {agent} must first establish an mTLS connection with {fleet-server}, with both client and server exchanging certificates and validating one another. Once the policy configuration is in place, it possesses the necessary details to establish an mTLS connection with the specific output it's configured to use. In the case of {fleet}-managed {agents}, certificates and certificate authorities essential for client-server authentication are configured through the {fleet} application in the {kib} user interface. As previously mentioned, the initial step involves establishing connectivity between {agent} and the {fleet-server}, allowing the subsequent configuration to take effect.
+
+//To facilitate the bootstrapping process and enable {agent} to establish an mTLS connection with {fleet-server}, all certificates and certificate authorities are configured using command-line parameters during the agent installation. Once the mTLS connection between {agent} and the {fleet-server} is established, the policy configuration enables the establishment of the mTLS connection between {agent} and the designated output as well.
+
+When mTLS is required, the secure setup between {agent}, {fleet}, and {fleet-server} is configured through the following steps:
+
+. mTLS is enabled.
+. The initial mTLS connection between {agent} and {fleet-server} is configured when {agent} is enrolled, using the parameters passed through the `elastic-agent install` or `elastic-agent enroll` command.
+. Once enrollment has completed, {agent} downloads the initial {agent} policy from {fleet-server}.
+.. If the {agent} policy contains mTLS configuration settings, those settings will take precedence over those used during enrollment: This includes both the mTLS settings used for connectivity between {agent} and {fleet-server} (and the {fleet} application in {kib}, for {fleet}-managed {agent}), and the settings used between {agent} and it's specified output.
+.. If the {agent} policy does not contain any TLS, mTLS, or proxy configuration settings, these settings will remain as they were specified when {agent} enrolled. Note that the initial TLS, mTLS, or proxy configuration settings can not be removed through the {agent} policy; they can only be updated.
+
+[discrete]
+[[mutual-tls-on-premise]]
+== On-premise deployments
+
+image::images/mutual-tls-on-prem.png[Diagram of mutual TLS on premise deployment model]
+
+Refer to the steps in <<secure-connections,Configure SSL/TLS for self-managed {fleet-server}s>>. To configure mutual TLS, include the following additional parameters when you install {agent} and {fleet-server}.
+
+[discrete]
+=== {agent} settings
+During {agent} installation on premise use the following options:
+
+[cols="1,1"]
+|===
+|`--certificate-authorities`
+|List of CA certificates that are trusted when {fleet-server} connects to {agent} 
+
+|`--elastic-agent-cert`
+|{agent} certificate to present to {fleet-server} during authentication
+
+|`--elastic-agent-cert-key`
+|{agent} certificate key to present to {fleet-server}
+|===
+
+[discrete]
+=== {fleet-server} settings
+During {fleet-server} installation on-premise {fleet-server} authenticates with {es} and {agents}. You can use the following CLI options to facilitate these secure connections:
+
+[cols="1,1"]
+|===
+|`--fleet-server-es-ca`
+|CA to use for the {es} connection 
+
+|`--fleet-server-es-cert`
+|{fleet-server} certificate to present to {es}
+
+|`--fleet-server-es-cert-key`
+|{fleet-server} certificate key to present to {es}
+
+|`--certificate-authorities`
+|List of CA certificates that are trusted when {agent} connects to {fleet-server} and when {fleet-server} validates the {agent} identity.
+
+|`--fleet-server-cert`
+|{fleet-server} certificate to present to {agents} during authentication
+
+|`--fleet-server-cert-key`
+|{fleet-server}'s private certificate key used to decrypt the certificate
+|===
+
+[discrete]
+=== {fleet} settings:
+
+In {kib}, navigate to {fleet}, open the **Settings** tab, and choose the **Output** that you'd like to configure.
+In the **Advanced YAML configuration**, add the following settings:
+
+[cols="1,1"]
+|===
+|`ssl.certificate_authorities`
+|List of CA certificates that are trusted when {fleet-server} connects to {agent}
+
+|`ssl.certificate`
+|This certificate will be passed down to all the agents that have this output configured in their policy. This certificate is used by the agent when establishing mTLS to the output. 
+
+You may either apply the full certificate, in which case all the agents get the same certificate OR alternatively point to a local directory on the agent where the certificate resides, if the certificates are to be unique per agent.
+
+|`ssl.key`
+|This certificate key will be passed down to all the agents that have this output configured in their policy. The certificate key is used to decrypt the SSL certificate.
+
+|===
+
+[IMPORTANT] 
+====
+Note the following when you specify these SSL settings:
+
+* The certificate authority, certificate, and certificate key need to be specified as a path to a local file. You cannot specify a directory.
+* You can define multiple CAs or paths to CAs.
+* Only one certificate and certificate key can be defined.
+====
+
+In the *Advanced YAML configuration* these settings should be added in the following format:
+
+[source,shell]
+----
+ssl.certificate_authorities:
+  - /path/to/ca
+ssl.certificate: /path/to/cert
+ssl.key: /path/to/cert_key
+----
+
+OR
+
+[source,shell]
+----
+ssl.certificate_authorities:
+  - /path/to/ca
+ssl.certificate: /path/to/cert
+ssl.key: /path/to/cert_key
+----
+
+image::images/mutual-tls-onprem-advanced-yaml.png[Screen capture of output advanced yaml settings]
+
+[discrete]
+[[mutual-tls-cloud]]
+== {fleet-server} on {ecloud}
+
+In this deployment model, all traffic ingress into {ecloud} has its TLS connection terminated at the {ecloud} boundary. Since this termination is not handled on a per-tenant basis, a client-specific certificate can NOT be used at this point.
+
+image::images/mutual-tls-cloud.png[Diagram of mutual TLS on cloud deployment model]
+
+We currently don't support mTLS in this deployment model. An alternate deployment model is shown below where you can deploy your own secure proxy where TLS connections are terminated.
+
+[discrete]
+[[mutual-tls-cloud-proxy]]
+== {fleet-server} on {ecloud} using a proxy
+
+In this scenario, where you have access to the proxy, you can configure mTLS between the agent and your proxy.
+
+image::images/mutual-tls-cloud-proxy.png[Diagram of mutual TLS on cloud deployment model with a proxy]
+
+[discrete]
+=== {agent} settings
+During {agent} installation on premise use the following options:
+
+[cols="1,1"]
+|===
+|`--certificate-authorities`
+|List of CA certificates that are trusted when {agent} connects to {fleet-server} or to the proxy between {agent} and {fleet-server}
+
+|`--elastic-agent-cert`
+|{agent} certificate to present  during authentication to {fleet-server} or to the proxy between {agent} and {fleet-server}
+
+|`--elastic-agent-cert-key`
+|{agent}'s private certificate key used to decrypt the certificate
+|===
+
+[discrete]
+[[mutual-tls-on-premise-hosted-es]]
+== {fleet-server} on-premise and Hosted Elasticsearch Service
+
+In some scenarios you may want to deploy {fleet-server} on your own premises. In this case, you're able to provide your own certificates and certificate authority to enable mTLS between {fleet-server} and {agent}.
+
+However, as with the <<mutual-tls-cloud,{fleet-server} on {ecloud}>> use case, the data plane TLS connections terminate at the {ecloud} boundary. {ecloud} is not a multi-tenanted service and therefore can't provide per-user certificates.
+
+image::images/mutual-tls-fs-onprem.png[Diagram of mutual TLS with Fleet Server on premise and hosted Elasticsearch Service deployment model]
+
+Similar to the {fleet-server} on {ecloud} use case, a secure proxy can be placed in such an environment to terminate the TLS connections and satisfy the mTLS requirements.
+
+image::images/mutual-tls-fs-onprem-proxy.png[Diagram of mutual TLS with Fleet Server on premise and hosted Elasticsearch Service deployment model with a proxy]
+
+[discrete]
+=== {agent} settings
+During {agent} installation on premise use the following options, similar to <<mutual-tls-on-premise,{agent} deployment on premises>>:
+
+[cols="1,1"]
+|===
+|`--certificate-authorities`
+|List of CA certificates that are trusted for when {agent} connects to {fleet-server} 
+
+|`--elastic-agent-cert`
+|{agent} certificate to present to {fleet-server} during authentication
+
+|`--elastic-agent-cert-key`
+|{agent}'s private certificate key used to decrypt the certificate
+|===
+
+[discrete]
+=== {fleet-server} settings
+During {fleet-server} installation on-premise use the following options so that {fleet-server} can authenticate itself to the agent and then also to the secure proxy server:
+
+[cols="1,1"]
+|===
+|`--fleet-server-es-ca`
+|CA to use for the {es} connection, via secure proxy. This CA is used to authenticate the TLS connection from a secure proxy
+
+|`--certificate-authorities`
+|List of CA certificates that are trusted when {agent} connects to {fleet-server}
+
+|`--fleet-server-cert`
+|{fleet-server} certificate to present to {agents} during authentication
+
+|`--fleet-server-cert-key`
+|{fleet-server}'s private certificate key used to decrypt the certificate
+|===
+
+[discrete]
+=== {fleet} settings
+
+This is the same as what's described for <<mutual-tls-on-premise,on premise deployments>>. The main difference is that you need to use certificates that are accepted by the secure proxy, as the mTLS is set up between the agent and the secure proxy.