Add mutual TLS docs

docs/en/ingest-management/index.asciidoc

@@ -91,6 +91,8 @@ include::security/generate-certificates.asciidoc[leveloffset=+1]
 
 include::security/certificates.asciidoc[leveloffset=+2]
 
+include::security/mutual-tls.asciidoc[leveloffset=+2]
+
 include::security/logstash-certificates.asciidoc[leveloffset=+2]
 
 include::fleet/fleet.asciidoc[leveloffset=+1]

docs/en/ingest-management/security/certificates.asciidoc

@@ -5,6 +5,8 @@ If you're running a self-managed cluster, configure Transport Layer Security
 (TLS) to encrypt traffic between {agent}s, {fleet-server}, and other components
 in the {stack}.
 
+For the install settings specific to mutual TLS, as opposed to one-way TLS, refer to <<mutual-tls>>.
+
 TIP: Our {ess-product}[hosted {ess}] on {ecloud} provides secure, encrypted
 connections out of the box!
 

docs/en/ingest-management/security/generate-certificates.asciidoc

@@ -8,4 +8,5 @@
 Some connections may require you to generate certificates and configure SSL/TLS.
 
 * <<secure-connections>>
+* <<mutual-tls>>
 * <<secure-logstash-connections>>

docs/en/ingest-management/security/mutual-tls.asciidoc

@@ -0,0 +1,200 @@
+[[mutual-tls]]
+= {agent} deployment models with mutual TLS
+
+Mutual Transport Layer Security (TLS) provides a higher level of security and trust compared to one-way TLS, where only the server presents a certificate. It ensures that not only the server is who it claims to be, but the client is also authenticated. This is particularly valuable in scenarios where both parties need to establish trust and validate each other's identities, such as in secure API communication, web services, or remote authentication.
+
+* <<mutual-tls-overview>>
+* <<mutual-tls-on-premise>>
+* <<mutual-tls-cloud>>
+* <<mutual-tls-cloud-proxy>>
+* <<mutual-tls-on-premise-hosted-es>>
+
+
+//[source,shell]
+//----
+//example
+//----
+
+//image::images/fleet-server-certs.png[Screen capture of a folder called fleet-server that contains two files: fleet-server.crt and fleet-server.key]
+
+[discrete]
+[[mutual-tls-overview]]
+== Overview
+
+With mutual TLS the following authentication and certification verification occurs:
+
+* **Client Authentication**: The client presents its digital certificate to the server during the TLS handshake. This certificate is issued by a trusted Certificate Authority (CA) and contains the client's public key.
+* **Server Authentication**: The server also presents its digital certificate to the client, proving its identity and sharing its public key. The server's certificate is also issued by a trusted CA.
+* **Certificate Verification**: Both the client and server verify each other's certificates by checking the digital signatures against the CA's public key. This process confirms the authenticity of the certificates and, by extension, the identities of the client and server.
+
+{fleet}-managed {agent} has two main connections to ensure correct operations:
+
+* Connectivity to {fleet-server} (the control plane, to check in, download policies, and similar).
+* Connectivity to an Output (the data plane, such as {es} or {ls}).
+
+In order to bootstrap, {agent} initially must establish a secure connection to the {fleet-server}, which can reside on-premises or in {ecloud}. This connectivity verification process ensures the agent's authenticity. Once verified, the agent receives the policy configuration. This policy download equips the agent with the knowledge of the other components it needs to engage with. For instance, it gains insights into the output destinations it should write data to.
+
+If mutual TLS (mTLS) is a requirement, {agent} must first establish an mTLS connection with {fleet-server}, with both client and server exchanging certificates and validating one another. Once the policy configuration is in place, it possesses the necessary details to establish an mTLS connection with the specific output it's configured to use. In the case of {fleet}-managed {agents}, certificates and certificate authorities essential for client-server authentication are configured through the {fleet} application in the {kib} user interface. As previously mentioned, the initial step involves establishing connectivity between {agent} and the {fleet-server}, allowing the subsequent configuration to take effect.
+
+To facilitate the bootstrapping process and enable {agent} to establish an mTLS connection with {fleet-server}, all certificates and certificate authorities are configured using command-line parameters during the agent installation. Once the mTLS connection between {agent} and the {fleet-server} is established, the policy configuration enables the establishment of the mTLS connection between {agent} and the designated output as well.
+
+[discrete]
+[[mutual-tls-on-premise]]
+== On-premise deployments
+
+image::images/mutual-tls-on-prem.png[Diagram of mutual TLS on premise deployment model]
+
+Refer to the steps in <<secure-connections,Configure SSL/TLS for self-managed {fleet-server}s>>. To configure mutual TLS, include the following additional parameters when you install {agent} and {fleet-server}.
+
+[discrete]
+=== {agent} settings
+During {agent} installation on premise use the following options:
+
+[cols="1,1"]
+|===
+|`-certificate-authorities`
+|List of CA certificates that are trusted when {fleet-server} connects to {agent} 
+
+|`-elastic-agent-cert`
+|{agent} certificate to present to {fleet-server} during authentication
+
+|`-elastic-agent-cert-key`
+|{agent} certificate key to present to {fleet-server}
+|===
+
+[discrete]
+=== {fleet-server} settings
+During {fleet-server} installation on-premise {fleet-server} authenticates with {es} and {agents}. You can use the following CLI options to facilitate these secure connections:
+
+[cols="1,1"]
+|===
+|`-fleet-server-es-ca`
+|CA to use for the {es} connection 
+
+|`-fleet-server-es-cert`
+|{fleet-server} certificate to present to {es}
+∫
+|`-fleet-server-es-cert-key`
+|{fleet-server} certificate key to present to {es}
+
+|`-certificate-authorities`
+|List of CA certificates that are trusted when {agent} connects to {fleet-server}
+
+|`-fleet-server-cert`
+|{fleet-server} certificate to present to {agents} during authentication
+
+|`-fleet-server-cert-key`
+|{fleet-server} certificate key to present to {agents}  
+|===
+
+[discrete]
+=== {fleet} settings:
+
+In {kib}, navigate to {fleet}, open the **Settings** tab, and choose the **Output** that you'd like to configure.
+In the **Advanced YAML configuration**, add the following settings:
+
+[cols="1,1"]
+|===
+|`-certificate-authorities`
+|List of CA certificates that are trusted when {fleet-server} connects to {agent}
+
+|`ssl.certificate`
+|This certificate will be passed down to all the agents that have this output configured in their policy. This certificate is used by the agent when establishing mTLS to the output. 
+
+You may either apply the full certificate, in which case all the agents get the same certificate OR alternatively point to a local directory on the agent where the certificate resides, if the certificates are to be unique per agent.
+
+|`ssl.key`
+|This certificate key will be passed down to all the agents that have this output configured in their policy. This key is used by the agent when establishing mTLS to the output. 
+
+Similarly the key may be stored in a local directory.
+|===
+
+image::images/mutual-tls-onprem-advanced-yaml.png[Screen capture of output advanced yaml settings]
+
+[discrete]
+[[mutual-tls-cloud]]
+== {fleet-server} on {ecloud}
+
+In this deployment model, all traffic ingress into {ecloud} has its TLS connection terminated at an Elastic-managed Cloud Proxy. Cloud Proxy is not provisioned on a per tenant basis so at this point a client specific certificate can NOT be used in this proxy.
+
+image::images/mutual-tls-cloud.png[Diagram of mutual TLS on cloud deployment model]
+
+We currently don't support mTLS in this deployment model. An alternate deployment model is shown below where you can deploy your own secure proxy where TLS connections are terminated.
+
+[discrete]
+[[mutual-tls-cloud-proxy]]
+== {fleet-server} on {ecloud} using a proxy
+
+In this scenario, unlike the one above, you have access to the proxy and therefore can configure it to do a TLS termination.
+
+image::images/mutual-tls-cloud-proxy.png[Diagram of mutual TLS on cloud deployment model with a proxy]
+
+[discrete]
+=== {agent} settings
+During {agent} installation on premise use the following options:
+
+[cols="1,1"]
+|===
+|`-certificate-authorities`
+|List of CA certificates that are trusted when {fleet-server} connects to {agent} 
+
+|`-elastic-agent-cert`
+|{agent} certificate to present to {fleet-server} during authentication
+
+|`-elastic-agent-cert-key`
+|{agent} certificate key to present to {fleet-server}
+|===
+
+[discrete]
+[[mutual-tls-on-premise-hosted-es]]
+== {fleet-server} on-premise and Hosted Elasticsearch Service
+
+In some scenarios you may want to deploy {fleet-server} on your own premises. In this case, you're able to provide your own certificates and certificate authority to facilitate mTLS between {fleet-server} and {agent}.
+
+However, as with the <<mutual-tls-cloud,{fleet-server} on {ecloud}>> use case, the data plane connections are terminated at Elastic's Cloud Proxy, which is not a multi-tenanted service and therefore can't provide per-user certificates.  
+
+image::images/mutual-tls-fs-onprem.png[Diagram of mutual TLS with Fleet Server on premise and hosted Elasticsearch Service deployment model]
+
+Similar to the {fleet-server} on {ecloud} use case, a secure proxy can be placed in such an environment to terminate the TLS connections and satisfy the mTLS requirements.
+
+image::images/mutual-tls-fs-onprem-proxy.png[Diagram of mutual TLS with Fleet Server on premise and hosted Elasticsearch Service deployment model with a proxy]
+
+[discrete]
+=== {agent} settings
+During {agent} installation on premise use the following options, similar to <<mutual-tls-on-premise,{agent} deployment on premises>>:
+
+[cols="1,1"]
+|===
+|`-certificate-authorities`
+|List of CA certificates that are trusted for when {fleet-server} connects to {agent} 
+
+|`-elastic-agent-cert`
+|{agent} certificate to present to {fleet-server} during authentication
+
+|`-elastic-agent-cert-key`
+|{agent} certificate key to present to {fleet-server}
+|===
+
+[discrete]
+=== {fleet-server} settings
+During {fleet-server} installation on-premise use the following options so that {fleet-server} can authenticate itself to the agent and then also to the secure proxy server:
+
+[cols="1,1"]
+|===
+|`-fleet-server-es-ca`
+|CA to use for the {es} connection, via secure proxy. This CA is used to authenticate the TLS connection from a secure proxy
+
+|`-certificate-authorities`
+|List of CA certificates that are trusted when {agent} connects to {fleet-server}
+
+|`-fleet-server-cert`
+|{fleet-server} certificate to present to {agents} during authentication
+
+|`-fleet-server-cert-key`
+|{fleet-server} certificate key to present to {agents}  
+|===
+
+[discrete]
+=== {fleet} settings
+
+This is the same as what's described for <<mutual-tls-on-premise,on premise deployments>>. The main difference is that you need to use certificates that are accepted by the secure proxy, as that is where the TLS connection will be terminated.