Integrate custom CA for http interface

[MHenn1g]

I have a request regarding the certificate management for the http certificate management.

The basic construct is as follows.

• ECK version: 1.5.0
• Multiple Elasticsearch Deployments, each deployed automatically via Helm through a deployment tool like Argo CD
• Apps need to communicate to the Elasticsearch Deployments via the LoadBalancer IP
• A central enterprise CA is integrated in the Apps trust store.
  Customers should not need to generate their own certificates.
  Instead, this CA should be utilized during the creation of the Cluster deployments to automatically enable a trusted communication.

As I take the documentation there are currently multiple ways to utilize certificates for the http interface in order to communicate with an Elasticsearch installation based on ECK:

1. Utilize a default self signed certificate (https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-tls-certificates.html#k8s-default-self-signed-certificate)
2. Setup your own certificate (https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-tls-certificates.html#k8s-setting-up-your-own-certificate)
3. Use a third party tool like the cert-manager (https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-custom-http-certificate.html#k8s_custom_self_signed_certificate_using_cert_manager)

Since developers should not generate their own certificates, option 2 is out of the question for our use case.

When utilizing version 1, using a self signed certificate, the operator generates a certificate including a whole bunch of SANs out of the box:

Subject Alternative Names:  
	quickstart-eck-es-http.ns.es.local,
	quickstart-eck-es-http,  
	quickstart-eck-es-http.ns.svc,
	quickstart-eck-es-http.ns, quickstart-eck-es-http,
	*.quickstart-eck-es-control-plane-az1.ns.svc,
	*.quickstart-eck-es-control-plane-az2.ns.svc,
	*.quickstart-eck-es-control-plane-az3.ns.svc,
	*.quickstart-eck-es-data-plane-az1.ns.svc,
	*.quickstart-eck-es-data-plane-az2.ns.svc,
	*.quickstart-eck-es-data-plane-az3.ns.svc,
	IP Address:192.168.0.121

Especially the IP Adress of the LoadBalancer gets injected directly, which is pretty handy, since this may be utilized to directly communicate through the service.

However, this only works as long as self-signed certificates are utilized.
Currently, if we want to use a centralized CA certificate in order to generate http certificates, we need to use third party tools like the cert-manager (https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-custom-http-certificate.html#k8s_custom_self_signed_certificate_using_cert_manager) to generate the http certificate, e.g. via:

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: quickstart-es-cert
spec:
  isCA: true
  dnsNames:
  - quickstart-es-http
  - quickstart-es-http.default.svc
  - quickstart-es-http.default.svc.cluster.local
  issuerRef:
  kind: Issuer
  name: cental-ca-issuer
  secretName: quickstart-es-cert

With this, there are no default SANs integrated in the certificate.
Each SAN needs to be generated programatically, which is okay for some cases, however one integration is missing, which is the IP of the LoadBalancer.
Since deployments are generated automatically, LoadBalancer IPs are generated automatically as well.

If we want to integrate the LoadBalancer IP, it seems we need even more additional tooling like Argo Events for example.
This would enable a workflow like -> check for the creation of a new LoadBalancer -> check out its IP -> generate the Certificate -> deploy the Elasticsearch Cluster.

Correct me if I'm wrong, I couldn't find an elegant solution and therefore am stuck with this fragile process.

Long story short :)
For the transport interface, there is a way to link a certificate authority (https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-transport-settings.html#k8s_configure_a_custom_certificate_authority).
Based on this CA, the certificates for the internal communication are generated.

Would it be possible to integrate a similar mechanism for the http interface, e.g. specifying a custom-ca.

spec:
  http:
    tls:
      certificate:
        secretName: custom-ca

And then generating the currently self-signed http certificate based on this custom-ca, e.g. with SANs like this out of the box:

Subject Alternative Names:  
	quickstart-eck-es-http.ns.es.local,
	quickstart-eck-es-http,  
	quickstart-eck-es-http.ns.svc,
	quickstart-eck-es-http.ns, quickstart-eck-es-http,
	*.quickstart-eck-es-control-plane-az1.ns.svc,
	*.quickstart-eck-es-control-plane-az2.ns.svc,
	*.quickstart-eck-es-control-plane-az3.ns.svc,
	*.quickstart-eck-es-data-plane-az1.ns.svc,
	*.quickstart-eck-es-data-plane-az2.ns.svc,
	*.quickstart-eck-es-data-plane-az3.ns.svc,
	IP Address:192.168.0.121

and the option to define own SANs as well:

spec:
  http:
    service:
      spec:
        type: LoadBalancer
    tls:
      selfSignedCertificate:
        subjectAltNames:
        - ip: 160.46.176.15
        - dns: hulk.example.com

This way no additional tooling would be needed for the certificate management, allowing us to build self-contained helm charts and to implement a more robust system.

[sebgl]

A few thoughts:

• We inject (and maintain up to date) the LoadBalancer IP in the generated self-signed certificates for convenience, but this is in general a bit brittle. It could make more sense to rely on DNS names instead, so the certificate matches the DNS name rather than the IP that may change at any time. Of course it means you need to handle (and likely automate) the DNS <> LoadBalancer IP relationship.
• Rather than using a LoadBalancer per cluster, you could setup an extra Ingress resource per cluster, and bind the corresponding Ingress proxy to a single public LoadBalancer IP. The flow would work as follow:

LoadBalancer IP <---ingress https cert---> Ingress proxy <---internal self-signed https cert---> Elasticsearch service

This way you would pre-provision a single LoadBalancer service, and a single public-facing HTTPS certificate for all clusters you want to manage. And keep using "internal" HTTPS certs internally only. See this gclb and traefik examples.

• Allowing users to provide their own CA for the HTTP layer would make sense, I think. Especially since we already provide that option for the transport layer. Note this means you need to provide public and private key of the CA though. I can imagine that for a centralized organization CA this may cause security issues. Would that be OK in your case?
• We don't support it officially, but currently you could provide your own CA in advance in the <elasticsearch-name>-es-http-ca-internal secret. The operator will notice this secret already exists and, as long as the certificate is valid and not expired, will use it to generate HTTPS certs. Since this is not an exposed feature, there is no compatibility guarantee with future versions of ECK (we don't plan to change this, but we could).

More generally, I think I'd be +1 with adding a feature to allow users to provide their own CA certs to use for the HTTP layer, for consistency with the transport layer.

[MHenn1g]

Thank you for your reply.

• Generally I would expect the LoadBalancer IP to stay constant for its lifetime, but of course you are right regarding the utilization of DNS. This would be a preferred scenario and I have had discussions with the team managing the k8s clusters and DNS, but they have a few concerns and on the other hand internal politics are blocking this path at the moment :/
• The CA (public and private) key would provided secluded from the deployment itself, so users would not have any access to them, so this would be okay.
• The hidden feature of providing a custom -es-http-ca-internal sounds helpful and I will test it out
  Consistency across the transport and http layer would be helpful, thank you very much :)

[pebrc]

We currently have the following layout in the secret for the HTTP layer of Elasticsearch as documented here:

ca.crt: optional CA certificate
tls.key: private key for the HTTP layer certificate
tls.crt: HTTP layer certificate

We could extend that as follows:

ca.key: private key for the CA certificate
ca.crt: CA certificate
tls.key: private key for the HTTP layer certificate
tls.crt: HTTP layer certificate

The following combinations would be valid:

New mode to generate the HTTP layer certificate from a CA cert/with key

ca.key: ...
ca.crt: ...

as before: HTTP layer certificate from a well known CA:

tls.key: ...
tls.crt: ...

as before: HTTP layer certificate from an internal CA

ca.crt: ...
tls.key: ...
tls.crt: ...

The only inconsistency I see is that for the transport layer we went with tls.key and tls.crt as well but they are meant to be filled with the CA certificates (!). I see a potential for user confusion there.