Customizable transport layer certificates

[MHenn1g]

I have a request regarding the certificate management for the transport layer certificates.

The basic construct is as follows.

• ECK version: 1.3.0
• A legacy app needs to communicate with the elasticsearch cluster on port 9300
• In order to validate the node certificates, I need to configure the app with the ca (quickstart-es-transport-ca-internal)

As long as I communicate directly with the Pods through the Pod-IPs everything works fine.
However, since these IPs are dynamic by nature I would like to implement the communication based on a LoadBalancer.

As described in https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-transport-settings.html, I deployed the LoadBalancer for the transport interface.

spec:
  transport:
    service:
      spec:
        type: LoadBalancer

Which works...

kubectl get svc
NAME                             TYPE           CLUSTER-IP        EXTERNAL-IP      PORT(S)          AGE
quickstart-es-transport          LoadBalancer   192.168.200.212   172.16.189.102   9300:30343/TCP   43h

...and registers the endpoints as expected.

kubectl describe svc quickstart-es-transport
Endpoints:                10.233.35.66:9300,10.233.35.68:9300,10.233.35.71:9300

However, I am not able to communicate with the cluster through the application since the communication is blocked with the following message.

[quickstart-es-coordinating-plane-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/172.16.181.159:47597}

As expected, the certificate returned by the LoadBalancer is one of the coordinating nodes.

openssl s_client -host 172.16.189.102 -port 9300

Subject Alternative Names: 
	othername:<unsupported>, 
	quickstart-es-coordinating-plane-2.node.quickstart.poc.es.local, 
	quickstart-es-transport.poc.svc, 
	quickstart-es-coordinating-plane-2.quickstart-es-coordinating-plane, 
	IP Address:10.233.35.71, 
	IP Address:127.0.0.1

It seems the error arrises, because the certificates are validated in full.
Therefore, since the application connects to the LoadBalancer on 172.16.189.102 and the certificate returns only the node IP 10.233.35.71, the communication is not trusted.

For the http interface there is the option to provide custom SANs for the certificates
https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-http-settings-tls-sans.html#k8s-elasticsearch-http-service-san

However, this does not seem to be possible for the transport interface.

Would it be possible to implement a the construct of customizing cerfiticates on the transport layer like this, to incorporate the LoadBalancer IP?

spec:
  transport:
    tls:
      selfSignedCertificate:
        subjectAltNames:
          - ip: 172.16.189.102
          - dns: custom.example.com

[pebrc]

Apologies for the very late reply. A few questions to better understand the problem:

As long as I communicate directly with the Pods through the Pod-IPs everything works fine.

I assume this means your legacy transport client application is running inside the Kubernetes cluster?

However, since these IPs are dynamic by nature I would like to implement the communication based on a LoadBalancer.

Would a Kubernetes service (ClusterIP without LoadBalancer) in that case not do the trick?

Having said that, your proposal to use a similar approach as on the HTTP layer seems reasonable. I am just trying to get a better understanding of the use cases for the feature.

[MHenn1g]

Hi, thanks for your feedback.

I assume this means your legacy transport client application is running inside the Kubernetes cluster?

No, not at all...I just phrased this suboptimal.
The software runs external to the cluster and should connect through a service, namely a LoadBalancer.
This was just an example to verify that the problems arise when connecting through the Service.

Would a Kubernetes service (ClusterIP without LoadBalancer) in that case not do the trick?

Since this is the case, sadly a ClusterIP would not suffice.

Short story: The software connects from externally through a LoadBalancer (or through an Passthrough-Ingress) to the transport interface. Either way, the POD answers with a certificate containing only the POD IPs and credentials for the cluster internal communication in the SANs. The switch to adapt the Transport Layer SAN entries is missing. Which is why the certificate chain cannot be verified and connections are rejected :)