Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hostname_verification not using SAN DNS entry #2054

Open
kkensy opened this issue Aug 29, 2022 · 14 comments
Open

hostname_verification not using SAN DNS entry #2054

kkensy opened this issue Aug 29, 2022 · 14 comments
Labels
bug Something isn't working help wanted Community contributions are especially encouraged for these issues. triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.

Comments

@kkensy
Copy link

kkensy commented Aug 29, 2022

What is the bug?
After successfully replacing the demo certificates with my own, I am having a problem enabling the plugins.security.ssl.transport.enforce_hostname_verification. Im following the documentation:
https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/
https://opensearch.org/docs/latest/opensearch/install/docker-security/

The X509v3 Subject Alternative Name entry and the plugins.security.nodes_dn property are set accordingly:

Subject: CN=opensearch-node1,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA
...
X509v3 Subject Alternative Name: 
   DNS:opensearch-node1
plugins.security.nodes_dn:
  - 'CN=opensearch-node1,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
  - 'CN=opensearch-node2,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'

After setting plugins.security.ssl.transport.enforce_hostname_verification: true the following errors occur:

...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
...
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 172.20.0.3 found
...
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
...

I have to insist on the DNS SAN record and cannot add an IP address SAN record to my certificates.

How can one reproduce the bug?
I have prepared an example with docker. Steps to reproduce the behavior:

  1. Clone https://github.com/kkensy/opensearch_hostname_verification
  2. docker compose up
  3. See errors in logs

What is the expected behavior?
plugins.security.ssl.transport.enforce_hostname_verification enabled, cluster up with all nodes, no related errors in logs

@kkensy kkensy added bug Something isn't working untriaged Require the attention of the repository maintainers and may need to be prioritized labels Aug 29, 2022
@kkensy kkensy changed the title hostname_verification using only SAN IP address hostname_verification not using SAN DNS entry Aug 29, 2022
@cwperks
Copy link
Member

cwperks commented Aug 29, 2022

@kkensy Thank you providing an example configuration. I noticed that the configuration works when all instances of network.host: 0.0.0.0 are removed from docker-compose.yml and custom-opensearch.yml. It looks like this issue arrises when binding to all interfaces.

@kkensy
Copy link
Author

kkensy commented Aug 29, 2022

@kkensy Thank you providing an example configuration. I noticed that the configuration works when all instances of network.host: 0.0.0.0 are removed from docker-compose.yml and custom-opensearch.yml. It looks like this issue arrises when binding to all interfaces.

If I remove all the network.host entries, it will bound to localhost and the cluster will not be formed. And with network.host:_site_ the same errors are occuring.

@peternied peternied added help wanted Community contributions are especially encouraged for these issues. and removed untriaged Require the attention of the repository maintainers and may need to be prioritized labels Aug 29, 2022
@peternied
Copy link
Member

@kkensy Thanks for filing this issue, this looks like a defect in our support for hostname verification.

@davidlago davidlago added the triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. label Oct 10, 2022
@kkensy
Copy link
Author

kkensy commented Nov 24, 2022

The problem persists in version 2.4

@andrey-gava
Copy link

Hi!
I have similar problem. I try to install 3 node cluster inside Kubernetes using modified official helm chart.
this is the part of security plugin config:

    plugins:
      security:
        ssl:
          transport:
            enabled: true
            pemkey_filepath: pkcs8-key/tls-pkcs8.key
            pemtrustedcas_filepath: certs/ca.crt
            enforce_hostname_verification: true
            resolve_hostname: true
          http:
            enabled: true
            pemkey_filepath: pkcs8-key/tls-pkcs8.key
            pemtrustedcas_filepath: certs/ca.crt
        allow_unsafe_democertificates: false
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=ADMIN
        nodes_dn:
          - 'CN=opensearch-cluster-master-0'
          - 'CN=opensearch-cluster-master-1'
          - 'CN=opensearch-cluster-master-2'  

The secret definition

{{- $ca := buildCustomCert (.Values.ca.Cert | b64enc)  (.Values.ca.Key | b64enc) -}}
{{- $cert0 := genSignedCert "opensearch-cluster-master-0" nil ( list "opensearch-cluster-master-0.opensearch-cluster-master-headless" ( printf "%s.%s" "opensearch-cluster-master-headless" .Values.shared_namespace ) ( printf "%s.%s.svc" "opensearch-cluster-master-headless" .Values.shared_namespace ) ) 730 $ca -}}
{{- $cert1 := genSignedCert "opensearch-cluster-master-1" nil ( list "opensearch-cluster-master-1.opensearch-cluster-master-headless" ( printf "%s.%s" "opensearch-cluster-master-headless" .Values.shared_namespace ) ( printf "%s.%s.svc" "opensearch-cluster-master-headless" .Values.shared_namespace ) ) 730 $ca -}}
{{- $cert2 := genSignedCert "opensearch-cluster-master-2" nil ( list "opensearch-cluster-master-2.opensearch-cluster-master-headless" ( printf "%s.%s" "opensearch-cluster-master-headless" .Values.shared_namespace ) ( printf "%s.%s.svc" "opensearch-cluster-master-headless" .Values.shared_namespace ) ) 730 $ca -}}
{{- $admin_cert := genSignedCert "ADMIN" nil nil 730 $ca -}}

apiVersion: v1
kind: Secret
metadata:
  name: oss-tls-secret
  labels:
    heritage: {{ .Release.Service | quote }}
    release: {{ .Release.Name | quote }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
type: Opaque
data:
  ca.crt: {{ $ca.Cert | b64enc | quote }}
  opensearch-cluster-master-0.key: {{ $cert0.Key | b64enc | quote }}
  opensearch-cluster-master-0.crt: {{ $cert0.Cert | b64enc | quote }}
  opensearch-cluster-master-1.key: {{ $cert1.Key | b64enc | quote }}
  opensearch-cluster-master-1.crt: {{ $cert1.Cert | b64enc | quote }}
  opensearch-cluster-master-2.key: {{ $cert2.Key | b64enc | quote }}
  opensearch-cluster-master-2.crt: {{ $cert2.Cert | b64enc | quote }}
  admin.key: {{ $admin_cert.Key | b64enc | quote }}
  admin.crt: {{ $admin_cert.Cert | b64enc | quote }}

And part of statefullset.
This one convert key to pkcs8

extraInitContainers: |
  - name: generate-pkcs8key
    image: {{ .Values.image_registry }}/openjdk-11:1.14-r2
    imagePullPolicy: IfNotPresent
    env:
    - name: NODENAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    command: ["/bin/sh"]
    args:
    - "-c"
    - |
        openssl pkcs8 -in /certs/$(NODENAME).key -topk8 -out /pkcs8-key/tls-pkcs8.key -nocrypt
    volumeMounts:
    - name: certs
      mountPath: /certs
    - name: pkcs8-key
      mountPath: /pkcs8-key

Here is a envs to mount each cert/key pair to specific pod and announce all discovery node hostnames:

        env:
        - name: NODENAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: SERVICENAME
          value: opensearch-cluster-master-headless
        - name: node.name
          value: $(NODENAME).$(SERVICENAME)
        {{- if eq .Values.roles.master "true" }}
        - name: cluster.initial_master_nodes
          value: "opensearch-cluster-master-0.$(SERVICENAME),opensearch-cluster-master-1.$(SERVICENAME),opensearch-cluster-master-2.$(SERVICENAME)"
        {{- end }}
        - name: discovery.seed_hosts
          value: "opensearch-cluster-master-0.$(SERVICENAME),opensearch-cluster-master-1.$(SERVICENAME),opensearch-cluster-master-2.$(SERVICENAME)"
        - name: cluster.name
          value: "{{ .Values.clusterName }}"
        - name: network.host
          value: "{{ .Values.networkHost }}"
        - name: OPENSEARCH_JAVA_OPTS
          value: "{{ .Values.opensearchJavaOpts }}"
        {{- range $role, $enabled := .Values.roles }}
        - name: node.{{ $role }}
          value: "{{ $enabled }}"
        {{- end }}
        - name: plugins.security.ssl.http.pemcert_filepath
          value: certs/$(NODENAME).crt
        - name: plugins.security.ssl.transport.pemcert_filepath
          value: certs/$(NODENAME).crt

As you see I set each node hostname specifically by providing its name like this:
"$pod_name.$headless_service_name"
which is resolved by DNS within each pod without problem.
But when it comes to create a cluster from logs I see that nodes try to verify IP instead of hostname.

[2023-01-11T14:05:23,575][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0.opensearch-cluster-master-headless] [connectToRemoteMasterNode[10.130.82.246:9300]] completed handshake with [{opensearch-cluster-master-1.opensearch-cluster-master-headless}{b1uf3VCbSzuTffnBwX47iA}{TB_8_wruRWyDe5I_v6ac0w}{10.130.82.246}{10.130.82.246:9300}{dimr}{shard_indexing_pressure_enabled=true}] but followup connection failed
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-1.opensearch-cluster-master-headless][10.130.82.246:9300] general node connection failure
	at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:1052) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:86) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.transport.TransportHandshaker$HandshakeResponseHandler.handleLocalException(TransportHandshaker.java:199) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:77) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.action.ActionListener.lambda$wrap$0(ActionListener.java:147) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:211) ~[opensearch-1.2.3.jar:1.2.3]
	at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:52) ~[opensearch-core-1.2.3.jar:1.2.3]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137) ~[?:?]
	at org.opensearch.common.concurrent.CompletableContext.complete(CompletableContext.java:74) ~[opensearch-core-1.2.3.jar:1.2.3]
	at org.opensearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:74) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[?:?]
	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1182) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:773) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:749) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1352) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472) ~[?:?]
	at io.netty.handler.ssl.SslUtils.handleHandshakeFailure(SslUtils.java:445) ~[?:?]
	at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1881) ~[?:?]
	at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1273) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1240) ~[?:?]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.opensearch.transport.TransportException: handshake failed because connection reset
	... 50 more
[2023-01-11T14:05:23,645][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [opensearch-cluster-master-0.opensearch-cluster-master-headless] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: No subject alternative names matching IP address 10.129.75.137 found
javax.net.ssl.SSLHandshakeException: No subject alternative names matching IP address 10.129.75.137 found
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1548) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1394) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.69.Final.jar:4.1.69.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.69.Final.jar:4.1.69.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 10.129.75.137 found
	at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165) ~[?:?]
	at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:426) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
	... 30 more

If I set enforce_hostname_verification to false than everything works. But such setting not suitable for us.

@yann-soubeyrand
Copy link

yann-soubeyrand commented Apr 25, 2023

Hello,
We’re also facing the same issue on a 2.6 cluster. From the debug logs, it seems that the function below is called with the address instead of the host name for the remoteAddress parameter:

public void connect(ChannelHandlerContext ctx, SocketAddress remoteAddress, SocketAddress localAddress, ChannelPromise promise) throws Exception {
SSLEngine engine = null;
try {
if (hostnameVerificationEnabled) {
final InetSocketAddress inetSocketAddress = (InetSocketAddress) remoteAddress;
String hostname = null;
if (hostnameVerificationResovleHostName) {
hostname = inetSocketAddress.getHostName();
} else {
hostname = inetSocketAddress.getHostString();
}
if(log.isDebugEnabled()) {
log.debug("Hostname of peer is {} ({}/{}) with hostnameVerificationResovleHostName: {}", hostname, inetSocketAddress.getHostName(), inetSocketAddress.getHostString(), hostnameVerificationResovleHostName);
}
engine = sks.createClientTransportSSLEngine(hostname, inetSocketAddress.getPort());
} else {
engine = sks.createClientTransportSSLEngine(null, -1);
}
} catch (final SSLException e) {
throw ExceptionsHelper.convertToOpenSearchException(e);
}
final SslHandler sslHandler = new SslHandler(engine);
ctx.pipeline().replace(this, "ssl_client", sslHandler);
super.connect(ctx, remoteAddress, localAddress, promise);
}
}

Maybe this is the expected behaviour and there must be a reverse DNS set on this address.

@peternied
Copy link
Member

@yann-soubeyrand I'd recommend enabling debug logging and seeing what is being recorded in the log statement Hostname of peer is {} ({}/{}) with hostnameVerificationResovleHostName: {}, it should help you troubleshoot.

Here is a snippet showing how this kind of logging was from another issue, or just enable debug logging at the root logger level.

logger.sslnetty.name = org.opensearch.security.ssl.transport.SecuritySSLNettyTransport
logger.sslnetty.level = debug

From #1689

@yann-soubeyrand
Copy link

@peternied there’s a mix of IP addresses and hostname for the same hosts (I modified the real domains and IP addresses):

[2023-04-26T10:00:51,369][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0.example.com (node-0.example.com/node-0.example.com) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,369][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2.example.com (node-2.example.com/node-2.example.com) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,376][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,377][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,377][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,377][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,377][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,377][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-0-IP (node-0-IP/node-0-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,380][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,380][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,380][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,380][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,380][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,380][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,381][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,381][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,381][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,381][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,381][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,381][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true
[2023-04-26T10:00:51,382][DEBUG][o.o.s.s.t.S.ClientSSLHandler] [node-1] Hostname of peer is node-2-IP (node-2-IP/node-2-IP) with hostnameVerificationResovleHostName: true

@cwperks
Copy link
Member

cwperks commented Apr 26, 2023

@yann-soubeyrand That's odd, it looks like sometimes its resolving and other times it is not. This is a class from OpenJDK14 where the name resolution is being performed if I am tracking the stack correctly: https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/net/InetAddress.java#L815-L860

Could there be some RuntimeException being thrown which is having it fall back to the host IP address? Thank you for providing the log output.

@yann-soubeyrand
Copy link

Hello @cwperks, here are the logs.
ospoc.example.com.log

@zalseryani
Copy link

zalseryani commented May 24, 2023

I am having the same issue with version 2.7 helm chart deployment.

I have created all the needed certificates, but I am using SAN and the issue still appears.

at first it was asking to add headless service name, I added it with the normal service name too, but it started checking the pods IP where it should not, I added the pods names of the statefulset and still showing an issue on the pods name, with certificate unknown.

Any updates on this issue?

  • opensearch.yml node_dn section
        nodes_dn:
          - 'CN=test.com'
  • subject alternatives in the certificate.
DNS:test.com, DNS:opensearch-cluster-master-headless, DNS:opensearch-cluster-master, DNS:opensearch-cluster-master-0, DNS:opensearch-cluster-master-1, DNS:opensearch-cluster-master-2

I tried also to revert hostname_verification to false while using the environment variables DISABLE_INSTALL_DEMO_CONFIG with value true in order to disable demo certificate generation,

but I am still getting the following error,

[2023-05-24T10:47:21,904][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [opensearch-cluster-master-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
  • Since the certificate is self-signed, it's not being trusted by the Java process, Kindly correct me if I am wrong.

Thank you.

@zalseryani
Copy link

I also tried to add the self-signed certificates within an Initial container using an empty dir and mounted the empty dir in /certs inside Opensearch container and added the following inside helm chart values.yaml

image

image

and the issue still exists, and the Java still showing the error Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

@zalseryani
Copy link

zalseryani commented May 28, 2023

The problem still happens in version v2.7, even with hostname verification being disabled, do you have any updates?

@wgnathanael
Copy link

We have the same problem. I'm fairly new to helm but figured if its possible for the network.host: 0.0.0.0 component of the config to basically be templated it out so it was network.host: <k9s-node-name> or something to that effect I think that would solve these types of problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Community contributions are especially encouraged for these issues. triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.
Projects
None yet
Development

No branches or pull requests

8 participants