Add SSL Configuration Library #37287

tvernum · 2019-01-10T05:05:05Z

This introduces a new ssl-config library that can parse
and validate SSL/TLS settings and files.

It supports the standard configuration settings as used in the
Elastic Stack such as "ssl.verification_mode" and
"ssl.certificate_authorities" as well as all file formats used
in other parts of Elasticsearch security (such as PEM, JKS,
PKCS#12, PKCS#8, et al).

Relates: #29755

This commit just adds the new library, a follow up change will make use of it in reindex.
At a future point in time I will also switch X-Pack security to use some/all of this library.

Most of this code is based on existing X-Pack code, but has been heavily refactored in the process.
The PEM parsing code has been modified slightly, mostly due to needing to be part of a standalone library.

This introduces a new ssl-config library that can parse and validate SSL/TLS settings and files. It supports the standard configuration settings as used in the Elastic Stack such as "ssl.verification_mode" and "ssl.certificate_authorities" as well as all file formats used in other parts of Elasticsearch security (such as PEM, JKS, PKCS#12, PKCS#8, et al).

elasticmachine · 2019-01-10T05:05:06Z

Pinging @elastic/es-security

tvernum · 2019-01-10T05:16:42Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/DerParser.java

+ * Based on https://github.com/groovenauts/jmeter_oauth_plugin/blob/master/jmeter/src/
+ * main/java/org/apache/jmeter/protocol/oauth/sampler/PrivateKeyReader.java
+ */
+final class DerParser {


Cloned from org.elasticsearch.xpack.core.ssl

tvernum · 2019-01-10T05:18:01Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/KeyStoreUtil.java

+/**
+ * A variety of utility methods for working with or constructing {@link KeyStore} instances.
+ */
+final class KeyStoreUtil {


Most of these methods existed (in some variation) in org.elasticsearch.xpack.core.ssl.CertParsingUtils.
I pulled in just what I needed.

tvernum · 2019-01-10T05:19:38Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/PemUtils.java

+import java.util.Map;
+import java.util.function.Supplier;
+
+final class PemUtils {


Cloned from org.elasticsearch.xpack.core.ssl with minor variations (mostly due to not being able to use core ES classes) and some additional detail in error messages

tvernum · 2019-01-10T05:20:59Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslConfigException.java

+/**
+ * A base exception for problems that occur while trying to configure SSL.
+ */
+public class SslConfigException extends RuntimeException {


A simple base class because code in libs/ can't use ElasticsearchException

tvernum · 2019-01-10T05:23:19Z

libs/ssl-config/src/test/resources/certs/pem-utils/README.asciidoc

@@ -0,0 +1,149 @@
+= Keystore Details
+This document details the steps used to create the certificate and keystore files in this directory.


This file and all the keys/certs in this directory were cloned from the x-pack pem-utils tests.

We can't control the message, and it varies by JRE release so there's no good reason to assert on it.

jkakavas

I did a first pass and it looks good. I didn't make it to the tests but I'll do so tomorrow with a cleaner head. It sure helped that you marked the cloned files 👍

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/KeyStoreUtil.java

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/PemKeyConfig.java

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslConfiguration.java

jkakavas · 2019-01-10T15:01:49Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslClientAuthenticationMode.java

+     */
+    public abstract void configure(SSLParameters sslParameters);
+
+    private static final Map<String, SslClientAuthenticationMode> LOOKUP = new LinkedHashMap<>(3);


Not that I see an issue with this, but for my own benefit/curiosity: Is there a specific reason to use an extra LinkedHashMap and not use Enum.valueOf, or a switch statement to get the SslClientAuthenticationMode from its name ?

The version I copied from had a switch, but that style has a few weaknesses (IMO)

You can't create reasonable error messages without duplicating the keys. The old code would switch on string constants, but then use Enum.name().toLowerCase() for the error message. It works, but there's no guarantees of consistency between those 2 sets of values.

It ends up longer and more verbose (particularly when you want helpful diagnostic erros) for no real beenfit.

The other option is to simply use Enum.valueOf and Enum.name which ties the configuration strings to the enum constants. There's a minor risk in that changes to the enum names change the accepted settings, but that risk is covered pretty weel by tests.
Personally that's my preference, but it's not what we normally do, so I didn't go that direction.

jkakavas · 2019-01-10T15:05:07Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslConfiguration.java

+    }
+
+    /**
+     * Picks the best (highest security / most recent standard) SSL/TLS algorithm that is supported by the


Suggested change

* Picks the best (highest security / most recent standard) SSL/TLS algorithm that is supported by the

* Picks the best (highest security / most recent standard) SSL/TLS version that is supported by the

I resolved this with a slightly different wording. Let me know if you still see an issue.

libs/ssl-config/build.gradle

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/KeyStoreUtil.java

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/PemKeyConfig.java

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslClientAuthenticationMode.java

jaymode · 2019-01-10T16:50:29Z

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslConfiguration.java

+    private String contextAlgorithm() {
+        if (supportedProtocols.isEmpty()) {
+            // shouldn't happen...
+            return "TLSv1.2";


if it shouldn't happen, throw an exception?

libs/ssl-config/src/main/java/org/elasticsearch/common/ssl/SslVerificationMode.java

libs/ssl-config/src/test/resources/certs/README.txt

alpar-t · 2019-01-11T09:47:21Z

libs/ssl-config/build.gradle

+ * under the License.
+ */
+
+archivesBaseName = 'elasticsearch-ssl-config'


It would be preferable to alter project.name in settings.gradle as that would have effects across the board, including what you are looking for here. Also means that if we were to publish this the correct pom would be generated without additional configuration and better visibility in the IDE.

There are some additional changes needed in settings.gradle to keep Eclipse happy, see the discussion in #36477 for a similar change.

@atorok I pushed a5d0144 which I think covers what you've recommended.
Can you let me know if there's more we need to do?

Thanks @tvernum , looks good! You can now also remove this line archiveBaseName since with the changed project name that's the default now.

jaymode

LGTM

tvernum · 2019-01-15T10:35:20Z

Hmm, @elasticmachine, let's run the default distro tests again.

tvernum · 2019-01-16T04:15:57Z

@elasticmachine -- run gradle build tests 1
@elasticmachine -- run gradle build tests 2

* master: Deprecate _type from LeafDocLookup (elastic#37491) Allow system privilege to execute proxied actions (elastic#37508) Update Put Watch to allow unknown fields (elastic#37494) AwaitsFix testAddNewReplicas SQL: Add protocol tests and remove jdbc_type from drivers response (elastic#37516) SQL: [Docs] Add an ES-SQL column for data types (elastic#37529) IndexMetaData#mappingOrDefault doesn't need to take a type argument. (elastic#37480) Simplify + Cleanup Dead Code in Settings (elastic#37341) Reject all requests that have an unconsumed body (elastic#37504) [Ml] Prevent config snapshot failure blocking migration (elastic#37493) Fix line length for aliases and remove suppression (elastic#37455) Add SSL Configuration Library (elastic#37287) SQL: Remove slightly used meta commands (elastic#37506) Simplify Snapshot Create Request Handling (elastic#37464) Remove the use of AbstracLifecycleComponent constructor elastic#37488 (elastic#37488) [ML] log minimum diskspace setting if forecast fails due to insufficient d… (elastic#37486)

albertzaharovits · 2019-01-18T17:59:33Z

FYI @tvernum for when you backport, I have pushed 633bd09 .

This introduces a new ssl-config library that can parse and validate SSL/TLS settings and files. It supports the standard configuration settings as used in the Elastic Stack such as "ssl.verification_mode" and "ssl.certificate_authorities" as well as all file formats used in other parts of Elasticsearch security (such as PEM, JKS, PKCS#12, PKCS#8, et al).

Adds reindex.ssl.* settings for reindex from remote. This uses the ssl-config/ internal library to parse and load SSL configuration and files. This is applied when using the low level rest client to connect to a remote ES node Relates: #37287 Resolves: #29755

Adds reindex.ssl.* settings for reindex from remote. This uses the ssl-config/ internal library to parse and load SSL configuration and files. This is applied when using the low level rest client to connect to a remote ES node Backport of: elastic#37527 Relates: elastic#37287 Resolves: elastic#29755

Adds reindex.ssl.* settings for reindex from remote. This uses the ssl-config/ internal library to parse and load SSL configuration and files. This is applied when using the low level rest client to connect to a remote ES node Backport of: #37527 Relates: #37287 Resolves: #29755

This is used by the reindex-client library which is published to maven Relates: elastic#37287, elastic#37527 Closes: elastic#38944

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Closes: #38944

This is used by the reindex-client library which is published to maven Relates: elastic#37287, elastic#37527 Backport of: elastic#39019

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Backport of: #39019

This is used by the reindex-client library which is published to maven Relates: elastic#37287, elastic#37527 Closes: elastic#38944 Backport of: elastic#39019

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Backport of: #39019

tvernum added >non-issue v7.0.0 :Security/TLS SSL/TLS, Certificates v6.7.0 labels Jan 10, 2019

tvernum requested review from jaymode and jkakavas January 10, 2019 05:05

Fix copyright on DerParser

ca341f6

tvernum commented Jan 10, 2019

View reviewed changes

tvernum added 2 commits January 10, 2019 18:12

Remove irrelevant assert from test

2361017

We can't control the message, and it varies by JRE release so there's no good reason to assert on it.

Merge branch 'master' into reindex-ssl-lib/1

9a33a7a

jkakavas reviewed Jan 10, 2019

View reviewed changes

jaymode requested changes Jan 10, 2019

View reviewed changes

tvernum added 3 commits January 11, 2019 15:20

Address feedback

82d0b84

Fix tests to run on BC-FIPS JVM

cde210f

Remove debugging output

32ba20e

alpar-t reviewed Jan 11, 2019

View reviewed changes

jaymode approved these changes Jan 11, 2019

View reviewed changes

tvernum added 3 commits January 15, 2019 15:26

Merge branch 'master' into reindex-ssl-lib/1

c64e31b

Address feedback on Gradle setup for ssl-config

a5d0144

Merge branch 'master' into reindex-ssl-lib/1

17bf8bd

tvernum added 2 commits January 16, 2019 13:41

Remove archivesBaseName from build.gradle

a052d35

Merge branch 'master' into reindex-ssl-lib/1

26b3f09

tvernum mentioned this pull request Jan 16, 2019

Remove TLS 1.0 as a default SSL protocol #37512

Merged

tvernum merged commit 6d99e79 into elastic:master Jan 16, 2019

tvernum mentioned this pull request Jan 16, 2019

Introduce ssl settings to reindex from remote #37527

Merged

tvernum added the backport pending label Jan 24, 2019

tvernum removed the backport pending label Jan 25, 2019

tvernum mentioned this pull request Feb 4, 2019

[BACKPORT] Introduce ssl settings to reindex from remote #38292

Merged

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

tvernum added a commit to tvernum/elasticsearch that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library

478d6b7

This is used by the reindex-client library which is published to maven Relates: elastic#37287, elastic#37527 Closes: elastic#38944

tvernum mentioned this pull request Feb 18, 2019

Generate mvn pom for ssl-config library #39019

Merged

tvernum added a commit that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library (#39019)

cbdda71

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Closes: #38944

tvernum added a commit to tvernum/elasticsearch that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library

86c7e40

This is used by the reindex-client library which is published to maven Relates: elastic#37287, elastic#37527 Backport of: elastic#39019

tvernum mentioned this pull request Feb 18, 2019

[BACKPORT 7.x] Generate mvn pom for ssl-config library #39026

Merged

tvernum added a commit to tvernum/elasticsearch that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library

8bb9a63

This is used by the reindex-client library which is published to maven Relates: elastic#37287, elastic#37527 Backport of: elastic#39019

tvernum mentioned this pull request Feb 18, 2019

[BACKPORT 7.0] Generate mvn pom for ssl-config library #39027

Merged

tvernum added a commit that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library (#39026)

8895bef

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Backport of: #39019

tvernum added a commit that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library (#39027)

9a50845

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Backport of: #39019

tvernum mentioned this pull request Feb 18, 2019

Generate mvn pom for ssl-config library #39033

Merged

tvernum added a commit that referenced this pull request Feb 18, 2019

Generate mvn pom for ssl-config library (#39033)

63baa61

This is used by the reindex-client library which is published to maven Relates: #37287, #37527 Backport of: #39019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SSL Configuration Library #37287

Add SSL Configuration Library #37287

tvernum commented Jan 10, 2019 •

edited

Loading

elasticmachine commented Jan 10, 2019

tvernum Jan 10, 2019

tvernum Jan 10, 2019

tvernum Jan 10, 2019

tvernum Jan 10, 2019

tvernum Jan 10, 2019

jkakavas left a comment

jkakavas Jan 10, 2019

tvernum Jan 11, 2019

jkakavas Jan 10, 2019

tvernum Jan 15, 2019

jaymode Jan 10, 2019

alpar-t Jan 11, 2019

tvernum Jan 15, 2019

alpar-t Jan 15, 2019

jaymode left a comment

tvernum commented Jan 15, 2019

tvernum commented Jan 16, 2019

albertzaharovits commented Jan 18, 2019

		@@ -0,0 +1,149 @@
		= Keystore Details
		This document details the steps used to create the certificate and keystore files in this directory.

	* Picks the best (highest security / most recent standard) SSL/TLS algorithm that is supported by the
	* Picks the best (highest security / most recent standard) SSL/TLS version that is supported by the

Add SSL Configuration Library #37287

Add SSL Configuration Library #37287

Conversation

tvernum commented Jan 10, 2019 • edited Loading

elasticmachine commented Jan 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkakavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaymode left a comment

Choose a reason for hiding this comment

tvernum commented Jan 15, 2019

tvernum commented Jan 16, 2019

albertzaharovits commented Jan 18, 2019

tvernum commented Jan 10, 2019 •

edited

Loading