expanding on TrustStore and KeyStore #4578 #4060 (#7015)

* expanding on TrustStore and KeyStore #4578 #4060

Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* expanding on TrustStore and KeyStore #4578 #4060

Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Update generate-certificates.md

Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Update opensearch-keystore.md

Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Apply suggestions from code review

Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>
Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Update security-admin.md

Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Apply suggestions from code review

Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>

---------

Signed-off-by: AntonEliatra <anton.rubin@eliatra.com>
Co-authored-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>
Co-authored-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
(cherry picked from commit 703d8a4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

_security/configuration/generate-certificates.md

@@ -128,7 +128,7 @@ openssl x509 -req -in node1.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreate
 ```
 
 
-## Sample script
+## Sample script to generate self-signed PEM certificates
 
 If you already know the certificate details and don't want to specify them interactively, use the `-subj` option in your `root-ca.pem` and CSR commands. This script creates a root certificate, admin certificate, two node certificates, and a client certificate, all with an expiration dates of two years (730 days):
 
@@ -174,6 +174,34 @@ rm client.csr
 rm client.ext
 ```
 
+## Sample script to convert PEM certificates to keystore and truststore files
+
+You can use the following script to generate a keystore and a truststore from the previously generated PEM certificates:
+
+```bash
+#!/bin/sh
+
+# Convert node certificate
+cat root-ca.pem node1.pem node1-key.pem > combined-node1.pem
+echo "Enter password for node1-cert.p12"
+openssl pkcs12 -export -in combined-node1.pem -out node1-cert.p12 -name node1
+echo "Enter password for keystore.jks"
+keytool -importkeystore -srckeystore node1-cert.p12 -srcstoretype pkcs12 -destkeystore keystore.jks
+
+# Convert admin certificate
+cat root-ca.pem admin.pem admin-key.pem > combined-admin.pem
+echo "Enter password for admin-cert.p12"
+openssl pkcs12 -export -in combined-admin.pem -out admin-cert.p12 -name admin
+echo "Enter password for keystore.jks"
+keytool -importkeystore -srckeystore admin-cert.p12 -srcstoretype pkcs12 -destkeystore keystore.jks
+
+# Import certificates to truststore
+keytool -importcert -keystore truststore.jks -file root-ca.cer -storepass changeit -trustcacerts -deststoretype pkcs12
+
+# Cleanup
+rm combined-admin.pem
+rm combined-node1.pem
+```
 
 ## Add distinguished names to opensearch.yml
 

_security/configuration/opensearch-keystore.md

@@ -9,6 +9,9 @@ nav_order: 50
 
 `opensearch-keystore` is a utility script used to manage an OpenSearch keystore. An OpenSearch keystore provides a secure method of storing sensitive information, such as passwords and keys, used in an OpenSearch cluster. The script allows you to securely create, list, add, and remove settings. It is included in the OpenSearch distribution. 
 
+This keystore is separate from the keystore and truststore used to store TLS certificates in JKS or PKCS12/PFX format in order to secure the transport and HTTP layers. For information about those keystores, refer to [Keystore and truststore files]({{site.url}}{{site.baseurl}}/security/configuration/tls/#keystore-and-truststore-files).
+{: .note} 
+
 ## Usage
 
 In order to use the `opensearch-keystore` script, you must have access to the file system containing the OpenSearch installation and the ability to execute OpenSearch scripts.
@@ -123,6 +126,9 @@ The following command removes a keystore setting:
 
 No response exists for this command. To confirm that the setting was deleted, use `opensearch-keystore list`.
 
-## KeyStore entries as OpenSearch settings
+For a complete list of secure settings that can be configured using `opensearch-keystore`, refer to [(Advanced) Using encrypted password settings for SSL]({{site.url}}{{site.baseurl}}/security/configuration/tls/#advanced-using-encrypted-password-settings-for-ssl).
+{: .note}
+
+## Keystore entries as OpenSearch settings
 
 After a setting has been added to a keystore, it is implicitly added to the OpenSearch configuration as if it were another entry in `opensearch.yml`. To modify a keystore entry use `./bin/opensearch-keystore upgrade <setting>`. To remove an entry, use `./bin/opensearch-keystore remove <setting>`.

_security/configuration/security-admin.md

@@ -18,6 +18,9 @@ The first job of the script is to initialize the `.opendistro_security` index. T
 
 The script can be found at `/plugins/opensearch-security/tools/securityadmin.sh`. This is a relative path showing where the `securityadmin.sh` script is located. The absolute path depends on the directory where you've installed OpenSearch. For example, if you use Docker to install OpenSearch, the path will resemble the following: `/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh`.
 
+The `securityadmin.sh` script requires SSL/TLS HTTP to be enabled for your OpenSearch cluster. Set `plugins.security.ssl.http.enabled: true` in your `opensearch.yml` file before proceeding. If your cluster does not use SSL/TLS on the HTTP layer but requires `securityadmin.sh`, enable SSL/TLS on a single node, such as the`ingest` node, and then run `securityadmin.sh` on that node. Enable this setting by configuring the [REST layer TLS]({{site.url}}{{site.baseurl}}/security/configuration/tls/#rest-layer-tls) settings on only one node. Restarting OpenSearch on that node is necessary following  any change to the `opensearch.yml` file.
+{: .note}
+
 ## A word of caution
 
 If you make changes to the configuration files in `config/opensearch-security`, OpenSearch does _not_ automatically apply these changes. Instead, you must run `securityadmin.sh` to load the updated files into the index.
@@ -86,8 +89,6 @@ You can't use node certificates as admin certificates. The two must be separate.
 
 The `securityadmin.sh` tool can be run from any machine that has access to the HTTP port of your OpenSearch cluster (the default port is 9200). You can change the Security plugin configuration without having to access your nodes through SSH.
 
-`securityadmin.sh` requires that SSL/TLS transport is enabled on your opensearch cluster. In other words, make sure that the `plugins.security.ssl.http.enabled: true` is set in `opensearch.yml` before proceeding.
-{: .note}
 
 Each node also includes the tool at `plugins/opensearch-security/tools/securityadmin.sh`. You might need to make the script executable before running it:
 
@@ -101,6 +102,8 @@ To print all available command line options, run the script with no arguments:
 ./plugins/opensearch-security/tools/securityadmin.sh
 ```
 
+## Using `securityadmin` with PEM files
+
 To load your initial configuration (all YAML files), you might use the following command:
 
 ```bash
@@ -124,6 +127,31 @@ Name | Description
 `-keypass` | The password of the private key of the admin certificate, if any.
 `-cacert` | The location of the PEM file containing the root certificate. You can use an absolute or relative path. Relative paths are resolved relative to the execution directory of `securityadmin.sh`.
 
+## Using `securityadmin` with keystore and truststore files
+
+JKS format keystore files are compatible with `securityadmin.sh`, as shown in the following example setting:
+
+```bash
+./securityadmin.sh -cd ../../../config/opensearch-security -icl -nhnv
+  -ts <path/to/truststore> -tspass <truststore password>
+  -ks <path/to/keystore> -kspass <keystore password>
+```
+
+Use the following options to control the keystore and truststore settings.
+
+Name | Description
+:--- | :---
+`-ks` | The location of the keystore containing the admin certificate and all intermediate certificates, if any. You can use an absolute or relative path. Relative paths are resolved relative to the `securityadmin.sh` execution directory.
+`-kspass` | The keystore password.
+`-kst` | The keystore type, either JKS or PKCS#12/PFX. If not specified, the Security plugin tries to determine the type based on the file extension.
+`-ksalias` | The alias of the admin certificate, if any.
+`-ts` | The location of the truststore containing the root certificate. You can use an absolute or relative path. Relative paths are resolved relative to the `securityadmin.sh` execution directory.
+`-tspass` | The truststore password.
+`-tst` | The truststore type, either JKS or PKCS#12/PFX. If not specified, the Security plugin tries to determine the type based on the file extension.
+`-tsalias` | The alias for the root certificate, if any.
+
+The certificate authority (CA) that signs the `admin` certificate can differ from the one used for signing transport or HTTP certificates. The CA does, however, need to be added to the truststore in order to validate the certificate. See [Generate node and client certificates]({{site.url}}{{site.baseurl}}/security/configuration/generate-certificates/#optional-generate-node-and-client-certificates) for more information.
+{: .note}
 
 ## Sample commands
 
@@ -162,30 +190,6 @@ Apply all YAML files in `config/opensearch-security/` with keystore and truststo
 ```
 
 
-## Using securityadmin with keystore and truststore files
-
-You can also use keystore files in JKS format in conjunction with `securityadmin.sh`:
-
-```bash
-./securityadmin.sh -cd ../../../config/opensearch-security -icl -nhnv
-  -ts <path/to/truststore> -tspass <truststore password>
-  -ks <path/to/keystore> -kspass <keystore password>
-```
-
-Use the following options to control the key and truststore settings.
-
-Name | Description
-:--- | :---
-`-ks` | The location of the keystore containing the admin certificate and all intermediate certificates, if any. You can use an absolute or relative path. Relative paths are resolved relative to the execution directory of `securityadmin.sh`.
-`-kspass` | The password for the keystore.
-`-kst` | The key store type, either JKS or PKCS#12/PFX. If not specified, the Security plugin tries to determine the type from the file extension.
-`-ksalias` | The alias of the admin certificate, if any.
-`-ts` | The location of the truststore containing the root certificate. You can use an absolute or relative path. Relative paths are resolved relative to the execution directory of `securityadmin.sh`.
-`-tspass` | The password for the truststore.
-`-tst` | The truststore type, either JKS or PKCS#12/PFX. If not specified, the Security plugin tries to determine the type from the file extension.
-`-tsalias` | The alias for the root certificate, if any.
-
-
 ### OpenSearch settings
 
 If you run a default OpenSearch installation, which listens on port 9200 and uses `opensearch` as a cluster name, you can omit the following settings altogether. Otherwise, specify your OpenSearch settings by using the following switches.
@@ -326,4 +330,4 @@ For example, to load your initial configuration (all YAML files), use the follow
   -cacert ..\..\..\config\root-ca.pem ^
   -cert ..\..\..\config\kirk.pem ^
   -key ..\..\..\config\kirk-key.pem
-```
+```