Merge pull request #537 from nebari-dev/536-eks-cluster-encryption

Add documentation for amazon_web_services.eks_kms_arn config option

docs/docs/explanations/advanced-provider-configuration.md

@@ -98,6 +98,46 @@ amazon_web_services:
   permissions_boundary: arn:aws:iam::01234567890:policy/<permissions-boundary-policy-name>
 ```
 
+### EKS KMS ARN (Optional)
+
+You can use AWS Key Management Service (KMS) to enhance security by encrypting Kubernetes secrets in
+Amazon Elastic Kubernetes Service (EKS). This approach adds an extra layer of protection for sensitive
+information, like passwords, credentials, and TLS keys, by applying user-managed encryption keys to Kubernetes
+secrets, supporting a [defense-in-depth strategy](https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/).
+
+Nebari supports setting an existing KMS key while deploying Nebari to implement encryption of secrets
+created in Nebari's EKS cluster. The KMS key must be a **Symmetric** key set to **encrypt and decrypt** data.
+
+:::warning
+Enabling EKS cluster secrets encryption, by setting `amazon_web_services.eks_kms_arn`, is an
+_irreversible_ action and re-deploying Nebari to try to remove a previously set `eks_kms_arn` will fail.
+On the other hand, if you try to change the KMS key in use for cluster encryption, by re-deploying Nebari
+after setting a _different_ key ARN, the re-deploy should succeed but the KMS key used for encryption will
+not actually change in the cluster config and the original key will remain set. The integrity of a faulty
+deployment can be restored, following a failed re-deploy attempt to remove a previously set KMS key, by
+simply re-deploying Nebari while ensuring `eks_kms_arn` is set to the original KMS key ARN.
+:::
+
+:::danger
+If the KMS key used for envelope encryption of secrets is ever deleted, then there is no way to recover
+the EKS cluster.
+:::
+
+:::note
+After enabling cluster encryption on your cluster, you must encrypt all existing secrets with the
+new key by running the following command:
+`kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - kms-encryption-timestamp="time value"`
+Consult [Encrypt K8s secrets with AWS KMS on existing clusters](https://docs.aws.amazon.com/eks/latest/userguide/enable-kms.html) for more information.
+:::
+
+Here is an example of how you would set KMS key ARN in `nebari-config.yaml`.
+
+```yaml
+amazon_web_services:
+  # the arn for the AWS Key Management Service key
+  eks_kms_arn: "arn:aws:kms:us-west-2:01234567890:key/<aws-kms-key-id>"
+```
+
 ### Launch Templates (Optional)
 
 Nebari supports configuring launch templates for your node groups, enabling you to customize settings like the AMI ID and pre-bootstrap commands. This is particularly useful if you need to use a custom AMI or perform specific actions before the node joins the cluster.