AWS KMS auto-unseal doesn't work when s3 storage and consul ha_storage configured

[adriananeci]

Describe the bug
When configuring S3 as a storage and consul as a ha_storage in vault config, aws kms seal option is not working properly and vault operator init command returns an error and is not able to initialise vault.

To Reproduce
Steps to reproduce the behavior:

1. Run systemctl start vault
2. Run vault operator init
3. See error

Error initializing: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/sys/init
Code: 500. Errors:

* unseal with stored key failed: Vault is not initialized

Expected behavior
Vault should successfully init and automatically unseal.

Environment:
vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   true
Total Recovery Shares    5
Threshold                3
Unseal Progress          0/3
Unseal Nonce             n/a
Version                  1.1.3
HA Enabled               true

vault version

Vault v1.1.3 ('9bc820f700f83a7c4bcab54c5323735a581b34eb')

lsb_release -a

Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial

Vault server configuration file(s):
cat /opt/vault/config/default.hcl:

ui = true
seal "awskms" {
  kms_key_id = "<aws_kms_key>"
  region     = "<aws_region>"
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/opt/vault/tls/vault.service.consul.crt"
  tls_key_file    = "/opt/vault/tls/vault.service.consul.key"
}

storage "s3" {
  bucket = "<path/to/vault/bucket/>"
  region = "<aws_region>"
}

ha_storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
  scheme  = "http"
  service = "vault"
}

# HA settings
cluster_addr  = "https://<vault_IP>:8201"
api_addr      = "https://<vault_IP>:8200"

Additional context
Based on vault logs it seems that it tries to correctly init vault

 ==> Vault server configuration:
            AWS KMS KeyID: <aws_kms_key>
           AWS KMS Region: <aws_region>
               HA Storage: consul
                Seal Type: awskms
              Api Address: https://<vault_IP>:8200
                      Cgo: disabled
          Cluster Address: https://<vault_IP>:8201
               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
                Log Level: info
                    Mlock: supported: true, enabled: true
                  Storage: s3
                  Version: Vault v1.1.3
              Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb
 ==> Vault server started! Log data will stream in below:
 2019-07-08T14:22:16.712Z [INFO]  core: stored unseal keys supported, attempting fetch
 2019-07-08T14:22:16.720Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
 2019-07-08T14:22:19.850Z [WARN]  core: stored keys supported on init, forcing shares/threshold to 1
 2019-07-08T14:22:19.864Z [INFO]  core: security barrier not initialized
 2019-07-08T14:22:19.936Z [INFO]  core: security barrier initialized: shares=1 threshold=1
 2019-07-08T14:22:20.083Z [INFO]  core: post-unseal setup starting
 2019-07-08T14:22:20.183Z [INFO]  core: loaded wrapping token key
 2019-07-08T14:22:20.183Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
 2019-07-08T14:22:20.198Z [INFO]  core: no mounts; adding default mount table
 2019-07-08T14:22:20.297Z [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
 2019-07-08T14:22:20.297Z [INFO]  core: successfully mounted backend: type=system path=sys/
 2019-07-08T14:22:20.298Z [INFO]  core: successfully mounted backend: type=identity path=identity/
 2019-07-08T14:22:20.561Z [INFO]  core: successfully enabled credential backend: type=token path=token/
 2019-07-08T14:22:20.562Z [INFO]  core: restoring leases
 2019-07-08T14:22:20.562Z [INFO]  rollback: starting rollback manager
 2019-07-08T14:22:20.577Z [INFO]  expiration: lease restore complete
 2019-07-08T14:22:20.639Z [INFO]  identity: entities restored
 2019-07-08T14:22:20.647Z [INFO]  identity: groups restored
 2019-07-08T14:22:20.655Z [INFO]  core: post-unseal setup complete
 2019-07-08T14:22:20.811Z [INFO]  core: root token generated
 2019-07-08T14:22:20.811Z [INFO]  core: pre-seal teardown starting
 2019-07-08T14:22:20.812Z [INFO]  rollback: stopping rollback manager
 2019-07-08T14:22:20.812Z [INFO]  core: pre-seal teardown complete
 2019-07-08T14:22:20.812Z [INFO]  core: stored unseal keys supported, attempting fetch
 2019-07-08T14:22:20.861Z [INFO]  core: security barrier not initialized
 2019-07-08T14:22:21.721Z [INFO]  core: stored unseal keys supported, attempting fetch
 2019-07-08T14:22:21.755Z [INFO]  core: security barrier not initialized
 2019-07-08T14:22:21.756Z [WARN]  failed to unseal core: error="unseal with stored key failed: Vault is not initialized"
 2019-07-08T14:22:26.756Z [INFO]  core: stored unseal keys supported, attempting fetch
 2019-07-08T14:22:26.809Z [INFO]  core: security barrier not initialized
 2019-07-08T14:22:26.810Z [WARN]  failed to unseal core: error="unseal with stored key failed: Vault is not initialize

Vault is also able to write in the provided s3 bucket since it creates the core and sys folders inside this bucket.

If I remove the S3 storage and consul ha_storage options and configure consul as a storage, vault is able to init and also performs auto-unseal without any errors.
vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.1.3
Cluster Name             vault-cluster-f1f0413a
Cluster ID               6784f035-6e15-9c62-b5b9-cb6a980d35d9
HA Enabled               true
HA Cluster               https://<vault_IP>:8201
HA Mode                  active

==> Vault server configuration:
            AWS KMS KeyID: <aws_kms_key>
           AWS KMS Region: <aws_region>
                Seal Type: awskms
             Api Address: https://<vault_IP>:8200
                      Cgo: disabled
          Cluster Address: https://<vault_IP>:8201
               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
                Log Level: info
                    Mlock: supported: true, enabled: true
                 Storage: consul (HA available)
                  Version: Vault v1.1.3
              Version Sha: 9bc820f700f83a7c4bcab54c5323735a581b34eb
ui = true
 ==> Vault server started! Log data will stream in below:

[bartdzkan]

You should only have one storage stanza.
S3 or Consul. Not both.

Consul is the preferred storage backend.
S3 storage backend does not support high availability.
Please Note:
The S3 storage backend is supported by the community. While it has undergone review by HashiCorp employees, they may not be as knowledgeable about the technology. If you encounter problems with them, you may be referred to the original author.

[adriananeci]

It's only one storage stanza configured(S3), the other one is an ha_storage stanza(consul).

Based on the documentation (https://www.vaultproject.io/docs/configuration/#ha_storage) if storage backend does not support HA (which is the case of S3), you can use ha_storage stanza to configure a storage backend where Vault HA coordination will take place.

How can I find the author that is actively supporting the S3 backend vault plugin?

[bartdzkan]

What I meant was you can only have storage or HA storage not both.

[adriananeci]

@bartdzkan are you sure? Then what's the difference between storage and ha_storage?

storage (StorageBackend: <required>) – Configures the storage backend where Vault data is stored. Please see the storage backends documentation for the full list of available storage backends. Running Vault in HA mode would require coordination semantics to be supported by the backend. If the storage backend supports HA coordination, HA backend options can also be specified in this parameter block. If not, a separate ha_storage parameter should be configured with a backend that supports HA, along with corresponding HA options.

ha_storage (StorageBackend: nil) – Configures the storage backend where Vault HA coordination will take place. This must be an HA-supporting backend. If not set, HA will be attempted on the backend given in the storage parameter. This parameter is not required if the storage backend supports HA coordination and if HA specific options are already specified with storage parameter.

[bartdzkan]

@adriananeci

You don't have to use HA storage, and Storage will automatically know if its HA storage.
In your example, you have two storage backends. Consul and S3.

You need to pick which one you want to use. Once you do that, your vault should function properly.

[adriananeci]

@bartdzkan you're wrong.
Vault is able to use both storage and ha_storage on the same time if the storage backend is not HA capable.
Just tested it and the problem seems to be with the S3 vault plugin which is not able to handle S3 bucket subpaths .
In my config, under storage stanza, the bucket value is somethings like <bucket_name>/test/vault/

[adriananeci]

Looks like this issue is related to #7101 so it can be closed.

[riuvshyn]

@bartdzkan it is definitely possible to have separate HA and data storages like S3 + MySQL(for HA coordination) there are many advantages of separating these backends, so @adriananeci is right.

[anouarchattouna]

Hello @adriananeci, did you have the vault service registered as in consul when using s3 storage and consul ha_storage configured ?
I'm using the same configuration than yours, but I can not see vault service in the consul UI actually. And consul catalog services is returning only consul ?