Cannot unseal vault when S3 storage backend configured #7101

adriananeci · 2019-07-10T11:45:32Z

Describe the bug
When configuring S3 as a backend storage, vault cannot be unsealed.

To Reproduce
Steps to reproduce the behaviour:

Run vault operator init
Run vault operator unseal <Unseal Key 1>
See error

Error unsealing: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/sys/unseal
Code: 400. Errors:

* Vault is not initialized

Expected behavior
Be able to unseal vault after init.

Environment:

Vault Server Version (retrieve with vault status):

vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.2.0-beta2
HA Enabled         false

Vault CLI Version (retrieve with vault version):

Vault v1.2.0-beta2

Server Operating System/Architecture:

/opt/vault$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial

Vault server configuration file(s):

ui = true

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/opt/vault/tls/vault.service.consul.crt"
  tls_key_file    = "/opt/vault/tls/vault.service.consul.key"
}

storage "s3" {
  bucket = "path/to/vault/"
  region = "<region>"
}

# HA settings
cluster_addr  = "https://<vault_IP>:8201"
api_addr      = "https://<vault_IP:8200"

Additional context
Vault is able to write into configured S3 bucket since it is adding core and sys directories into it.
/opt/vault$ aws s3 ls ****/test/vault/ --recursive

2019-07-08 11:12:51          0 test/vault/
2019-07-10 11:27:11         90 test/vault/core/audit
2019-07-10 11:27:11        320 test/vault/core/auth
2019-07-10 11:27:10        110 test/vault/core/cluster/local/info
2019-07-10 11:27:10        227 test/vault/core/keyring
2019-07-10 11:27:11         90 test/vault/core/local-audit
2019-07-10 11:27:11         89 test/vault/core/local-auth
2019-07-10 11:27:11        337 test/vault/core/local-mounts
2019-07-10 11:27:10        147 test/vault/core/master
2019-07-10 11:27:11        466 test/vault/core/mounts
2019-07-10 11:27:10        116 test/vault/core/seal-config
2019-07-10 11:27:11        534 test/vault/core/wrapping/jwtkey
2019-07-10 11:27:11        249 test/vault/sys/policy/control-group
2019-07-10 11:27:11       2685 test/vault/sys/policy/default
2019-07-10 11:27:11        232 test/vault/sys/policy/response-wrapping
2019-07-10 11:27:11        138 test/vault/sys/token/accessor/b11ba74bd78d91a604851d48c1d1071eb6a704d1
2019-07-10 11:27:11        459 test/vault/sys/token/id/hc3bddbde9cba394b0d55b9fd978933c2d470a73c2c97ed72a47d333b149f676d
2019-07-10 11:27:11         69 test/vault/sys/token/salt

The text was updated successfully, but these errors were encountered:

michelvocks · 2019-07-11T06:16:47Z

Hi @adriananeci,

I tried to reproduce your issue but wasn't able to.
It would be really helpful to have a look at the complete Vault logs output to see if something suspicious has been printed there. Would it be possible for you to provide these?

Cheers,
Michel

adriananeci · 2019-07-11T08:02:26Z

Hi @michelvocks ,

These are all the logs with log-level=trace set on vault config

Started \"HashiCorp Vault - A tool for managing secrets\".
==> Vault server configuration:
             Api Address: https://<vault_IP>:8200
                     Cgo: disabled
         Cluster Address: https://<vault_IP>:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
                 Storage: s3
                 Version: Vault v1.2.0-beta2
==> Vault server started! Log data will stream in below:
2019-07-11T07:56:47.368Z [DEBUG] storage.cache: creating LRU cache: size=0
2019-07-11T07:56:47.392Z [DEBUG] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
2019-07-11T07:57:06.970Z [INFO]  core: security barrier not initialized
2019-07-11T07:57:07.067Z [INFO]  core: security barrier initialized: shares=5 threshold=3
2019-07-11T07:57:07.120Z [DEBUG] core: cluster name not found/set, generating new
2019-07-11T07:57:07.121Z [DEBUG] core: cluster name set: name=vault-cluster-31d4cb05
2019-07-11T07:57:07.121Z [DEBUG] core: cluster ID not found, generating new
2019-07-11T07:57:07.121Z [DEBUG] core: cluster ID set: id=babe0362-448a-d2b4-4aef-c8392333c38b
2019-07-11T07:57:07.140Z [INFO]  core: post-unseal setup starting
2019-07-11T07:57:07.140Z [DEBUG] core: clearing forwarding clients
2019-07-11T07:57:07.140Z [DEBUG] core: done clearing forwarding clients
2019-07-11T07:57:07.183Z [INFO]  core: loaded wrapping token key
2019-07-11T07:57:07.183Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
2019-07-11T07:57:07.197Z [INFO]  core: no mounts; adding default mount table
2019-07-11T07:57:07.250Z [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2019-07-11T07:57:07.251Z [INFO]  core: successfully mounted backend: type=system path=sys/
2019-07-11T07:57:07.251Z [INFO]  core: successfully mounted backend: type=identity path=identity/
2019-07-11T07:57:07.521Z [INFO]  core: successfully enabled credential backend: type=token path=token/
2019-07-11T07:57:07.522Z [INFO]  core: restoring leases
2019-07-11T07:57:07.522Z [INFO]  rollback: starting rollback manager
2019-07-11T07:57:07.522Z [DEBUG] expiration: collecting leases
2019-07-11T07:57:07.536Z [DEBUG] expiration: leases collected: num_existing=0
2019-07-11T07:57:07.536Z [INFO]  expiration: lease restore complete
2019-07-11T07:57:07.621Z [DEBUG] identity: loading entities
2019-07-11T07:57:07.630Z [DEBUG] identity: entities collected: num_existing=0
2019-07-11T07:57:07.631Z [INFO]  identity: entities restored
2019-07-11T07:57:07.631Z [DEBUG] identity: identity loading groups
2019-07-11T07:57:07.640Z [DEBUG] identity: groups collected: num_existing=0
2019-07-11T07:57:07.640Z [INFO]  identity: groups restored
2019-07-11T07:57:07.646Z [INFO]  core: post-unseal setup complete
2019-07-11T07:57:07.743Z [INFO]  core: root token generated
2019-07-11T07:57:07.744Z [INFO]  core: pre-seal teardown starting
2019-07-11T07:57:07.744Z [DEBUG] expiration: stop triggered
2019-07-11T07:57:07.744Z [DEBUG] expiration: finished stopping
2019-07-11T07:57:07.744Z [INFO]  rollback: stopping rollback manager
2019-07-11T07:57:07.744Z [INFO]  core: pre-seal teardown complete
2019-07-11T07:57:17.898Z [DEBUG] core: unseal key supplied
2019-07-11T07:57:17.928Z [INFO]  core: security barrier not initialized
2019-07-11T07:57:35.402Z [DEBUG] core: unseal key supplied
2019-07-11T07:57:35.442Z [INFO]  core: security barrier not initialized

And this is the systemd vault service config:

[Unit]
Description=\"HashiCorp Vault - A tool for managing secrets\"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/opt/vault/config/default.hcl
[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/opt/vault/bin/vault server -config /opt/vault/config -log-level=trace
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

michelvocks · 2019-07-11T12:18:39Z

I can see from the logs that the initiation worked but then suddenly Vault is unable to find the file core/keyring again in the bucket.

I noticed that the Vault data is stored below a subpath test/vault/ in your S3 bucket. I'm wondering how did you achieve that? It looks like there is no option to define a subpath in your bucket.

Could you also double-check your Vault config again? Is the provided configuration still valid and identical?

adriananeci · 2019-07-11T13:35:58Z

@michelvocks
Indeed, I've set a subpath for the S3 bucket. And yes, the provided configuration is identical.
The storage config is:

storage "s3" {
  bucket = "<bucket_name>/test/vault/"
  region = "<region>"
}

bartdzkan · 2019-07-11T13:50:49Z

@adriananeci Looks like your service is setup incorrectly.

ExecStart=/opt/vault/bin/vault server -config /opt/vault/config -log-level=trace

Should be default.hcl or whatever your config file is.
ExecStart=/opt/vault/bin/vault server -config /opt/vault/config/default.hcl -log-level=trace

adriananeci · 2019-07-11T14:58:21Z

@bartdzkan you're wrong.
Based on documentation (https://www.vaultproject.io/docs/commands/server.html#config), if the path is a directory, all files which end in .hcl or .json are loaded.
I can confirm that, based on the logs, the config file is loaded correctly.

==> Vault server configuration:
             Api Address: https://<vault_IP>:8200
                     Cgo: disabled
         Cluster Address: https://<vault_IP>:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
                 Storage: s3
                 Version: Vault v1.2.0-beta2

michelvocks · 2019-07-12T12:11:30Z

@michelvocks
Indeed, I've set a subpath for the S3 bucket. And yes, the provided configuration is identical.
The storage config is:
storage "s3" {
  bucket = "<bucket_name>/test/vault/"
  region = "<region>"
}

Thanks. I was now able to reproduce your issue. The problem is the subpath of your S3 bucket.
The initialization works with a subpath but the unseal process does not support a subpath.

I think the best way to solve that issue is to support an additional configuration parameter called path (like the Consul storage backend does) which allows you to set a custom storage path inside of your bucket.

adriananeci · 2019-07-15T18:52:57Z

I can confirm that with a simple bucket without any path configured it is working as intended.
For the path option should I open a new feature request ticket or can this ticket be label as a feature request?

michelvocks · 2019-07-16T07:40:31Z

We can use this ticket for tracking. I started already working on this.

adriananeci mentioned this issue Jul 15, 2019

AWS KMS auto-unseal doesn't work when s3 storage and consul ha_storage configured #7085

Closed

michelvocks self-assigned this Jul 16, 2019

michelvocks mentioned this issue Jul 22, 2019

Added s3 storage path parameter #7157

Merged

jefferai closed this as completed in #7157 Jul 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot unseal vault when S3 storage backend configured #7101

Cannot unseal vault when S3 storage backend configured #7101

adriananeci commented Jul 10, 2019

michelvocks commented Jul 11, 2019

adriananeci commented Jul 11, 2019

michelvocks commented Jul 11, 2019 •

edited

Loading

adriananeci commented Jul 11, 2019

bartdzkan commented Jul 11, 2019 •

edited

Loading

adriananeci commented Jul 11, 2019

michelvocks commented Jul 12, 2019

adriananeci commented Jul 15, 2019

michelvocks commented Jul 16, 2019

Cannot unseal vault when S3 storage backend configured #7101

Cannot unseal vault when S3 storage backend configured #7101

Comments

adriananeci commented Jul 10, 2019

michelvocks commented Jul 11, 2019

adriananeci commented Jul 11, 2019

michelvocks commented Jul 11, 2019 • edited Loading

adriananeci commented Jul 11, 2019

bartdzkan commented Jul 11, 2019 • edited Loading

adriananeci commented Jul 11, 2019

michelvocks commented Jul 12, 2019

adriananeci commented Jul 15, 2019

michelvocks commented Jul 16, 2019

michelvocks commented Jul 11, 2019 •

edited

Loading

bartdzkan commented Jul 11, 2019 •

edited

Loading