Cannot unseal vault with wrong plugin checksum

[schulzh]

This issue is probably related to #3241, but covers another failure mode.
When having a vault plugin mounted, if the checksum set in sys/plugins/catalog/stuff is wrong, vault cannot be unsealed, and thus, the offending mount cannot be removed, essentially bricking the vault database. The unsealing fails with the following log:

2017/11/20 15:53:41.048250 [INFO ] core: vault is unsealed
2017/11/20 15:53:41.050187 [INFO ] core: post-unseal setup starting
2017/11/20 15:53:41.052123 [INFO ] core: loaded wrapping token key
2017/11/20 15:53:41.052161 [INFO ] core: successfully setup plugin catalog: plugin-directory=/vault/plugins
2017/11/20 15:53:41.055848 [INFO ] core: successfully mounted backend: type=generic path=secret/
2017/11/20 15:53:41.056002 [INFO ] core: successfully mounted backend: type=system path=sys/
2017/11/20 15:53:41.056228 [INFO ] core: successfully mounted backend: type=identity path=identity/
2017/11/20 15:53:41.111671 [ERROR] core: failed to create mount entry: path=somepath/ error=checksums did not match
2017/11/20 15:53:41.111731 [INFO ] core: pre-seal teardown starting
2017/11/20 15:53:41.111783 [INFO ] core: cluster listeners not running
2017/11/20 15:53:41.111869 [INFO ] core: pre-seal teardown complete
2017/11/20 15:53:41.111916 [ERROR] core: post-unseal setup failed: error=failed to setup mount table
2017/11/20 15:53:41.111941 [WARN ] core: vault is sealed

Expected Behavior:
Vault should unseal with a warnign and leave the offending mount inacessible. It should be possible to update the checksum and/or remove the mount.

[schulzh]

The same happens when deleting the plugin file, it also complains that it cannot verify the checksum:

2017/11/22 14:27:25.057578 [ERROR] core: failed to create mount entry: path=somepath/ error=error verifying checksum: open /some/path/to/some/plugin: The system cannot find the file specified.

[calvn]

What version of Vault are you running?

[schulzh]

Im running vault version 0.9.0

[calvn]

This issue is closely related to #3266 which might have been closed without being addressed.

[jefferai]

3266 was addressed, it's a different failure mode. One is unmounting, the other is unsealing.

[calvn]

The comments later in the 3266 thread also touch on the unseal case. Sorry, I was referring to that and not the original case.

[jefferai]

Oh you're right. Yeah, I think the OP problem was addressed but looking at the comments I think we have a decent idea of how to go about this.

[calvn]

We actually partially addressed the problem for the unseal scenario too (in a7dd937), but only for the case where the plugin is removed from the plugin catalog. If the plugin is removed from the catalog before sealing, during unseal the mount entry will be kept in place, but the backend will not be initialized, and that's where I remember why your wrote your last comment on 3266 (that maybe we should just entirely remove the entry).

2017/11/22 12:12:07.151058 [INFO ] core: vault is unsealed
2017/11/22 12:12:07.151272 [INFO ] core: post-unseal setup starting
2017/11/22 12:12:07.151301 [TRACE] core: clearing forwarding clients
2017/11/22 12:12:07.151314 [TRACE] core: done clearing forwarding clients
2017/11/22 12:12:07.151718 [INFO ] core: loaded wrapping token key
2017/11/22 12:12:07.151755 [INFO ] core: successfully setup plugin catalog: plugin-directory=/Users/cleung/code/tmp/vault_plugins
2017/11/22 12:12:07.152563 [INFO ] core: successfully mounted backend: type=kv path=secret/
2017/11/22 12:12:07.152928 [INFO ] core: successfully mounted backend: type=system path=sys/
2017/11/22 12:12:07.153382 [INFO ] core: successfully mounted backend: type=identity path=identity/
2017/11/22 12:12:07.153525 [ERROR] core: failed to create mount entry: path=mock/ error=plugin not found in the catalog: mock-plugin
2017/11/22 12:12:07.153575 [INFO ] core: successfully mounted backend: type=plugin path=mock/
2017/11/22 12:12:07.153633 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/

[calvn]

So I think this issue is more on missing binary (or sha256 mismatch) during unseal or an unmount operation while the plugin is still registered in the plugin catalog.

[jefferai]

Yeah, I think we should just treat any plugin failure (missing, sha mismatch) during postUnseal to behave in that way.