Vault CA Provider does not clean up old issuers and keys in Vault

[kisunji]

Overview of the Issue

If Consul is using Vault as CA, it does not clean up old issuers and keys when generating a new leaf-signing certificate (e.g. when a new leader is elected).

This is usually not a problem in stable clusters. Leadership change is infrequent and Consul configures its intermediate mount to auto-tidy its expired issuers (leaf-signing certificates have a default expiry time of 1 year).

However, in clusters which experience frequent leadership change, the rate at which issuers and keys are created may outpace the rate at which they expire and get auto-tidied. In one case, a user reported degraded Vault performance at 1850 issuers created by Consul's CA system.

Workaround

List all non-default issuer IDs:

vault list -detailed -format json <INTERMEDIATE_PKI_MOUNT>/issuers \
| jq '.data.key_info | with_entries( select(.value.is_default | not) ) | keys'

Find the key ID associated with each issuer:

vault read -format json <INTERMEDIATE_PKI_MOUNT>/issuer/<ISSUER_ID> \
| jq '.data.key_id'

Delete the issuer first:

vault delete <INTERMEDIATE_PKI_MOUNT>/issuer/<ISSUER_ID>

Delete the key after issuer is deleted:

vault delete <INTERMEDIATE_PKI_MOUNT>/key/<KEY_ID>

[jkirschner-hashicorp]

Fixed by #18773