Rotating root password while MySQL storage backend is down breaks the connection config

[ahilsend]

Describe the bug
When rotating root password fails to be stored in the storage backend the database connection config is broken. The used storage backend is MySQL. It appears vault is rotating the root credentials even though the new credentials are never persisted. Breaking the database config.

To Reproduce

I've created a docker-compose test case with some scripts to ease reproducing the issue.

my.cnf:

[client]
host = localhost
user = root
password = root

config.hcl:

log_level = "debug"

listener "tcp" {
  address     = "[::]:8200"
  tls_disable = "true"
}

storage "mysql" {
  address    = "vaultdb"
  username   = "vault"
  password   = "vault"
  ha_enabled = "false"
}

scripts/configure-mysql.sh:

#!/bin/sh

mysql -e "GRANT ALL PRIVILEGES ON *.* TO 'vault'@'%' WITH GRANT OPTION;"

scripts/configure-vault.sh:

#!/bin/sh

set -ex

source /scripts/.vault-env

vault secrets enable database || true
vault write database/config/users \
          plugin_name=mysql-database-plugin \
          allowed_roles="*" \
          connection_url="{{username}}:{{password}}@tcp(userdb:3306)/${USERS_DB_DATABASE}" \
          username="${USERS_DB_USERNAME}" \
          password="${USERS_DB_PASSWORD}"

vault write database/roles/users-readonly \
          db_name=users \
          creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%';" \
          default_ttl=1h max_ttl=24h

vault read database/creds/users-readonly

scripts/rotate-userdb-pw.sh:

#!/bin/sh

set -ex

source /scripts/.vault-env

vault write -force database/rotate-root/users

scripts/test-vault.sh:

#!/bin/sh

set -ex

source /scripts/.vault-env

vault read database/creds/users-readonly

scripts/vault-unseal.sh:

#!/bin/sh

set -ex

source /scripts/.vault-env

vault operator unseal <unseal-key-1>
vault operator unseal <unseal-key-2>
vault operator unseal <unseal-key-3>

scripts/.vault-env:

export VAULT_TOKEN="<vault-token>"

docker-compose.yaml:

version: "2.4"

services:
  vault:
    image: vault:1.0.1
    cap_add:
      - IPC_LOCK
    command: "vault server -config /config/"
    environment:
      VAULT_ADDR: "http://localhost:8200"
      USERS_DB_USERNAME: "vault"
      USERS_DB_PASSWORD: "vault"
    volumes:
      - ./scripts:/scripts
      - ./config.hcl:/config/config.hcl
    depends_on:
      - vaultdb

  vaultdb:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: "root"
      MYSQL_DATABASE: "vault"
      MYSQL_USER: "vault"
      MYSQL_PASSWORD: "vault"

  userdb:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: "root"
      MYSQL_DATABASE: "users"
      MYSQL_USER: "vault"
      MYSQL_PASSWORD: "vault"
    volumes:
      - ./scripts:/scripts
      - ./my.cnf:/root/.my.cnf

Steps to reproduce the behavior:

1. Start everything: docker-compose up -d
2. Configure mysql with proper vault privileges: docker-compose exec userdb /scripts/configure-mysql.sh
3. Restart vault (as initial mysql start takes a while, and it wasn't ready when vault came up): docker-compose restart vault
4. Initialize vault: docker-compose exec vault vault operator init
5. Fill in token in scripts/.vault-env and unseal keys in scripts/vault-unseal.sh
6. Unseal vault docker-compose exec vault /scripts/vault-unseal.sh
7. Configure database config: docker-compose exec vault /scripts/configure-vault.sh
8. Test database role credentials: docker-compose exec vault /scripts/test-vault.sh
9. Stop vault storage backend: docker-compose stop vaultdb
10. Rotate database root credentials: docker-compose exec vault /scripts/rotate-userdb-pw.sh
11. Start storage backend again: docker-compose start vaultdb
12. Restart vault: docker-compose restart vault
13. Unseal vault docker-compose exec vault /scripts/vault-unseal.sh
14. Test database role credentials: docker-compose exec vault /scripts/test-vault.sh - This fails

Expected behavior
If the there are errors during the root password rotation, everything should be rolled back and continue to work.

Environment:

• Vault Server Version (retrieve with vault status): 1.0.1
• Vault CLI Version (retrieve with vault version): 1.0.1
• Server Operating System/Architecture: offical docker image

Vault server configuration file(s):

log_level = "debug"

listener "tcp" {
  address     = "[::]:8200"
  tls_disable = "true"
}

storage "mysql" {
  address    = "vaultdb"
  username   = "vault"
  password   = "vault"
  ha_enabled = "false"
}