fix: Consul template mount path customization

[akijakya]

Overview

Fixes #236

As the /vault/secrets mount path was hardcoded into the vault-agent container, customizing the path via the vault.security.banzaicloud.io/vault-ct-secrets-mount-path or vault.security.banzaicloud.io/vault-configfile-path annotations did not work as expected.

Notes for reviewer

Using something like this now should work:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: my-app
    my-app.kubernetes.io/name: my-app-vault-agent
    branches: "true"
  name: my-app-vault-agent
data:
  config.hcl: |
    vault {
      // This is needed until https://github.com/hashicorp/vault/issues/7889
      // gets fixed, otherwise it is automated by the webhook.
      ca_cert = "/vault/tls/ca.crt"
    }
    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "default"
        }
      }
      sink "file" {
        config = {
          path = "/vault/.vault-token"
        }
      }
    }
    template {
      contents = <<EOH
        {{- with secret "secret/accounts/aws" }}
        {
          "id": "{{ .Data.data.AWS_ACCESS_KEY_ID }}",
          "key": "{{ .Data.data.AWS_SECRET_ACCESS_KEY }}"
        }
        {{ end }}
      EOH
      destination = "/tmp/secrets/config.yaml"
      command     = "/bin/sh -c \"kill -HUP $(pidof vault-demo-app) || true\""
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment-template
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: test-deployment-template
  template:
    metadata:
      labels:
        app.kubernetes.io/name: test-deployment-template
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault:8200" # optional, the address of the Vault service, default values is https://vault:8200
        vault.security.banzaicloud.io/vault-role: "default" # optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default"
        vault.security.banzaicloud.io/vault-skip-verify: "false" # optional, skip TLS verification of the Vault server certificate
        vault.security.banzaicloud.io/vault-tls-secret: "vault-tls" # optional, the name of the Secret where the Vault CA cert is, if not defined it is not mounted
        vault.security.banzaicloud.io/vault-agent: "false" # optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authentication
        vault.security.banzaicloud.io/vault-path: "kubernetes" # optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes"
        vault.security.banzaicloud.io/vault-agent-configmap: "my-app-vault-agent"
        vault.security.banzaicloud.io/vault-configfile-path: "/tmp/secrets"
    spec:
      serviceAccountName: default
      containers:
      - name: alpine
        image: alpine
        command: ["sh", "-c", "cat /tmp/secrets/config.yaml && echo going to sleep... && sleep 10000"]
        resources:
          limits:
            cpu: 20m
            memory: 10Mi
          requests:
            cpu: 20m
            memory: 10Mi

[rsafonseca]

This was a breaking change, it broke things for everyone that were already using this feature.
Basically, the mount path was already correct on the app container, it was only hardcoded for the vault-agent, hence anyone who was using this feature had the destination set to /vault/secrets in the configmap.
When doing an upgrade to >= 1.21.1, any existing configs will render this in /vault/secrets, but this will no longer be on the volume that is shared with the app container, hence the apps no longer get their secrets.

Even though this change is semantically correct, there should be at the very least a note that this is breaking for existing deployments that use this feature.

There's also no simple middle ground upgrade path, in our env we have several apps that make use of this, so upgrading VSW requires changing their deployment manifests all at the same time

@ramizpolic @akijakya