How to migrate storeAccountKey from true to false

[susanwangmcpod]

My customer uses AKS with Azure Files PVs, which were dynamically provisioned previously, and the storage account keys are stored in the secrets.

Now due to security policies, they need to rotate storage account keys periodically. If the PVs are dynamically provisioned from SC with parameter storeAccountKey: false, then it just needs to recreate pods after storage account keys are changed.

However, customer now has many existing PVs which were dynamically provisioned from SC without specifying the parameter storeAccountKey, so the storage account keys are stored in the secrets now.

So are there any ways to migrate storeAccountKey from true to false? If this is not possible, then whether we have any other methods to update secrets with new storage account keys automatically? Thanks in advance.

[andyzhangx]

@susanwangmcpod if you don't want to store AccountKey as secret, you need to:

1. grant kubelet identity reader permission to the storage accounts
2. delete existing secrets with naming rule azure-storage-account-{accountname}-secret
3. when you rotate account key, existing azure file volume mount would become invalid, recreate those pods with azure file mounts, and azure file driver would use kubelet identity to get the updated account key directly.

[susanwangmcpod]

Thanks, Andy. I will have a test.

[susanwangmcpod]

@andyzhangx my customer spent some time in testing above shared steps. It works. But they feedbacked that this is still unable to meet their requirements, because pods are unable to access files after storage account keys rotate. This means their app still have some downtime until these pods are recreated. So customer is asking whether there exists any other methods to avoid such downtime when storage account keys rotate? Thanks in advance.

[andyzhangx]

@andyzhangx my customer spent some time in testing above shared steps. It works. But they feedbacked that this is still unable to meet their requirements, because pods are unable to access files after storage account keys rotate. This means their app still have some downtime until these pods are recreated. So customer is asking whether there exists any other methods to avoid such downtime when storage account keys rotate? Thanks in advance.

@susanwangmcpod
there is a better way to reduce downtime, e.g.

1. update secret with secondary account key
2. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen)
3. rotate primary account key
4. update secret with primary account key
5. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen)

[susanwangmcpod]

Thanks, Andy.

[susanwangmcpod]

Hello @andyzhangx ,

I did some tests, and it looks like that kubelet identity always fetches the first storage account key (key1). Is this by design?

My guess is if kubelet identity can fetch the latest rotated key, then we may follow the similar steps as above to reduce downtime with kubelet identity as well.

Thanks.

[andyzhangx]

Hello @andyzhangx ,

I did some tests, and it looks like that kubelet identity always fetches the first storage account key (key1). Is this by design?

My guess is if kubelet identity can fetch the latest rotated key, then we may follow the similar steps as above to reduce downtime with kubelet identity as well.

Thanks.

@susanwangmcpod that's a good suggestion, currently it's always getting the first key, but I think we make a change in the next version to always get the key with latest CreationTime, by that way, if you are using kubelet identity to get account key directly, it could be:

1. rotate secondary key
2. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen)
3. rotate primary account key
4. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen)

// AccountKey an access key for the storage account.
type AccountKey struct {
	// KeyName - READ-ONLY; Name of the key.
	KeyName *string `json:"keyName,omitempty"`
	// Value - READ-ONLY; Base 64-encoded value of the key.
	Value *string `json:"value,omitempty"`
	// Permissions - READ-ONLY; Permissions for the key -- read-only or full permissions. Possible values include: 'KeyPermissionRead', 'KeyPermissionFull'
	Permissions KeyPermission `json:"permissions,omitempty"`
	// CreationTime - READ-ONLY; Creation time of the key, in round trip date format.
	CreationTime *date.Time `json:"creationTime,omitempty"`
}

[susanwangmcpod]

Many thanks, Andy.

[yzhang0727]

@andyzhangx hello andy, for the time of next version release, do you have an estimation time?

[andyzhangx]

@andyzhangx hello andy, for the time of next version release, do you have an estimation time?

@yzhang0727 working on this now

[rkarra08101992]

Hello Andy, I see that there so many manual steps involved to rotate keys. Is there a way to automatically do this without having to perform those steps?

[andyzhangx]

@rkarra08101992 after setting following two parameters in storage class, you could try following steps to validate it works, and then you could automate those cli commands

• option#1: account key is already stored as k8s secret

parameters:
  getLatestAccountKey: "true"

1. update secret with secondary account key
2. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen, and with getLatestAccountKey: "true" CSI driver will pick the secondary account key)
3. rotate primary account key
4. update secret with primary account key
5. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen, and with getLatestAccountKey: "true" CSI driver will pick the primary account key)

• option#2: account key is NOT already stored (more straightforward)

parameters:
  storeAccountKey: "false"
  getLatestAccountKey: "true"

1. rotate secondary account key
2. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen, and with getLatestAccountKey: "true" CSI driver will pick the secondary account key)
3. rotate primary account key
4. recreate pods by moving pods to the node which does not have this file share mount (this could make sure remount happen, and with getLatestAccountKey: "true" CSI driver will pick the primary account key)