Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
UPSTREAM: 40417: Always detach volumes in operator executor
Instead of marking a volume as detached immediately in Kubelet's reconciler, delegate the marking asynchronously to the operator executor. This is necessary to prevent race conditions with other operations mutating the same volume state. An example of one such problem: pod is created, volume is added to desired state of the world reconciler process starts reconciler starts MountVolume, which is kicked off asynchronously via operation_executor.go MountVolume mounts the volume, but hasn't yet marked it as mounted pod is deleted, volume is removed from desired state of the world reconciler reaches detach volume section, detects volume is no longer in desired state of world, removes it from volumes in use MountVolume tries to mark mount, throws an error because volume is no longer in actual state of world list. After this, kubelet isn't aware of the mount so doesn't try to unmount again. controller-manager tries to detach the volume, this fails because it is still mounted to the OS. EBS gets stuck indefinitely in busy state trying to detach.
- Loading branch information