PipelineRun volumeClaimTemplate creation can race with PVC deletion

[wlynch]

Expected Behavior

I should be able to run kubectl delete -f pipelinerun.yaml; kubectl apply -f pipelinerun.yaml repeatedly with a volumeClaimTemplate.

Actual Behavior

Talking with @sbwsg, we suspect that something along these lines is happening:

1. apply a pipelinerun that uses volumeclaimtemplate
2. delete that pipelinerun
3. the pvc created for the volumeClaimTemplate starts terminating
4. apply the pipelinerun again with the same name
5. the new pipelinerun (with the same name) tries to utilize the terminating pvc, hanging forever

When in the stuck state, kubectl describe shows:

Events:
  Type    Reason   Age                From     Message
  ----    ------   ----               ----     -------
  Normal  Started  83s                TaskRun  
  Normal  Pending  83s                TaskRun  Pending
  Normal  Pending  83s (x2 over 83s)  TaskRun  pod status "PodScheduled":"False"; message: "persistentvolumeclaim \"pvc-120ad2aa01\" is being deleted"

We suspect that this is a race condition where the new PipelineRun is waiting for the PVC to be deleted since it's in the terminating state, but the PVC isn't being deleted because it thinks the the new PipelineRun has a claim on it.

Deleting the pipelinerun and waiting lets the PVC finish deleting, at which point the new pipelinerun can be applied.

Steps to Reproduce the Problem

1. Use a volumeClaimTemplate in a PipelineRun (we discovered this with https://github.com/tektoncd/catalog/blob/54315c082b257ed2a27df3de51014aa75d135cad/task/kind/0.1/tests/run.yaml)
2. Delete then recreate the PipelineRun before the PVC can be deleted (kubectl delete ; kubectl apply)
3. New PipelineRun waits for PVC to be deleted

Additional Info

• Kubernetes version:

  Output of kubectl version:

  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.12-gke.1210", GitCommit:"199a41188dc0ca5d6d95b1cc7e8ba96e05f9dd0a", GitTreeState:"clean", BuildDate:"2021-02-05T18:03:16Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"} 
  

• Tekton Pipeline version:

  Client version: 0.15.0
  Pipeline version: v0.22.0
  

[wlynch]

/assign wlynch