Grant pipeline-runner k8s service account admin permission

[IronPan]

Currently if user's pipeline wants to launch a tf-job, job or other K8s resources, it will fail with following error

level=error msg="handle object: patching object from cluster: merging object with existing state: jobs.batch \"search-index-creator\" is forbidden: User \"system:serviceaccount:kubeflow:pipeline-runner\" cannot get jobs.batch in the namespace \"kubeflow\""

We need to grant pipeline-runner the admin permission so it can launch anything user specifies.

[IronPan]

To quickly unblock, you can run following command to grant the pipeline runner with enough permission

kubectl create clusterrolebinding pipelinerunnerbinding \
  --clusterrole=cluster-admin \
  --serviceaccount=kubeflow:pipeline-runner

[vicaire]

Should it really be admin? Is there a way we could grant lesser privileges?

[IronPan]

We can also add permission incrementally. but it's hard to predict what resource user want to launch from a pipeline. it could be, for example, a customized resource.

admin permission is fine as long as it's a single user scenario. for multi user case, i would hope to see how kubeflow sets things up and follow the same convention.

[IronPan]

Update - The current set of permission is working fine, after adding admin permission for all Kubeflow resources and PV/PVC
https://github.com/kubeflow/pipelines/pull/1576/files