remove duplicate volumes

[gaoning777]

This change is 

[gaoning777]

/cc IronPan

[Ark-kun]

This is not a fully correct solution.

Fully correct solution would be to dedup based on the volume itself, not the name. Then, if we have name conflicts, we should rename the volume+volumeMount (we have to do that when processing a task (ContainerOp), because later we won't know which volume is used by which volumeMount).

One solution to this is to have a method like task.add_mounted_volume(volume, volume_mount) which will automatically set the volume name to volume structure hash. That way different volumes will get different names and same volumes will get same names.

[gaoning777]

Maybe addvolume should be pipeline level thing because it gets converted to workflow level volumes in argo yaml.

[Ark-kun]

Maybe addvolume should be pipeline level thing because it gets converted to workflow level volumes in argo yaml.

It's complicated because in Kubernetes volumes are specified at the Pod level. Argo has many properties that can be set both at workflow and at template levels. Maybe some day volumes will also be available in template. argoproj/argo-workflows#1094

I find it convenient to be able to specify resource requirements for the tasks at the time of creation without splitting that between two places.

[gaoning777]

There is no pod level volumes, is there. AFAIK, there is pod level volume mount.
Correct me if I'm wrong.

[Ark-kun]

There is no pod level volumes, is there. AFAIK, there is pod level volume mount.

volumes are Pod-level
volumeMounts are Container-level

See https://kubernetes.io/docs/tasks/configure-pod-container/configure-volume-storage/

apiVersion: v1
kind: Pod
metadata:
  name: redis
spec:
  containers:
  - name: redis
    image: redis
    volumeMounts:
    - name: redis-storage
      mountPath: /data/redis
  volumes:
  - name: redis-storage
    emptyDir: {}

[gaoning777]

Discussed offline.
Make sense. Added a todo to add the support later.

[Ark-kun]

/lgtm
Issue: #494

[Ark-kun]

/lgtm

[Ark-kun]

/test kubeflow-pipeline-build-image

[gaoning777]

/test kubeflow-pipeline-e2e-test

[gaoning777]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaoning777

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [gaoning777]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gaoning777]

/test kubeflow-pipeline-build-image