Set TTL for argo workflow

[IronPan]

Argo supports ttl https://github.com/argoproj/argo/blob/master/examples/gc-ttl.yaml
When DSL compiles pipeline to argo workflow, we should set this so that it can be GCed after finish. The TTL should leave enough time for Persistence agent to persist the data.

[gaoning777]

Are you suggesting a default ttl for all pipelines?
BTW, the DSL already exposes the ttl conf:

pipelines/sdk/python/kfp/dsl/_pipeline.py

Line 84 in 3dc73ba

def set_ttl_seconds_after_finished(self, seconds: int):

[gaoning777]

Is the purpose to reduce the storage size? in which case we can set the default value to a large number to prevent large pipelines from failing due to small ttl.

[paveldournov]

The TTL seems to kick off after the workload completes, so we don't need to worry about long running workflows. I think we should consider setting a low TTL by default to GC the resources promptly after the run is recorded in metadata. It would probably affect the logging UX experience and we will need to switch over to showing the persisted logs (stackdriver, etc) instead by default.

[IronPan]

With change #1802 this issue can be closed.

[IronPan]

Agreed with @paveldournov we need find a better way to persist logs with aggressive GC.
For GCP stackdriver is ok.
For non-GCP, we need a better story.

[IronPan]

File an issue for on-prem logs #1803
/close

[k8s-ci-robot]

@IronPan: Closing this issue.

In response to this:

File an issue for on-prem logs #1803
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.