Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Marking the UI-metadata and Metrics artifacts as optional #1260

Merged
merged 3 commits into from
Apr 30, 2019
Merged

Marking the UI-metadata and Metrics artifacts as optional #1260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
621bd0c
Marking the UI-metadata and Metrics artifacts as optional
Ark-kun Apr 29, 2019
6e47807
Updated the compiler tests
Ark-kun Apr 29, 2019
13004f8
Updates one more test.
Ark-kun Apr 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions sdk/python/kfp/compiler/_op_to_template.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,10 @@ def _op_to_template(op: BaseOp):
for name, path in output_artifact_paths.items()
]

for output_artifact in output_artifacts:
if output_artifact['name'] in ['mlpipeline-ui-metadata', 'mlpipeline-metrics']:
output_artifact['optional'] = True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this field working with current argo image?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes =)
I've discovered that Argo is using slightly broken way to parse YAML, so unknown fields are ignored.
Let's see what the tests say.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All tests have passed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the sample test passing for the last commit. Actually, it seems that it didn't run for the last commit. Do you know why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It did. https://prow.k8s.io/pr-history/?org=kubeflow&repo=pipelines&pr=1260
But the unfinished tests are often shown at the rightmost column under the "Unknown" commit label.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sg. thanks.

/lgtm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// When I wrote the comment that the tests have passed, the sample test has already completed the main part and was deleting the cluster. I guess, when you've looked at it, the Prow still has not detected it as finished.

/approve


# workflow template
template = {
'name': processed_op.name,
Expand Down
2 changes: 2 additions & 0 deletions sdk/python/tests/compiler/compiler_tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ def test_operator_to_template(self):
'artifacts': [{
'name': 'mlpipeline-ui-metadata',
'path': '/mlpipeline-ui-metadata.json',
'optional': True,
's3': {
'accessKeySecret': {
'key': 'accesskey',
Expand All @@ -111,6 +112,7 @@ def test_operator_to_template(self):
},{
'name': 'mlpipeline-metrics',
'path': '/mlpipeline-metrics.json',
'optional': True,
's3': {
'accessKeySecret': {
'key': 'accesskey',
Expand Down
6 changes: 6 additions & 0 deletions sdk/python/tests/compiler/testdata/basic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -72,6 +73,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -102,6 +104,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -115,6 +118,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -157,6 +161,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -170,6 +175,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
8 changes: 8 additions & 0 deletions sdk/python/tests/compiler/testdata/coin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -105,6 +106,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -133,6 +135,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -146,6 +149,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -200,6 +204,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -213,6 +218,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -237,6 +243,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -250,6 +257,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
6 changes: 6 additions & 0 deletions sdk/python/tests/compiler/testdata/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -51,6 +52,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -113,6 +115,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -126,6 +129,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -158,6 +162,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -171,6 +176,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
4 changes: 4 additions & 0 deletions sdk/python/tests/compiler/testdata/default_value.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -71,6 +72,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -101,6 +103,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -114,6 +117,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
2 changes: 2 additions & 0 deletions sdk/python/tests/compiler/testdata/imagepullsecret.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -40,6 +41,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
4 changes: 4 additions & 0 deletions sdk/python/tests/compiler/testdata/immediate_value.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -50,6 +51,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -80,6 +82,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -93,6 +96,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
4 changes: 4 additions & 0 deletions sdk/python/tests/compiler/testdata/pipelineparams.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: "/mlpipeline-ui-metadata.json"
optional: true
s3:
endpoint: minio-service.kubeflow:9000
insecure: true
Expand All @@ -53,6 +54,7 @@ spec:
key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
- name: mlpipeline-metrics
path: "/mlpipeline-metrics.json"
optional: true
s3:
endpoint: minio-service.kubeflow:9000
insecure: true
Expand Down Expand Up @@ -91,6 +93,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: "/mlpipeline-ui-metadata.json"
optional: true
s3:
endpoint: minio-service.kubeflow:9000
insecure: true
Expand All @@ -104,6 +107,7 @@ spec:
key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
- name: mlpipeline-metrics
path: "/mlpipeline-metrics.json"
optional: true
s3:
endpoint: minio-service.kubeflow:9000
insecure: true
Expand Down
10 changes: 10 additions & 0 deletions sdk/python/tests/compiler/testdata/recursive_do_while.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -46,6 +47,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -74,6 +76,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -87,6 +90,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -115,6 +119,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -128,6 +133,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down Expand Up @@ -206,6 +212,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -219,6 +226,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -243,6 +251,7 @@ spec:
artifacts:
- name: mlpipeline-ui-metadata
path: /mlpipeline-ui-metadata.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand All @@ -256,6 +265,7 @@ spec:
name: mlpipeline-minio-artifact
- name: mlpipeline-metrics
path: /mlpipeline-metrics.json
optional: true
s3:
accessKeySecret:
key: accesskey
Expand Down
Loading