feat(backend): stop compressing artifacts #1640

Ark-kun · 2019-07-18T00:11:27Z

DIsabled compressing artifacts to .tar.gz archives.
Fixes #1383

This change is

DIsabled compressing artifacts to .tar.gz archives. Fixes kubeflow#1383

Ark-kun · 2019-07-25T20:39:46Z

@rileyjbauer Can you please check that UI and visualizations work fine when the artifacts are not compressed?

Ark-kun · 2019-08-13T23:07:50Z

@rileyjbauer I find it surprising that e2e frontend tests do not fail change. Can you please take a look?
@gaoning777 I find it surprising that sample tests do not fail after this change. Can you please take a look?

Ark-kun · 2019-09-27T02:18:34Z

/retest

Ark-kun · 2019-09-27T21:06:45Z

I've changed the PR to only stop compressing the non-metadata artifacts. So this should be safe.

Bobgy · 2019-11-25T09:07:00Z

/retest

Bobgy · 2019-11-25T09:07:46Z

After you stopped compressing those artifacts, is there any code that could break?

I'm not very clear on where these artifacts are used.

Ark-kun · 2019-11-25T22:21:37Z

After you stopped compressing those artifacts, is there any code that could break?

I'm not very clear on where these artifacts are used.

The UX that shows the artifacts can break with high probability.

@IronPan WDYT? How does the backend API handler compressed artifact blobs? What about directories?

StefanoFioravanzo · 2020-02-19T15:13:54Z

Any update on this PR? It would be very useful!

Ark-kun · 2020-02-20T02:05:25Z

I think one problem right now is that some APIs like ReadArtifact probably only handle single files and will fail with directories.

stale · 2020-06-24T04:09:13Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Bobgy · 2020-06-24T04:30:03Z

This is sth I'd want to try sooner or later

stale · 2020-09-22T12:42:57Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale · 2020-09-30T03:36:03Z

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

k8s-ci-robot · 2020-10-30T20:57:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~backend/OWNERS~~ [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Bobgy · 2020-11-02T00:39:39Z

/retest

Bobgy

I think this is a better onboarding experience, also UX should already support this thanks to @eterna2's previous improvements.

I'd still want to make sure archive is an option, because text files can have a large compress rate, so it saves network IO. Can we add this as an option to API server config?

Bobgy · 2020-11-02T00:40:35Z

backend/src/apiserver/resource/resource_manager.go

@@ -218,6 +218,8 @@ func (r *ResourceManager) CreateRun(apiRun *api.Run) (*model.RunDetail, error) {
 		for artIdx, artifact := range template.Outputs.Artifacts {
 			if artifact.Name == "mlpipeline-ui-metadata" || artifact.Name == "mlpipeline-metrics" {
 				workflow.Workflow.Spec.Templates[templateIdx].Outputs.Artifacts[artIdx].Optional = true
+			} else {
+				artifact.Archive = &workflowapi.ArchiveStrategy{None: &workflowapi.NoneStrategy{}}


nit: why do we add this in an else clause here?
I think we can add it to all.

Another issue is that, there should be some similar code in CreateJob method, you should add this part there too

nit: why do we add this in an else clause here?

I'm not sure that Frontend and Backend Metrics are prepared for uncompressed artifacts.

Frontend server should support both compressed and uncompressed artifacts.

pipelines/frontend/server/minio-helper.ts

Line 134 in 4e5d724

export async function getObjectStream({

stale · 2021-02-08T03:58:38Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

eterna2 · 2021-05-29T02:42:12Z

This PR should also resolve #5558

Bobgy · 2021-05-29T03:28:22Z

@eterna2 in v2 compatible mode, artifacts are already not compressed by default

eterna2 · 2021-06-01T12:05:16Z

Nice! I was reading thru the codes for v2. Am very excited with all the new stuff coming up!

stale · 2022-03-02T10:04:58Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Checks whether env var awsAnonymousCredential true to use S3 resource config anonymous addresses issue kubeflow#1581 Signed-off-by: Malini Bhandaru <mbhandaru@vmware.com>

rimolive · 2024-03-24T14:48:52Z

Closing this PR. No activity for more than a year.

/close

google-oss-prow · 2024-03-24T14:48:55Z

@rimolive: Closed this PR.

In response to this:

Closing this PR. No activity for more than a year.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

SDK - Stop compressing the artifacts

c159ed1

DIsabled compressing artifacts to .tar.gz archives. Fixes kubeflow#1383

k8s-ci-robot added the size/XS label Jul 18, 2019

k8s-ci-robot requested review from gaoning777 and hongye-sun July 18, 2019 00:11

Ark-kun added 2 commits July 17, 2019 18:42

Setting the archive strategy in the backend.

738a896

Remove compression option in SDK

b83e4cb

Ark-kun assigned IronPan and rileyjbauer Jul 26, 2019

Ark-kun requested a review from rileyjbauer August 12, 2019 23:26

Ark-kun requested a review from numerology August 13, 2019 23:08

Leave the mlmetadata- artifacts compressed for now

8fc7900

k8s-ci-robot unassigned rileyjbauer Sep 26, 2019

Ark-kun assigned numerology Sep 27, 2019

Ark-kun assigned Bobgy Nov 13, 2019

stale bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Jun 24, 2020

stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Jun 24, 2020

stale bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Sep 22, 2020

stale bot closed this Sep 30, 2020

Ark-kun reopened this Oct 30, 2020

stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Oct 30, 2020

k8s-ci-robot added the approved label Oct 30, 2020

Ark-kun mentioned this pull request Oct 30, 2020

feat(frontend): UX change to support downloading directory artifacts. Fixes #3667 #4696

Merged

2 tasks

Bobgy reviewed Nov 2, 2020

View reviewed changes

Bobgy changed the title ~~SDK - Stop compressing the artifacts~~ feat(backend): stop compressing artifacts Nov 2, 2020

stale bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Feb 8, 2021

chensun force-pushed the master branch 2 times, most recently from 7542f0e to 44d22a6 Compare February 12, 2021 09:23

stale bot removed the lifecycle/stale The issue / pull request is stale, any activities remove this label. label May 28, 2021

eterna2 mentioned this pull request May 29, 2021

[feature] Configurable KFP Artifact types while storing to Minio #5558

Closed

stale bot added the lifecycle/stale The issue / pull request is stale, any activities remove this label. label Mar 2, 2022

google-oss-prow bot closed this Mar 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(backend): stop compressing artifacts #1640

feat(backend): stop compressing artifacts #1640

Ark-kun commented Jul 18, 2019 •

edited by jlewi

Loading

Ark-kun commented Jul 25, 2019

Ark-kun commented Aug 13, 2019

Ark-kun commented Sep 27, 2019

Ark-kun commented Sep 27, 2019

Bobgy commented Nov 25, 2019

Bobgy commented Nov 25, 2019

Ark-kun commented Nov 25, 2019

StefanoFioravanzo commented Feb 19, 2020

Ark-kun commented Feb 20, 2020

stale bot commented Jun 24, 2020

Bobgy commented Jun 24, 2020

stale bot commented Sep 22, 2020

stale bot commented Sep 30, 2020

k8s-ci-robot commented Oct 30, 2020

Bobgy commented Nov 2, 2020

Bobgy left a comment

Bobgy Nov 2, 2020

Bobgy Nov 2, 2020

Ark-kun May 28, 2021

eterna2 May 29, 2021

stale bot commented Feb 8, 2021

eterna2 commented May 29, 2021

Bobgy commented May 29, 2021

eterna2 commented Jun 1, 2021

stale bot commented Mar 2, 2022

rimolive commented Mar 24, 2024

google-oss-prow bot commented Mar 24, 2024

feat(backend): stop compressing artifacts #1640

feat(backend): stop compressing artifacts #1640

Conversation

Ark-kun commented Jul 18, 2019 • edited by jlewi Loading

Ark-kun commented Jul 25, 2019

Ark-kun commented Aug 13, 2019

Ark-kun commented Sep 27, 2019

Ark-kun commented Sep 27, 2019

Bobgy commented Nov 25, 2019

Bobgy commented Nov 25, 2019

Ark-kun commented Nov 25, 2019

StefanoFioravanzo commented Feb 19, 2020

Ark-kun commented Feb 20, 2020

stale bot commented Jun 24, 2020

Bobgy commented Jun 24, 2020

stale bot commented Sep 22, 2020

stale bot commented Sep 30, 2020

k8s-ci-robot commented Oct 30, 2020

Bobgy commented Nov 2, 2020

Bobgy left a comment

Choose a reason for hiding this comment

Bobgy Nov 2, 2020

Choose a reason for hiding this comment

Bobgy Nov 2, 2020

Choose a reason for hiding this comment

Ark-kun May 28, 2021

Choose a reason for hiding this comment

eterna2 May 29, 2021

Choose a reason for hiding this comment

stale bot commented Feb 8, 2021

eterna2 commented May 29, 2021

Bobgy commented May 29, 2021

eterna2 commented Jun 1, 2021

stale bot commented Mar 2, 2022

rimolive commented Mar 24, 2024

google-oss-prow bot commented Mar 24, 2024

Ark-kun commented Jul 18, 2019 •

edited by jlewi

Loading