Support downloads from other workflows

[owenfarrell]

Fixes #501

[owenfarrell]

I've started putting together a unit test to validate these changes, but I'm running in to a bit of a hurdle with mocking Octokit. Before I get too far down that rabbit hole, I would love some feedback on the overall direction of these changes.

Thanks!
~Owen

[konradpabjan]

Hi @owenfarrell

Thanks for opening this PR! Sharing some information as I'm not entirely sure yet if we want to go down this approach as there have been some recent discussions going on internally.

Better continuous deployment is huge ask that we're starting to pick up work in the area. Artifacts is a big piece of the puzzle and the ability to easily download from other workflows is paramount.

The API you see documented here https://developer.github.com/v3/actions/artifacts/#download-an-artifact is basically an external API. You can use it to download artifacts from any workflow, however it's fairly limited as you can only get a zip. Networking resiliency is a big issue so we need the ability to retry downloads without starting the whole thing over.

The current APIs used in @actions/artifact are special internal APIs that can download files individually from an artifact with retries and everything however they are scoped only to a single workflow run.

There has been some internal discussions about possibly expanding the scope of the internal APIs to allow artifacts from other workflows and runs to be downloaded using the same api token (this is tricky as there are a lot of security implications). There are also considerations to expand the APIs available here: https://developer.github.com/v3/actions/artifacts/ but that poses it's own challenges.

Internally we're still coming up with a plan as there are a lot of architectural & engineering considerations that we have to take into account.

I wanted to offer at least some feedback and transparency regarding the direction of your PR. If there are any new developments I'll make sure to communicate them 😃

[owenfarrell]

Hi @konradpabjan,

tl;dr I truly appreciate the acknowledgment and response. And I recognize that this isn't a straightforward problem to solve given the blend of internally-, privately-, and publicly-available assets.

Better continuous deployment is huge ask that we're starting to pick up work in the area. Artifacts is a big piece of the puzzle and the ability to easily download from other workflows is paramount.

Your comment here hints at the concept, but I want to call this out explicitly - this isn't just a continuous deployment problem. Continuous deployment is certainly part of it, but the principle of building once applies to lots of aspects across the entire SDLC (e.g. analysis, integration, testing). And users' desire to adhere to that "build once" principle is demonstrated by the fact that this was the very first issue raised against the download-artifact action.

However, I think there is a willingness to take on a truly minimum viable product (or something "fairly limited" as you put it) since users are leveraging workarounds that use the public, external API already. I don't say that in an attempt to endorse my own PR, but rather to highlight the continued cost of delay. Without a de facto standard, we're going to continue to rely on workarounds... and those workarounds aren't going to adhere to the non-functional requirements you've identified (e.g. security concerns like leaking private artifacts).

I'm looking forward to seeing what you and your team come up with in order to help close that gap. Please let me know if I can provide any support.

Cheers,
~Owen

[jimis]

There has been some internal discussions about possibly expanding the scope of the internal APIs to allow artifacts from other workflows and runs to be downloaded using the same api token (this is tricky as there are a lot of security implications).

+1 on this one, is there a ticket I can follow?

The API you see documented here https://developer.github.com/v3/actions/artifacts/#download-an-artifact is basically an external API. You can use it to download artifacts from any workflow, however it's fairly limited as you can only get a zip.

@konradpabjan Indeed this is restrictive. In my case I have already compressed (tar+zstd) my artifact into one file. The API gives me the choice of downloading it, but with limitations:

• the response time is slow for big artifacts - is it zipping on the fly?
• I now need double decompression of my artifacts, and zip is far slower than zstd
• the extra zip is not needed for single-file artifacts.

I appreciate the team's work, please do let us know of updates as workarounds are getting finicky.

EDIT: I just notice this is a very specific pull request (that is definitely interesting as the start of solving the issues described). I don't want to distract the conversation, if there are github issues tracking the progress I would be happy to comment there, otherwise I'll open new ones, I guess under the upload-artifact project.

[ghost]

Created owenfarrell#2 to update this PR's branch with the latest code from main

[owenfarrell]

@vaughan-clare In order to keep the commit log clean(er), I rebased over the top of main. Thanks for the nudge!

[kbutto]

any updates on the support for this?

[StephenHodgson]

LGTM