Nextflow installation issues in GHA

[FriederikeHanssen]

Hey!,

i am running into various issues with the Nextflow installation in my Github action runs:
a couple of examples:

• https://github.com/nf-core/sarek/actions/runs/3456705070/jobs/5769631081:

3s
Run nf-core/setup-nextflow@v1
Error: Could not retrieve Nextflow release matching 21.10.3.
API rate limit exceeded for installation ID 1565271.
Error: Could not parse the download URL
Cannot read properties of undefined (reading 'filter')
Error: versionSpec parameter is required
Warning: Nextflow appears to have installed correctly, but an error was thrown while running it.

• or this one:
  https://github.com/nf-core/modules/actions/runs/3454606071/jobs/5766033679

Run wget -qO- get.nextflow.io | bash
Downloading nextflow dependencies. It may require a few seconds, please wait .. 
Downloading nextflow dependencies. It may require a few seconds, please wait .. 
Unable to initialize nextflow environment

Error: Process completed with exit code 1.

[FriederikeHanssen]

More of this: https://github.com/nf-core/sarek/actions/runs/3460660807

[MillironX]

The action uses the GitHub API to parse out version numbers. If you run 1000 pipelines in an hour, then you can overload your API token, which is what happened in the first example. You can bump this up to 5000 pipelines per hour (with caveats) if you provide a user token. See https://docs.github.com/en/rest/overview/resources-in-the-rest-api#requests-from-github-actions.

The second example doesn't use this action, so I don't know what happened there.

The third example links to a run that completed successfully for me.

---

The rate limit can be avoided entirely using the old method of wget -qO- get.nextflow.io | bash; sudo mv nextflow /usr/bin. The tradeoffs are

Manual install

• No rate limit
• Better testing (depends on the same installer as end users get)
• No easy way to determine latest release (i.e. stable vs edge, see Ignore NXF_EDGE if edge is older than latest release nextflow-io/nextflow#2467)
• "Worse" performance due to requiring a full VM on the backend rather than a single-use Node container (negligible in most use cases)
• Brittle runner interface

Action

• Rate limited
• Less testing
• Can parse version numbers easily and exactly
• "Better" Node performance
• Caches Nextflow install for faster runs
• Uses GitHub-provided methods for manipulating $PATH (i.e., less likely to break on runner updates)

[MillironX]

This might be able to be overcome using conditional requests. I will look into that more.

[MillironX]

Another idea (i.e. note-to-self):

graph TD;
  start(Start)-->octokit[Try to access Octokit API to get the list of Nextflow releases];
  octokit-->octokitsuccess{Was Octokit query successful?};
  octokitsuccess--Yes-->normal[Proceed as normal];
  octokitsuccess--No-->checkcache[Check Octokit cache for tag];
  checkcache-->cachesuccess{Does Octokit cache contain tag?};
  cachesuccess--Yes-->warn[Issue warning to user];
  warn-->install[Install Nextflow];
  normal-->addcache[Add Octokit results to Octokit cache];
  addcache-->install;
  install-->endproc(End);
  cachesuccess--No-->error[Throw exception];
  error-->enderror(Process failed)  

Loading

[nvnieuwk]

Any news on this? :)

[MillironX]

@nvnieuwk: reponse in Slack