-
Notifications
You must be signed in to change notification settings - Fork 616
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added ability to run linux workflows on large runners (#6273)
**Context:**
Currently the CI gets congested when large amounts of pull requests are
being updated simultaneously. This pull request gives PRs an escape
hatch and use large runners and use different queue to have CI jobs be
picked up.
**Description of the Change:**
This pull request adds two new features:
- Ability to add the `urgent` label to any pull request and switch it
over to large runners
- Automatic swap of rc branch to large runner
- This assumes the rc branch is of the format `vX.Y.Z-rcN`
Large runners, albeit slightly more powerful than standard runners, can
be spawned at a much higher volume than standard runners ... this is
because we pay per minute for these runners vs being included on our
GitHub Plan.
If a PR needs CI run without waiting for a runner, **add the `urgent`
label to the pull request**.
Important Note:
- This only affect jobs that run on `pull_request` and use `ubuntu`
runners.
- This change is already in-place in lightning and catalyst.
- PennyLaneAI/pennylane-lightning#774
- PennyLaneAI/catalyst#846
**Benefits:**
Ability to leverage large runner to have quick time for a runner to pick
up a job.
**Possible Drawbacks:**
Though we dictate the pool size of large runners, it is possible to
still saturate it.
**Related GitHub Issues:**
None.
[sc-73711](https://app.shortcut.com/xanaduai/story/73711/update-pennylane-ci-to-use-large-runner-group)
---------
Co-authored-by: Mudit Pandey <mudit.pandey@xanadu.ai>- Loading branch information
Showing
6 changed files
with
148 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| name: Determine Workflow Runner group | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| default_runner: | ||
| description: The runner type that is used by the calling workflow by default | ||
| required: true | ||
| type: string | ||
| large_runner_group_name: | ||
| description: The name of the runner group that should be used for large jobs | ||
| required: false | ||
| type: string | ||
| default: 'pl-4-core-large-runner' | ||
| force_large_runner: | ||
| description: Whether to force the use of a large runner | ||
| required: false | ||
| type: boolean | ||
| default: false | ||
| outputs: | ||
| runner_group: | ||
| description: The runner all subsequent jobs within the calling workflow should run on | ||
| value: >- | ||
| ${{ | ||
| jobs.determine_workflow_runner.outputs.runner_group || | ||
| ( | ||
| inputs.force_large_runner && | ||
| inputs.large_runner_group_name || | ||
| inputs.default_runner | ||
| ) | ||
| }} | ||
| jobs: | ||
| determine_workflow_runner: | ||
| runs-on: >- | ||
| ${{ | ||
| ( | ||
| github.event_name == 'pull_request' | ||
| && contains(github.event.pull_request.labels.*.name, 'urgent') | ||
| ) && inputs.large_runner_group_name || 'ubuntu-latest' | ||
| }} | ||
| outputs: | ||
| runner_group: ${{ steps.runner_group.outputs.runner_group }} | ||
|
|
||
| steps: | ||
| - name: Output Runner Group name | ||
| if: >- | ||
| ${{ | ||
| github.event_name == 'pull_request' | ||
| && startsWith(inputs.default_runner, 'ubuntu') | ||
| }} | ||
| id: runner_group | ||
| env: | ||
| # We are not able to use \d to check numeric values as bash does not allow them (not POSIX compliant) | ||
| RC_BRANCH_FORMAT_REGEX: v[0-9]+\.[0-9]+\.[0-9]+-rc[0-9]? | ||
| REPO_FULL_NAME: PennyLaneAI/pennylane | ||
| LARGE_RUNNER_GROUP_NAME: ${{ inputs.large_runner_group_name }} | ||
| run: | | ||
| if [[ '${{ contains(github.event.pull_request.labels.*.name, 'urgent') }}' == 'true' || ('${{ github.event.pull_request.head.repo.full_name }}' == "$REPO_FULL_NAME" && '${{ github.event.pull_request.base.ref }}' =~ $RC_BRANCH_FORMAT_REGEX) ]]; then | ||
| echo "This job requires usage of the large runner group '$LARGE_RUNNER_GROUP_NAME'"; | ||
| echo "runner_group=$LARGE_RUNNER_GROUP_NAME" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "This job does not require usage of large runners ..."; | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters