Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trigger AWS runs via tower API #1160

Merged
merged 8 commits into from
Jul 9, 2021
Merged

Trigger AWS runs via tower API #1160

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1986d99
aws fulltest via tower
ggabernet Jul 5, 2021
1f9d1ab
aws test via tower
ggabernet Jul 5, 2021
ed88861
update changelog
ggabernet Jul 5, 2021
20efacc
remove unnecessary fields
ggabernet Jul 6, 2021
efc134d
remove unnecessary fields
ggabernet Jul 6, 2021
a6d26fe
Merge branch 'dev' of github.com:nf-core/tools into dev
ggabernet Jul 8, 2021
86e2db0
AWS test use tower action
ggabernet Jul 8, 2021
46a121c
fix AWS wf name
ggabernet Jul 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
### Template

* Move TODO item of `contains:` map in a YAML string [[#1082](https://github.com/nf-core/tools/issues/1082)]
* Trigger AWS tests via Tower API [[#1160](https://github.com/nf-core/tools/pull/1160)]

### General

Expand Down
42 changes: 22 additions & 20 deletions nf_core/pipeline-template/.github/workflows/awsfulltest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,9 @@ on:

{% raw %}
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }}
AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }}
AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }}
TOWER_WORKSPACE_ID: ${{ secrets.TOWER_WORKSPACE_ID }}
TOWER_BEARER_TOKEN: ${{ secrets.TOWER_BEARER_TOKEN }}
TOWER_COMPUTE_ENV: ${{ secrets.TOWER_COMPUTE_ENV }}
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
{% endraw %}

Expand All @@ -24,22 +22,26 @@ jobs:
if: github.repository == '{{ name }}'
runs-on: ubuntu-latest
steps:
- name: Setup Miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: 3.7
- name: Install awscli
run: conda install -c conda-forge awscli
- name: Start AWS batch job
- name: Launch workflow via tower
# TODO nf-core: You can customise AWS full pipeline tests as required
# Add full size test data (but still relatively small datasets for few samples)
# on the `test_full.config` test runs with only one set of parameters
# Then specify `-profile test_full` instead of `-profile test` on the AWS batch command
run: |
aws batch submit-job \
--region eu-west-1 \
--job-name nf-core-{{ short_name }} \
--job-queue $AWS_JOB_QUEUE \
--job-definition $AWS_JOB_DEFINITION \
--container-overrides '{"command": ["{{ name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'
curl -X POST "https://api.tower.nf/workflow/launch?workspaceId=${TOWER_WORKSPACE_ID}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${TOWER_BEARER_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"launch": {
"id" : "string",
"computeEnvId" : "'"${TOWER_COMPUTE_ENV}"'",
"pipeline" : "{{ name }}",
"workDir" : "s3://'"${AWS_S3_BUCKET}"'/work/{{ short_name }}/commit-'"${GITHUB_SHA}"'",
"paramsText" : "{\"outdir\":\"s3://'"${AWS_S3_BUCKET}"'/{{ short_name }}/results-'"${GITHUB_SHA}"'\"}",
"revision" : "'"${GITHUB_SHA}"'",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come some of these are quoted with "'" and some are not?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah it's because env variables don't get into single-quoted strings, but you need that to avoid having to escape every ". Ok, lesser of evils maybe!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I played with different options on how to do proper escaping and variable setting, and that's the only option that was working for me

"configProfiles" : ["test_full","aws_tower"],
"resume" : false,
"dateCreated" : "1970-01-01T00:00:00.000Z"
ggabernet marked this conversation as resolved.
Show resolved Hide resolved
}
}' \
--fail
53 changes: 28 additions & 25 deletions nf_core/pipeline-template/.github/workflows/awstest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,16 @@
name: nf-core AWS test
# This workflow is triggered on push to the master branch.
name: nf-core AWS tests
# This workflow is triggered only on manual dispatch.
# It can be additionally triggered manually with GitHub actions workflow dispatch.
# It runs the -profile 'test' on AWS batch.
# It runs the -profile 'test' on AWS batch

on:
workflow_dispatch:

{% raw %}
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }}
AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }}
AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }}
TOWER_WORKSPACE_ID: ${{ secrets.TOWER_WORKSPACE_ID }}
TOWER_BEARER_TOKEN: ${{ secrets.TOWER_BEARER_TOKEN }}
TOWER_COMPUTE_ENV: ${{ secrets.TOWER_COMPUTE_ENV }}
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
{% endraw %}

Expand All @@ -22,21 +20,26 @@ jobs:
if: github.repository == '{{ name }}'
runs-on: ubuntu-latest
steps:
- name: Setup Miniconda
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: 3.7
- name: Install awscli
run: conda install -c conda-forge awscli
- name: Start AWS batch job
# TODO nf-core: You can customise CI pipeline run tests as required
# For example: adding multiple test runs with different parameters
# Remember that you can parallelise this by using strategy.matrix
- name: Launch workflow via tower
# TODO nf-core: You can customise AWS full pipeline tests as required
# Add full size test data (but still relatively small datasets for few samples)
# on the `test_full.config` test runs with only one set of parameters
run: |
aws batch submit-job \
--region eu-west-1 \
--job-name nf-core-{{ short_name }} \
--job-queue $AWS_JOB_QUEUE \
--job-definition $AWS_JOB_DEFINITION \
--container-overrides '{"command": ["{{ name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'
curl -X POST "https://api.tower.nf/workflow/launch?workspaceId=${TOWER_WORKSPACE_ID}" \
-H "Accept: application/json" \
-H "Authorization: Bearer ${TOWER_BEARER_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"launch": {
"id" : "string",
"computeEnvId" : "'"${TOWER_COMPUTE_ENV}"'",
"pipeline" : "{{ name }}",
"workDir" : "s3://'"${AWS_S3_BUCKET}"'/work/{{ short_name }}/commit-'"${GITHUB_SHA}"'",
"paramsText" : "{\"outdir\":\"s3://'"${AWS_S3_BUCKET}"'/{{ short_name }}/results-'"${GITHUB_SHA}"'\"}",
"revision" : "'"${GITHUB_SHA}"'",
"configProfiles" : ["test","aws_tower"],
"resume" : false,
"dateCreated" : "1970-01-01T00:00:00.000Z"
}
}' \
--fail