Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revive CI on Lumi #334

Closed
wants to merge 37 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
871815d
ci: enable CI on Lumi
skosukhin Jan 7, 2025
0711d4e
ci: drop commented code
skosukhin Jan 7, 2025
103e707
ci: levante: hardcode non-sensitive values
skosukhin Jan 7, 2025
3f9420d
ci: lumi: update and hardcode non-sensitive values
skosukhin Jan 7, 2025
e75c9d8
ci: lumi: drop lines of unknown origin
skosukhin Jan 7, 2025
68f76bd
ci: avoid string conversion
skosukhin Jan 7, 2025
e9d3052
ci: gitlab: enable running via web UI at DKRZ
skosukhin Jan 8, 2025
d8c0408
ci: gitlab: prepare Levante config to be merged with the Lumi one
skosukhin Jan 8, 2025
3110508
ci: gitlab: prepare Lumi config to be merged with the Levante one
skosukhin Jan 8, 2025
d9e4e1b
ci: gitlab: merge Levante and Lumi configs
skosukhin Jan 8, 2025
a65a88d
ci: gitlab: set id_tokens for Lumi jobs
skosukhin Jan 8, 2025
fa6f6f7
ci: gitlab: formatting
skosukhin Jan 8, 2025
de9bd98
ci: gitlab: rename hidden jobs
skosukhin Jan 8, 2025
6e5fe89
ci: gitlab: drop trivial hidden jobs
skosukhin Jan 8, 2025
0578809
ci: gitlab: rename more hidden jobs
skosukhin Jan 8, 2025
b6bc982
ci: gitlab: drop and combine hidden jobs
skosukhin Jan 8, 2025
c45822d
ci: gitlab: combine scheduler hidden jobs
skosukhin Jan 8, 2025
2b34734
ci: gitlab: reorder jobs
skosukhin Jan 8, 2025
9687892
ci: gitlab: drop the python setup job
skosukhin Jan 8, 2025
cff9067
ci: gitlab: refactore scheduling parameters
skosukhin Jan 8, 2025
4935502
ci: gitlab: drop excessive variable
skosukhin Jan 8, 2025
cea2280
ci: gitlab: increase time limit for all jobs
skosukhin Jan 8, 2025
914f5ef
ci: gitlab: adjust git depth on Levante
skosukhin Jan 8, 2025
86e0f1c
Allow for longer arguments in the test programs
skosukhin Jan 8, 2025
5bcf3bd
ci: gitlab: suppress output from lmod on Lumi
skosukhin Jan 8, 2025
ef04d13
ci: gitlab: switch to variable recognized by CMake
skosukhin Jan 8, 2025
0373ce7
ci: gitlab: polish the Python environment caching
skosukhin Jan 8, 2025
6d2c4a7
ci: gitlab: set DP by default
skosukhin Jan 8, 2025
6de8dbc
ci: gitlab: set accel by default
skosukhin Jan 8, 2025
500f04a
ci: gitlab: request and use resources consistently
skosukhin Jan 8, 2025
9e1f070
ci: gitlab: comment on why the variable is empty
skosukhin Jan 8, 2025
161a64e
ci: gitlab: drop seemingly redundant parameters
skosukhin Jan 8, 2025
9c2e724
ci: update gitlab-ci configuration
skosukhin Jan 8, 2025
81f3b4d
ci: gitlab: nagfor needs gcc
skosukhin Jan 8, 2025
f969482
ci: gitlab: better name for the configuration file
skosukhin Jan 8, 2025
ee72e20
Temporarily revert "ci: gitlab: better name for the configuration file"
skosukhin Jan 14, 2025
9b84fe5
ci: gitlab: sign the commit
skosukhin Jan 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 38 additions & 150 deletions .github/workflows/gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ defaults:

jobs:
#
# Deferred GitLab pipelines on Levante at DKRZ (see .gitlab/levante.yml):
# Deferred GitLab pipelines at gitlab.dkrz.de (see .gitlab/levante.yml):
#
levante-init:
trigger:
if: |
github.repository_owner == 'earth-system-radiation' &&
( github.event_name != 'pull_request' ||
Expand All @@ -29,174 +29,62 @@ jobs:
ref-name: ${{ steps.g-push-rev.outputs.ref-name }}
pipeline-id: ${{ steps.gl-trigger-pipeline.outputs.pipeline-id }}
steps:
#
# Check out GitHub repository
#
- name: Check out GitHub repository
uses: actions/checkout@v4
with:
fetch-depth: 0
#
# Push to GitLab repository
#
- name: Push to GitLab repository
id: g-push-rev
uses: "skosukhin/git-ci-hub-lab/g-push-rev@v1"
with:
remote-url: ${{ vars.DKRZ_GITLAB_SERVER }}/${{ vars.DKRZ_GITLAB_PROJECT }}.git
password: ${{ secrets.DKRZ_GITLAB_TOKEN }}
ref-type: tag
ref-message: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
force-push: true
#
# Trigger GitLab CI/CD Pipeline
#
- name: Trigger GitLab CI/CD Pipeline
id: gl-trigger-pipeline
uses: "skosukhin/git-ci-hub-lab/gl-trigger-pipeline@v1"
with:
server-url: ${{ vars.DKRZ_GITLAB_SERVER }}
project-name: ${{ vars.DKRZ_GITLAB_PROJECT }}
token: ${{ secrets.DKRZ_GITLAB_TRIGGER_TOKEN }}
ref-name: ${{ steps.g-push-rev.outputs.ref-name }}
expected-sha: ${{ github.sha }}
levante:
- name: Check out GitHub repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Push to GitLab repository
id: g-push-rev
uses: "skosukhin/git-ci-hub-lab/g-push-rev@v1"
with:
remote-url: ${{ vars.DKRZ_GITLAB_SERVER }}/${{ vars.DKRZ_GITLAB_PROJECT }}.git
password: ${{ secrets.DKRZ_GITLAB_TOKEN }}
rev-id: ${{ github.sha }}
rev-signing-format: ssh
rev-signing-key: ${{ secrets.GITLAB_SIGNING_KEY }}
ref-type: branch
force-push: true
- name: Trigger GitLab CI/CD Pipeline
id: gl-trigger-pipeline
uses: "skosukhin/git-ci-hub-lab/gl-trigger-pipeline@v1"
with:
server-url: ${{ vars.DKRZ_GITLAB_SERVER }}
project-name: ${{ vars.DKRZ_GITLAB_PROJECT }}
token: ${{ secrets.DKRZ_GITLAB_TRIGGER_TOKEN }}
ref-name: ${{ steps.g-push-rev.outputs.ref-name }}
expected-sha: ${{ steps.g-push-rev.outputs.ref-commit }}
attach:
runs-on: ubuntu-latest
needs: levante-init
needs: trigger
strategy:
fail-fast: false
matrix:
config-name:
- cce-gpu-openacc-DP
- cce-gpu-openacc-SP
- nag-cpu-accel-DP
- nag-cpu-default-SP
- nvhpc-gpu-openacc-DP
- nvhpc-gpu-openacc-SP
#- nag-cpu-default-DP
- nag-cpu-default-SP
- nag-cpu-accel-DP
#- nag-cpu-accel-SP
steps:
#
# Build, run and check (fetch the log)
#
- name: Build, run and check (fetch the log)
uses: "skosukhin/git-ci-hub-lab/gl-attach-job@v1"
with:
server-url: ${{ vars.DKRZ_GITLAB_SERVER }}
project-name: ${{ vars.DKRZ_GITLAB_PROJECT }}
token: ${{ secrets.DKRZ_GITLAB_TOKEN }}
pipeline-id: ${{ needs.levante-init.outputs.pipeline-id }}
pipeline-id: ${{ needs.trigger.outputs.pipeline-id }}
job-name: ${{ matrix.config-name }}
levante-cleanup:
cleanup:
runs-on: ubuntu-latest
needs: [levante-init, levante]
if: always() && needs.levante-init.result != 'skipped'
needs: [trigger, attach]
if: always() && needs.trigger.result != 'skipped'
continue-on-error: true
steps:
- uses: "skosukhin/git-ci-hub-lab/g-delete-ref@v1"
with:
remote-url: ${{ vars.DKRZ_GITLAB_SERVER }}/${{ vars.DKRZ_GITLAB_PROJECT }}.git
password: ${{ secrets.DKRZ_GITLAB_TOKEN }}
ref-type: ${{ needs.levante-init.outputs.ref-type }}
ref-name: ${{ needs.levante-init.outputs.ref-name }}
force: true
#
# Deferred GitLab pipelines on Lumi at CSC (see .gitlab/lumi.yml):
#
lumi-init:
if: |
false &&
github.repository_owner == 'earth-system-radiation' &&
( github.event_name != 'pull_request' ||
( github.event.pull_request.head.repo.owner.login == github.repository_owner &&
github.event.pull_request.user.login != 'dependabot[bot]' ))
runs-on: ubuntu-latest
outputs:
ref-type: ${{ steps.g-push-rev.outputs.ref-type }}
ref-name: ${{ steps.g-push-rev.outputs.ref-name }}
pipeline-id: ${{ steps.gl-create-pipeline.outputs.pipeline-id }}
steps:
#
# Check out GitHub repository
#
- name: Check out GitHub repository
uses: actions/checkout@v4
with:
fetch-depth: 0
#
# Push to GitLab repository
#
- name: Push to GitLab repository
id: g-push-rev
uses: "skosukhin/git-ci-hub-lab/g-push-rev@v1"
with:
remote-url: ${{ vars.GITLAB_SERVER }}/${{ vars.GITLAB_PROJECT }}.git
password: ${{ secrets.GITLAB_TOKEN }}
rev-id: ${{ github.sha }}
rev-signing-format: ssh
rev-signing-key: ${{ secrets.GITLAB_SIGNING_KEY }}
ref-type: branch
force-push: true
#
# Create GitLab CI/CD Pipeline
#
- name: Create GitLab CI/CD Pipeline
id: gl-create-pipeline
uses: "skosukhin/git-ci-hub-lab/gl-create-pipeline@v1"
with:
server-url: ${{ vars.GITLAB_SERVER }}
project-name: ${{ vars.GITLAB_PROJECT }}
token: ${{ secrets.GITLAB_TOKEN }}
ref-name: ${{ steps.g-push-rev.outputs.ref-name }}
expected-sha: ${{ steps.g-push-rev.outputs.ref-commit }}
#
# Set up Python virtual environment (fetch the log)
#
- name: Set up Python virtual environment (fetch the log)
uses: "skosukhin/git-ci-hub-lab/gl-attach-job@v1"
with:
server-url: ${{ vars.GITLAB_SERVER }}
project-name: ${{ vars.GITLAB_PROJECT }}
token: ${{ secrets.GITLAB_TOKEN }}
pipeline-id: ${{ steps.gl-create-pipeline.outputs.pipeline-id }}
job-name: setup-python
lumi:
runs-on: ubuntu-latest
needs: lumi-init
strategy:
fail-fast: false
matrix:
config-name:
- cce-gpu-openacc-DP
- cce-gpu-openacc-SP
steps:
#
# Build, run and check (fetch the log)
#
- name: Build, run and check (fetch the log)
uses: "skosukhin/git-ci-hub-lab/gl-attach-job@v1"
with:
server-url: ${{ vars.GITLAB_SERVER }}
project-name: ${{ vars.GITLAB_PROJECT }}
token: ${{ secrets.GITLAB_TOKEN }}
pipeline-id: ${{ needs.lumi-init.outputs.pipeline-id }}
job-name: ${{ matrix.config-name }}
lumi-cleanup:
runs-on: ubuntu-latest
needs: [lumi-init, lumi]
if: always() && needs.lumi-init.result != 'skipped'
continue-on-error: true
steps:
- uses: "skosukhin/git-ci-hub-lab/gl-cancel-pipeline@v1"
with:
server-url: ${{ vars.GITLAB_SERVER }}
project-name: ${{ vars.GITLAB_PROJECT }}
token: ${{ secrets.GITLAB_TOKEN }}
pipeline-id: ${{ needs.lumi-init.outputs.pipeline-id }}
force: true
- uses: "skosukhin/git-ci-hub-lab/gl-delete-ref@v1"
with:
server-url: ${{ vars.GITLAB_SERVER }}
project-name: ${{ vars.GITLAB_PROJECT }}
token: ${{ secrets.GITLAB_TOKEN }}
ref-type: ${{ needs.lumi-init.outputs.ref-type }}
ref-name: ${{ needs.lumi-init.outputs.ref-name }}
ref-type: ${{ needs.trigger.outputs.ref-type }}
ref-name: ${{ needs.trigger.outputs.ref-name }}
force: true
29 changes: 0 additions & 29 deletions .gitlab/common.yml

This file was deleted.

Loading
Loading