Skip to content

Latest commit

 

History

History
644 lines (471 loc) · 12.7 KB

README.md

File metadata and controls

644 lines (471 loc) · 12.7 KB

Reusable CI pipelines for poetry projects

This is meant for personal use for my own projects.

Requirements

There are 4 different pipeline templates, one for testing, one for building docs, one for releasing the library to PyPI, and one for approving pull requests by certain authors automatically.

All templates require the project to use poetry. The testing pipeline requires tox and coverage, as well as Coverage results are sent to coveralls. The docs building pipeline requires mkdocs.

How to use

Here is a minimal pyproject.toml setup to get started:

# ...

[tool.poetry.group.test.dependencies]
pytest = "..."  # use latest version
coverage = "..."  # use latest version
tox = "..."  # use latest version
tox-gh-actions = "..."  # use latest version
# 'distutils' is not included in virtual environments from Python 3.12 onwards,
# so you might need to explcitly include it via setuptools.
setuptools = {version = "...", python = ">=3.12"}  # use latest version

# This is only needed for the docs CI
[tool.poetry.group.docs.dependencies]
mkdocs = "..."  # use latest version

[tool.coverage.run]
relative_files = true

[tool.coverage.report]
omit = [
    "tests/*",
    ".tox/*",
]

[tool.tox]
legacy_tox_ini = """
[tox]
envlist = py{310, 311, 312, 313}
isolated_build = true

[gh-actions]
python =
    3.10: py310
    3.11: py311
    3.12: py312
    3.13: py313

[testenv]
allowlist_externals =
    poetry
setenv =
    PYTHONPATH = {toxinidir}
commands =
    poetry install
    poetry run coverage run -m pytest
"""

[build-system]
requires = ["poetry-core>=1.9.0"]
build-backend = "poetry.core.masonry.api"

Testing pipeline

This pipeline uses a job stategy matrix to run tests in a number of python environments and operating systems in parallel. All dependencies are cached for each os and environment resulting from the strategy to ensure the CI runs fast when dependencies are not updated.

To set up the pipeline, add a yml file to ./.github/workflows/ with the following job configuration.

name: Tests

on:
  push:
    branches:
      - main
    paths:
      - "**.py"
      - "pyproject.toml"
      - "poetry.lock"
  pull_request:
  workflow_dispatch:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13

This job can take a number of inputs via the with-keyword.

python-version

Configure the pyhton versions the test will be run with. The tox environments used are configured with the [gh-actions] setting in pyproject.toml.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      python-version: '["3.10", "3.11", "3.12", "3.13"]'

os

Configure the operating systems the tests will be run with.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      os: '["ubuntu-latest", "macos-latest", "windows-latest"]'

poetry-version

Configure the poetry version used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      poetry-version: "2.0.0"

exclude

GitHub job stategy matrix exclusion pattern, in JSON form. Using yaml flow style, a list of dicts can be conveted into multiple exclusions if necessary.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      exclude: '[{"os": "none", "python-version": "none"}]'  # this ignores nothing

submodules

Should submodules be checked out with the repository?

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      submodules: false

fetch-depth

How many commit to fetch from the branch history? If set to 0, every commit is fetched.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      fetch-depth: 1

coveralls

Should coverage results be submitted to coveralls? Requires coveralls setup to already exist.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      coveralls: true

Docs building pipeline

This pipeline can be used to build and push the docs used for mkdocs from the docs/ directory into GitHub pages.

Note that mkdocs also requires a separate configuration file where the pages are set up. Here's a minimal example configuration.

site_name: {{ Site name here }}

nav:
  - Home: index.md  # ./docs/index.md

To set up the pipeline, add a yml file to ./.github/workflows/ with the following job configuration.

name: Docs

on:
  push:
    branches:
      - main
    paths:
      - "docs/**"
      - "mkdocs.yml"
  workflow_dispatch:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/docs.yml@v0.4.13

This job can take a number of inputs via the with-keyword.

poetry-version

Configure the poetry version used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/docs.yml@v0.4.13
    with:
      poetry-version: "2.0.0"

python-version

Configure the python version used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/docs.yml@v0.4.13
    with:
      python-version: "3.13"

os

Configure the operating system used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/docs.yml@v0.4.13
    with:
      os: "ubuntu-latest"

submodules

Should submodules be checked out with the repository?

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      submodules: false

fetch-depth

How many commit to fetch from the branch history? If set to 0, every commit is fetched.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      fetch-depth: 1

PyPI release pipeline

This pipeline can be used to build and release the library to PyPI with poetry using a PyPI token stored in the repository's actions secrets.

Note that the poetry version configuration needs to be updated and match the tag created for the release or this job will fail (can include v-prefix, e.g., v0.0.1).

To set up the pipeline, add a yml file to ./.github/workflows/ with the following job configuration. The pypi-token input is required.

name: Release

on:
  release:
    types:
      - released

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/release.yml@v0.4.13
    secrets:
      pypi-token: ${{ secrets.PYPI_API_TOKEN }}

Replace PYPI_API_TOKEN with the secret name of your choice

This job can take a number of inputs via the with-keyword.

poetry-version

Configure the poetry version used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/release.yml@v0.4.13
    with:
      poetry-version: "2.0.0"

python-version

Configure the python version used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/release.yml@v0.4.13
    with:
      python-version: "3.13"

os

Configure the operating system used in the pipeline.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/release.yml@v0.4.13
    with:
      os: "ubuntu-latest"

submodules

Should submodules be checked out with the repository?

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      submodules: false

fetch-depth

How many commit to fetch from the branch history? If set to 0, every commit is fetched.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      fetch-depth: 1

Pull request approval pipeline

This pipeline can be used to automatically approve pull request by some users. By default, it is set to approve pull requests by the dependabot and pre-commit.ci bots.

To set up the pipeline, add a yml file to ./.github/workflows/ with the following job configuration.

name: Auto approve PRs

on: 
  pull_request_target:

jobs:
  approve:
    permissions:
      pull-requests: write
      contents: write
    uses: MrThearMan/CI/.github/workflows/approve.yml@v0.4.13

This job can take a number of inputs via the with-keyword.

users

Configure the users whose pull requests can be automatically approved.

Default configuration:

jobs:
  approve:
    uses: MrThearMan/CI/.github/workflows/approve.yml@v0.4.13
    with:
      users: '["dependabot[bot]", "pre-commit-ci[bot]"]'

submodules

Should submodules be checked out with the repository?

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      submodules: false

fetch-depth

How many commit to fetch from the branch history? If set to 0, every commit is fetched.

Default configuration:

jobs:
  test:
    uses: MrThearMan/CI/.github/workflows/test.yml@v0.4.13
    with:
      fetch-depth: 1

Pipeline hooks

Some of the pipeline templates have hooks that can be defined to run additional setup and postprocessing steps if necessary. To enable them, simply add a yaml file to the appropriate directory in your project and write a composite action. Here is a template for one.

runs:
  using: composite
  
  steps:
    - name: "Setup"
      shell: bash
      run: ...

Here is a little trick you can do to pass the inputs from the testing job to the composite action when needed

inputs:
  python-version:
    default: ${{ matrix.python-version }}

For the testing pipeline, the hooks are:

  • Pre-test: Add the file .github/actions/pre-test/action.yml
  • Post-test: Add the file .github/actions/post-test/action.yml

For the release pipeline, the hooks are:

  • Pre-release: Add the file .github/actions/pre-release/action.yml
  • Post-release: Add the file .github/actions/post-release/action.yml

Extra actions

Poetry install action

Installs poetry to the current python version, e.g., if used after actions/setup-python, poetry is installed with that python version.

jobs:
  <foo>:
    steps:
      - ...
      - uses: MrThearMan/CI/.github/actions/poetry@v0.4.13
        with:
          os: "ubuntu-latest"
          poetry-version: "2.0.0"

However, actions/setup-python poetry caching cannot be used if poetry is not installed. In this case, a custom cache must be created:

jobs:
  <foo>:
    steps:
      - ...
      - name: "Load cached poetry environment"
        uses: actions/cache@v4
        with:
          path: .venv
          key: <unique-key-per-env>

Git changed filetypes

Not tested yet.

Can be used to check if certain filetypes were changed in a pull request.

jobs:
  <foo>:
    steps:
      - uses: MrThearMan/CI/.github/actions/get-changed-filetypes@v0.4.13
        id: changed
        with:
          filetypes: "py|yaml"
      - if: ${{ changed.changed-filetypes }}