feat: add version constraints, poetry, and poetry.lock file, and us…

…e `nox-poetry` and py3.12 in CI (#202)

* add version constraints fro2b5295fc1fe6bd90863571d3b74fb6ef70d4b266

* try to convert setuptools pyproject to poetry pyproject

* add pybedtools and dev dependencies

* try to lock versions

* determine min working versions

* lock at min versions but allow for anthing >=

* move docs and tests into separate groups

* use poetry indev install instructions

* require importlib-metadata for versions of py<3.7

* add poetry to release workflow

* copy github action from haptools

* use poetry in readthedocs config

* add nox and nox-poetry to pyproject.toml

* move nox and nox-poetry to global env

* simplify installation directions

* try running nox once

* move pytest cli args into pyproject

* try moving coverage config to pyproject.toml

* finish running nox on py3.8
move pytest-cov command line args to noxfile so they don't always run

* ensure noxfile is properly ignored from pytest

* describe how to run python version tests locally

* add py 3.9+ to the set of py versions that are incompatible with our lock file

* copy tests.yml github actions from haptools

* add coverage testing as a step in the workflow

* exclude files from sdist properly

* remove --with params from install instructions

* add checks for dist size and refactor coverage check

* remove classifiers which are handled automatically by poetry

* doc: describe how to manage dependencies with poetry

* execute coverage test with locked dependency versions

* use tests session in coverage CI check

* clarify rule that all dependencies should be locked at min versions

* clarify installation instructions in README

* check that bcftools is installed

* ensure newer bcftools is installed

* try installing bcftools before pytest

* do not specify certain channel priority

* relax version constraints

* use versions at time of first commit to repo, which failed

* bump versions to get tests to pass

* add poetry-specific items to PR checklist

* build: release pipeline - lock poetry at the version used in dev-env.yml

* ci: add test for python 3.12

* remove pybedtools as a dependency in pyproject.toml

* revert to older pyproject and rtd_conda to resolve conflicts

* replay edits to pyproject and rtd_conda after updating branch

* ensure numpy 2.0 doesn't break anything for now

* add file sizes test too

* specify that new files must be smaller than 0.5 MB in PR template

* list files in sdist within CI

* ensure tests are run with installed version

.devcontainer/devcontainer.json → .devcontainer.json

@@ -18,7 +18,7 @@
 	// "forwardPorts": [],
 
 	// Use 'postCreateCommand' to run commands after the container is created.
-	"postCreateCommand": "mamba env create -y -n trtools --file .devcontainer/environment.yml && conda run -n trtools pip install --user -e .",
+	"postCreateCommand": "mamba env create -n trtools -f dev-env.yml && conda run -n trtools poetry install",
 
 	// Configure tool-specific properties.
 	"customizations": {
@@ -29,7 +29,7 @@
 				"python.defaultInterpreterPath": "/opt/conda/envs/trtools/bin/python",
 				"python.terminal.activateEnvironment": true,
 				"python.terminal.activateEnvInCurrentTerminal": true,
-				"python.venvFolders": ["/opt/conda/envs"],
+				"python.venvFolders": ["/home/vscode/.cache/pypoetry/virtualenvs"],
 				"terminal.integrated.environmentChangesRelaunch": true,
 				"terminal.integrated.hideOnStartup": "always"
 			}

.github/pull_request_template.md

@@ -5,7 +5,8 @@
 * [ ] At the top of the PR, I've [listed any open issues that this PR will resolve](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword). For example, "resolves #0" if this PR resolves issue #0
 - [ ] I've explained my changes in a manner that will make it possible for both users and maintainers of TRTools to understand them
 * [ ] I've added tests for any new functionality. Or, if this PR fixes a bug, I've added test(s) that replicate it
+* [ ] All directories with large test files are listed in [the "exclude" section](https://python-poetry.org/docs/pyproject/#include-and-exclude) of our pyproject.toml so that they do not appear in our PyPI distribution. All new files are also smaller than 0.5 MB.
 * [ ] I've updated the relevant REAMDEs with any new usage information and checked that the newly built documentation is formatted properly
 * [ ] All functions, modules, classes etc. still conform to [numpy docstring standards](https://numpydoc.readthedocs.io/en/latest/format.html)
-* [ ] (if applicable) I've updated the pyproject.toml file with any changes I've made to TRTools's dependencies
+* [ ] (if applicable) I've updated the pyproject.toml file with any changes I've made to TRTools's dependencies, and I've run `poetry lock --no-update` to ensure the lock file stays up to date and that our dependencies are locked to their minimum versions
 * [ ] In the body of this PR, I've included a short address to the reviewer highlighting one or two items that might deserve their focus

.github/workflows/release.yml

@@ -31,13 +31,18 @@ jobs:
         if: ${{ steps.release.outputs.release_created }}
         run: |
           pip install --upgrade pip
-          pip install build
           pip --version
 
+      - name: Install Poetry
+        if: ${{ steps.release.outputs.release_created }}
+        run: |
+          pip install 'poetry==1.7.1'
+          poetry --version
+
       - name: Build package
         if: ${{ steps.release.outputs.release_created }}
         run: |
-          python -m build
+          poetry build --ansi -n
 
       - name: Publish package on PyPI
         if: ${{ steps.release.outputs.release_created }}

.github/workflows/tests.yml

@@ -8,45 +8,101 @@ concurrency:
 
 jobs:
   tests:
-    name: Test with py${{ matrix.python-version }}
-    runs-on: ubuntu-latest
+    name: ${{ matrix.session }} / ${{ matrix.python }} / ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
     strategy:
+      fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        include:
+          - { python: "3.7", os: "ubuntu-latest", session: "tests" }
+          - { python: "3.8", os: "ubuntu-latest", session: "tests" }
+          - { python: "3.9", os: "ubuntu-latest", session: "tests" }
+          - { python: "3.10", os: "ubuntu-latest", session: "tests" }
+          - { python: "3.11", os: "ubuntu-latest", session: "tests" }
+          - { python: "3.12", os: "ubuntu-latest", session: "tests" }
+          # - { python: "3.11", os: "windows-latest", session: "tests" }
+          - { python: "3.8", os: "macos-latest", session: "tests" }
+          - { python: "3.8", os: "ubuntu-latest", session: "size" }
+          - { python: "3.8", os: "ubuntu-latest", session: "coverage" }
+
+    env:
+      NOXSESSION: ${{ matrix.session }}
+      FORCE_COLOR: "1"
+      PRE_COMMIT_COLOR: "always"
 
     steps:
-      - uses: actions/checkout@v4
-      - name: Set up mamba
-        uses: conda-incubator/setup-miniconda@v2
+      - name: Check out the repository
+        uses: actions/checkout@v4
+
+      - name: Setup Mambaforge
+        uses: conda-incubator/setup-miniconda@v3
         with:
-          activate-environment: test
+          activate-environment: trtools
+          miniforge-variant: Mambaforge
           auto-activate-base: false
-          python-version: ${{ matrix.python-version }}
           miniforge-version: latest
-          miniforge-variant: Mambaforge
           use-mamba: true
-          mamba-version: "*"
-      - name: Install test dependencies
+
+      - name: Get Date
+        id: get-date
+        run: echo "today=$(/bin/date -u '+%Y%m%d')" >> $GITHUB_OUTPUT
+        shell: bash
+
+      - name: Cache Conda env
+        uses: actions/cache@v4
+        with:
+          path: ${{ env.CONDA }}/envs
+          key:
+            conda-${{ runner.os }}--${{ runner.arch }}--${{ steps.get-date.outputs.today }}-${{ hashFiles('dev-env.yml') }}-${{ env.CACHE_NUMBER }}
+        env:
+          # Increase this value to reset cache if dev-env.yml has not changed
+          CACHE_NUMBER: 0
+        id: cache
+
+      - name: Install dev environment
+        run:
+          mamba env update -n trtools -f dev-env.yml
+        if: steps.cache.outputs.cache-hit != 'true'
+
+      - name: Try to build trtools
         shell: bash -el {0}
         run: |
-          mamba install -c conda-forge -c bioconda bcftools pip pytest pytest-cov art
-      - name: Upgrade pip and install our package
-        shell: bash -el {0}
+          poetry build --no-ansi
+
+      - name: Check distribution size
+        if: matrix.session == 'size'
         run: |
-          python -m pip install --upgrade pip setuptools wheel
-          pip install -e .
-      - name: Test with pytest
+          du -csh dist/*
+          tar -ztvf dist/*.tar.gz
+          # check that the generated dist/ directory does not exceed 0.3 MB
+          # if this check fails, it's because you forgot to list large files in the "exclude" section of our pyproject.toml
+          # https://python-poetry.org/docs/pyproject/#include-and-exclude
+          [ $(du -b dist | cut -f1) -lt 300000 ]
+
+      - name: Check code coverage
+        if: matrix.session == 'coverage'
         shell: bash -el {0}
         run: |
-          python -m pytest --cov=. --cov-report term-missing --cov-fail-under 90 --doctest-modules --junitxml=junit/test-results-${{ matrix.python-version }}.xml
-      - name: Test command line
+          export NOXSESSION=tests
+          # check that code coverage is not lower than specific percent
+          nox --verbose --python=${{ matrix.python }} -- --cov=. --cov-report=term-missing --cov-fail-under=94
+
+      - name: Run tests with nox
+        if: matrix.session == 'tests'
         shell: bash -el {0}
         run: |
-          ./test/cmdline_tests.sh
-      - name: Upload pytest test results
-        uses: actions/upload-artifact@v3
+          nox --verbose --python=${{ matrix.python }}
+
+  large-files:
+    name: File sizes
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out the repository
+        uses: actions/checkout@v4
+
+      - name: Check for large files
+        uses: actionsdesk/lfs-warning@v3.2
         with:
-          name: pytest-results-${{ matrix.python-version }}
-          path: junit/test-results-${{ matrix.python-version }}.xml
-        # Use always() to always run this step to publish test results when there are test failures
-        if: ${{ always() }}
+          token: ${{ secrets.GITHUB_TOKEN }} # Optional
+          filesizelimit: 500000b
+          labelName: large-files

.gitignore

@@ -30,6 +30,7 @@ coverage.xml
 xx-*
 test-*
 c57*
+.nox/
 
 # Build files
 *egg-info*

.readthedocs.yml

@@ -2,23 +2,30 @@
 # Read the Docs configuration file
 # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
 
-# Required
+# Note: I used https://docs.readthedocs.io/en/stable/build-customization.html#install-dependencies-with-poetry for inspiration
+
 version: 2
 
 build:
   os: "ubuntu-22.04"
   tools:
-    python: "mambaforge-22.9"
+    python: "3.8"
+  jobs:
+    post_create_environment:
+      # Install poetry
+      # https://python-poetry.org/docs/#installing-manually
+      - pip install poetry
+      # Tell poetry to not use a virtual environment
+      - poetry config virtualenvs.create false
+    post_install:
+      # Install dependencies with 'docs' dependency group
+      # https://python-poetry.org/docs/managing-dependencies/#dependency-groups
+      - poetry install --only main,docs
 
-# Build documentation in the docs/ directory with Sphinx
 sphinx:
   configuration: doc/conf.py
   fail_on_warning: true
 
 # Optionally build your docs in additional formats such as PDF
 formats:
   - pdf
-
-conda:
-  environment: .readthedocs_conda_env.yml
-

PUBLISHING.rst

@@ -15,12 +15,7 @@ Other branches are used for development.
 
 New Dependencies
 ----------------
-If you've added dependencies to trtools or its tests, those dependencies should be listed in
-
-  * pyproject.toml
-  * the .readthedocs_conda_env.yml file in the root of the repository that's used for building
-    TRTool's Read The Docs webpage.
-  * the appropriate section of the bioconda recipe (see below)
+If you've added dependencies to trtools or its tests, those dependencies should be listed in our pyproject.toml file and the appropriate section of the bioconda recipe (see below).
 
 Publishing Steps
 ----------------

README.rst

@@ -31,7 +31,7 @@ If you use TRTools in your work, please cite: Nima Mousavi, Jonathan Margoliash,
 Install
 -------
 
-Note: TRTools supports Python versions 3.8 and up. We do not officially support python versions 3.6 and 3.7 as they are `end of life <https://devguide.python.org/versions/#status-of-python-versions>`_, but we believe TRTools likely works with them from previous testing results.
+Note: TRTools supports Python versions 3.8 and up. We do not officially support python version 3.7 as it is `end of life <https://devguide.python.org/versions/#status-of-python-versions>`_, but we believe TRTools likely works with it from previous testing results.
 
 With conda
 ^^^^^^^^^^
@@ -61,20 +61,25 @@ Then run:
 
 Note: TRTools installation may fail for pip version 10.0.1, hence the need to upgrade pip first
 
+Note: if you will run or test :code:`simTR`, you will also need to install `ART <https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm>`_. The simTR tests will only run if the executable :code:`art_illumina` is found on your :code:`PATH`. If it has been installed, :code:`which art_illumina` should return a path.
+
 From source
 ^^^^^^^^^^^
 
-To install from source (only recommended for development) download the TRTools repository from `github <https://github.com/gymrek-lab/TRTools/>`_,
-checkout the branch you're interested in, and run the following command from the base directory of the repo. e.g.::
+To install from source (only recommended for development) clone the TRTools repository from `github <https://github.com/gymrek-lab/TRTools/>`_ and checkout the branch you're interested in::
 
-        git clone https://github.com/gymrek-lab/TRTools
+        git clone -b master https://github.com/gymrek-lab/TRTools
         cd TRTools/
-        pip install --upgrade pip
-        pip install -e .
 
-Note: make sure TRTools is not installed in the environment via a different method before installing from source. :code:`which dumpSTR` should return nothing.
+Now, create 1) a conda environment with our development tools and 2) a virtual environment with our dependencies and an editable install of TRTools::
 
-Note: if you will run or test :code:`simTR`, you will also need to install `ART <https://www.niehs.nih.gov/research/resources/software/biostatistics/art/index.cfm>`_. The simTR tests will only run if the executable :code:`art_illumina` is found on your :code:`PATH`. If it has been installed, :code:`which art_illumina` should return a path.
+        conda env create -n trtools -f dev-env.yml
+        conda run -n trtools poetry install
+
+Now, whenever you'd like to run/import pytest or TRTools, you will first need to activate both environments::
+
+        conda activate trtools
+        poetry shell
 
 With Docker
 ^^^^^^^^^^^
@@ -131,6 +136,12 @@ See our description of the `features and example use-cases <https://trtools.read
 .. _HipSTR: https://hipstr-tool.github.io/HipSTR/
 .. _PopSTR: https://github.com/DecodeGenetics/popSTR
 
+Testing
+-------
+After you've installed TRTools, we recommend running our tests to confirm that TRTools works properly on your system. Just execute the following::
+
+        test_trtools.sh
+
 Development Notes
 -----------------
 
@@ -148,10 +159,14 @@ We appreciate contributions to TRTools. If you would like to contribute a fix or
 
 1. Consider `discussing <https://github.com/gymrek-lab/TRTools/issues>`_ your solution with us first so we can provide help or feedback if necessary.
 #. Install TRTools from source `as above <From source_>`_.
-#. Additionally, install :code:`pytest`, `pytest-cov <https://anaconda.org/conda-forge/pytest-cov>`_, :code:`sphinx>=3` and :code:`sphinx_rtd_theme`, in your environment.
 #. Fork the TRTools repository.
 #. Create a branch off of :code:`master` titled with the name of your feature.
 #. Make your changes. 
+#. If you need to add a dependency or update the version of a dependency, you can use the :code:`poetry add` command.
+
+    * You should specify a `version constraint <https://python-poetry.org/docs/master/dependency-specification#version-constraints>`_ when adding a dependency. Use the oldest version compatible with your code. Don't worry if you're not sure at first, since you can (and should!) always update it later. For example, to specify a version of :code:`numpy>=1.23.0`, you can run :code:`poetry add 'numpy>=1.23.0'`.
+    * Afterwards, double-check that the :code:`poetry.lock` file contains 1.23.0 in it. **All of our dependencies should be locked to their minimum versions at all times.** To downgrade to a specific version of :code:`numpy` in our lock file, you can explicitly add the version via :code:`poetry add 'numpy==1.23.0'`, manually edit the pyproject.toml file to use a :code:`>=` sign in front of the version number, and then run :code:`poetry lock --no-update`.
+
 #. Document your changes.
 
    * Ensure all functions, modules, classes etc. conform to `numpy docstring standards <https://numpydoc.readthedocs.io/en/latest/format.html>`_.
@@ -165,9 +180,10 @@ We appreciate contributions to TRTools. If you would like to contribute a fix or
 
 #. Add tests to test any new functionality. Add them to the :code:`tests/` folder in the directory of the code you modified.
 
-   * :code:`cd` to the root of the project and run :code:`python -m pytest --cov=. --cov-report term-missing` to make sure that (1) all tests pass and (2) any code you have added is covered by tests. (Code coverage may **not** go down).
+   * :code:`cd` to the root of the project and run :code:`poetry run pytest --cov=. --cov-report term-missing` to make sure that (1) all tests pass and (2) any code you have added is covered by tests. (Code coverage may **not** go down).
+   * :code:`cd` to the root of the project and run :code:`nox` to make sure that the tests pass on all versions of python that we support.
 
-#. Submit a pull request (PR) **to the master branch** of the central repository with a description of what changes you have made. Title the PR according to the `conventional commits spec <https://www.conventionalcommits.org>`_.
+#. Submit a pull request (PR) **to the master branch** of the central repository with a description of what changes you have made. Prefix the title of the PR according to the `conventional commits spec <https://www.conventionalcommits.org>`_.
    A member of the TRTools team will reply and continue the contribution process from there, possibly asking for additional information/effort on your part.
 
    * If you are reviewing a pull request, please double-check that the PR addresses each item in `our PR checklist <https://github.com/gymrek-lab/TRTools/blob/master/.github/pull_request_template.md>`_

dev-env.yml

@@ -0,0 +1,15 @@
+channels:
+  - conda-forge
+  - bioconda
+  - nodefaults
+dependencies:
+  - conda-forge::python=3.8 # the lowest version of python that we formally support
+  - conda-forge::pip==23.3.2
+  - bioconda::bcftools==1.19
+  - bioconda::art==2016.06.05
+  - conda-forge::poetry==1.7.1
+  - conda-forge::nox==2023.04.22
+  - conda-forge::poetry-plugin-export==1.6.0
+  - pip:
+    - nox-poetry==1.0.3
+    - poetry-conda==0.1.1