Custom API reference options (#60)

New options for 'create-api-reference-docs':
* Add '--special-option' to api-ref-docs task.
* Add `--full-docs-file` multiple input option.

New tests for special-option + full-docs-file.
As well as an updated test file structure.

Clean up imports in `tasks.py`.

.github/workflows/ci_cd_updated_default_branch.yml

@@ -78,6 +78,16 @@ on:
         required: false
         type: string
         default: ""
+      full_docs_files:
+        description: "A single or multi-line string of relative paths to files in which to include everything - even those without documentation strings. This may be useful for a file full of data models or to ensure all class attributes are listed."
+        required: false
+        type: string
+        default: ""
+      special_file_api_ref_options:
+        description: "A single or multi-line string of combinations of a relative path to a Python file and a fully formed mkdocstrings option that should be added to the generated MarkDown file for the Python API reference documentation. Example: 'my_module/py_file.py,show_bases:false'. Encapsulate the value in double quotation marks (\") if including spaces ( ). Important: If multiple `package_dirs` are supplied, the relative path MUST include/start with the appropriate 'package_dir' value, e.g., '\"my_package/my_module/py_file.py,show_bases: false\"'."
+        required: false
+        type: string
+        default: ""
       landing_page_replacements:
         description: "A single or multi-line string of replacements (mappings) to be performed on README.md when creating the documentation's landing page (index.md). This list ALWAYS includes replacing `'docs/'` with an empty string to correct relative links, i.e., this cannot be overwritten. By default `'(LICENSE)'` is replaced by `'(LICENSE.md)'`."
         required: false
@@ -178,6 +188,8 @@ jobs:
         UNWANTED_FOLDERS=()
         UNWANTED_FILES=()
         FULL_DOCS_FOLDERS=()
+        FULL_DOCS_FILES=()
+        SPECIAL_OPTIONS=()
         while IFS= read -r line; do
           if [ -n "${line}" ]; then PACKAGE_DIRS+=(--package-dir="${line}"); fi
         done <<< "${{ inputs.package_dirs }}"
@@ -190,14 +202,22 @@ jobs:
         while IFS= read -r line; do
           if [ -n "${line}" ]; then FULL_DOCS_FOLDERS+=(--full-docs-folder="${line}"); fi
         done <<< "${{ inputs.full_docs_dirs }}"
+        while IFS= read -r line; do
+          if [ -n "${line}" ]; then FULL_DOCS_FILES+=(--full-docs-file="${line}"); fi
+        done <<< "${{ inputs.full_docs_files }}"
+        while IFS= read -r line; do
+          if [ -n "${line}" ]; then SPECIAL_OPTIONS+=(--special-option="${line}"); fi
+        done <<< "${{ inputs.special_file_api_ref_options }}"
 
         ci-cd create-api-reference-docs ${DEBUG} ${RELATIVE} \
           --pre-clean \
           --root-repo-path=${PWD} \
           "${PACKAGE_DIRS[@]}" \
           "${UNWANTED_FOLDERS[@]}" \
           "${UNWANTED_FILES[@]}" \
-          "${FULL_DOCS_FOLDERS[@]}"
+          "${FULL_DOCS_FOLDERS[@]}" \
+          "${FULL_DOCS_FILES[@]}" \
+          "${SPECIAL_OPTIONS[@]}"
 
     - name: Update landing page
       if: env.RELEASE_RUN == 'false' && inputs.update_docs_landing_page

.github/workflows/ci_tests.yml

@@ -108,6 +108,16 @@ on:
         required: false
         type: string
         default: ""
+      full_docs_files:
+        description: "A single or multi-line string of relative paths to files in which to include everything - even those without documentation strings. This may be useful for a file full of data models or to ensure all class attributes are listed."
+        required: false
+        type: string
+        default: ""
+      special_file_api_ref_options:
+        description: "A single or multi-line string of combinations of a relative path to a Python file and a fully formed mkdocstrings option that should be added to the generated MarkDown file for the Python API reference documentation. Example: 'my_module/py_file.py,show_bases:false'. Encapsulate the value in double quotation marks (\") if including spaces ( ). Important: If multiple `package_dirs` are supplied, the relative path MUST include/start with the appropriate 'package_dir' value, e.g., '\"my_package/my_module/py_file.py,show_bases: false\"'."
+        required: false
+        type: string
+        default: ""
       landing_page_replacements:
         description: "A single or multi-line string of replacements (mappings) to be performed on README.md when creating the documentation's landing page (index.md). This list ALWAYS includes replacing `'docs/'` with an empty string to correct relative links, i.e., this cannot be overwritten. By default `'(LICENSE)'` is replaced by `'(LICENSE.md)'`."
         required: false
@@ -274,6 +284,8 @@ jobs:
         UNWANTED_FOLDERS=()
         UNWANTED_FILES=()
         FULL_DOCS_FOLDERS=()
+        FULL_DOCS_FILES=()
+        SPECIAL_OPTIONS=()
         while IFS= read -r line; do
           if [ -n "${line}" ]; then PACKAGE_DIRS+=(--package-dir="${line}"); fi
         done <<< "${{ inputs.package_dirs }}"
@@ -286,14 +298,22 @@ jobs:
         while IFS= read -r line; do
           if [ -n "${line}" ]; then FULL_DOCS_FOLDERS+=(--full-docs-folder="${line}"); fi
         done <<< "${{ inputs.full_docs_dirs }}"
+        while IFS= read -r line; do
+          if [ -n "${line}" ]; then FULL_DOCS_FILES+=(--full-docs-file="${line}"); fi
+        done <<< "${{ inputs.full_docs_files }}"
+        while IFS= read -r line; do
+          if [ -n "${line}" ]; then SPECIAL_OPTIONS+=(--special-option="${line}"); fi
+        done <<< "${{ inputs.special_file_api_ref_options }}"
 
         ci-cd create-api-reference-docs ${DEBUG} ${RELATIVE} \
           --pre-clean \
           --root-repo-path=${PWD} \
           "${PACKAGE_DIRS[@]}" \
           "${UNWANTED_FOLDERS[@]}" \
           "${UNWANTED_FILES[@]}" \
-          "${FULL_DOCS_FOLDERS[@]}"
+          "${FULL_DOCS_FOLDERS[@]}" \
+          "${FULL_DOCS_FILES[@]}" \
+          "${SPECIAL_OPTIONS[@]}"
 
     - name: Update landing page
       if: inputs.update_docs_landing_page

README.md

@@ -17,7 +17,7 @@ See the [pre-commit website](https://pre-commit.com) to learn more about how to
 
 ## License & copyright
 
-This repository licensed under the  [MIT LICENSE](LICENSE) with copyright © 2022 Casper Welzel Andersen ([CasperWA](https://github.com/CasperWA)) & SINTEF ([on GitHub](https://github.com/SINTEF)).
+This repository is licensed under the [MIT LICENSE](LICENSE) with copyright © 2022 Casper Welzel Andersen ([CasperWA](https://github.com/CasperWA)) & SINTEF ([on GitHub](https://github.com/SINTEF)).
 
 ## Funding support
 

ci_cd/tasks.py

@@ -1,14 +1,17 @@
 """Repository management tasks powered by `invoke`.
 More information on `invoke` can be found at [pyinvoke.org](http://www.pyinvoke.org/).
 """
-# pylint: disable=import-outside-toplevel
+import os
 import re
+import shutil
 import sys
 import traceback
+from collections import defaultdict
 from enum import Enum
 from pathlib import Path
 from typing import TYPE_CHECKING
 
+import tomlkit
 from invoke import task
 
 if TYPE_CHECKING:  # pragma: no cover
@@ -303,8 +306,6 @@ def update_deps(  # pylint: disable=too-many-branches,too-many-locals,too-many-s
     context, root_repo_path=".", fail_fast=False, pre_commit=False
 ):
     """Update dependencies in specified Python package's `pyproject.toml`."""
-    import tomlkit
-
     if TYPE_CHECKING:  # pragma: no cover
         context: "Context" = context
         root_repo_path: str = root_repo_path
@@ -465,13 +466,38 @@ def update_deps(  # pylint: disable=too-many-branches,too-many-locals,too-many-s
             "models or to ensure all class attributes are listed. This input option "
             "can be supplied multiple times."
         ),
+        "full-docs-file": (
+            "A full relative path to a file in which to include everything - even "
+            "those without documentation strings. This may be useful for a file full "
+            "of data models or to ensure all class attributes are listed. This input "
+            "option can be supplied multiple times."
+        ),
+        "special-option": (
+            "A combination of a relative path to a file and a fully formed "
+            "mkdocstrings option that should be added to the generated MarkDown file. "
+            "The combination should be comma-separated. Example: "
+            "'my_module/py_file.py,show_bases:false'. Encapsulate the value in double "
+            'quotation marks (") if including spaces ( ). Important: If multiple '
+            "package-dir options are supplied, the relative path MUST include/start "
+            "with the package-dir value, e.g., "
+            "'\"my_package/my_module/py_file.py,show_bases: false\"'. This input "
+            "option can be supplied multiple times. The options will be accumulated "
+            "for the same file, if given several times."
+        ),
         "relative": (
             "Whether or not to use relative Python import links in the API reference "
             "markdown files."
         ),
         "debug": "Whether or not to print debug statements.",
     },
-    iterable=["package_dir", "unwanted_folder", "unwanted_file", "full_docs_folder"],
+    iterable=[
+        "package_dir",
+        "unwanted_folder",
+        "unwanted_file",
+        "full_docs_folder",
+        "full_docs_file",
+        "special_option",
+    ],
 )
 def create_api_reference_docs(  # pylint: disable=too-many-locals,too-many-branches,too-many-statements,line-too-long
     context,
@@ -483,13 +509,12 @@ def create_api_reference_docs(  # pylint: disable=too-many-locals,too-many-branc
     unwanted_folder=None,
     unwanted_file=None,
     full_docs_folder=None,
+    full_docs_file=None,
+    special_option=None,
     relative=False,
     debug=False,
 ):
     """Create the Python API Reference in the documentation."""
-    import os
-    import shutil
-
     if TYPE_CHECKING:  # pragma: no cover
         context: "Context" = context
         pre_clean: bool = pre_clean
@@ -505,6 +530,10 @@ def create_api_reference_docs(  # pylint: disable=too-many-locals,too-many-branc
         unwanted_file: list[str] = ["__init__.py"]
     if not full_docs_folder:
         full_docs_folder: list[str] = []
+    if not full_docs_file:
+        full_docs_file: list[str] = []
+    if not special_option:
+        special_option: list[str] = []
 
     def write_file(full_path: Path, content: str) -> None:
         """Write file with `content` to `full_path`"""
@@ -524,12 +553,33 @@ def write_file(full_path: Path, content: str) -> None:
     root_repo_path: Path = Path(root_repo_path).resolve()
     package_dirs: list[Path] = [root_repo_path / _ for _ in package_dir]
     docs_api_ref_dir = root_repo_path / docs_folder / "api_reference"
+
     if debug:
         print("package_dirs:", package_dirs, flush=True)
         print("docs_api_ref_dir:", docs_api_ref_dir, flush=True)
         print("unwanted_folder:", unwanted_folder, flush=True)
         print("unwanted_file:", unwanted_file, flush=True)
         print("full_docs_folder:", full_docs_folder, flush=True)
+        print("full_docs_file:", full_docs_file, flush=True)
+        print("special_option:", special_option, flush=True)
+
+    special_options_files = defaultdict(list)
+    for special_file, option in [_.split(",", maxsplit=1) for _ in special_option]:
+        if any("," in _ for _ in (special_file, option)):
+            if debug:
+                print(
+                    "Failing for special-option:",
+                    ",".join([special_file, option]),
+                    flush=True,
+                )
+            sys.exit(
+                "special-option values may only include a single comma (,) to "
+                "separate the relative file path and the mkdocstsrings option."
+            )
+        special_options_files[special_file].append(option)
+
+    if debug:
+        print("special_options_files:", special_options_files, flush=True)
 
     if any("/" in _ for _ in unwanted_folder + unwanted_file):
         sys.exit(
@@ -539,8 +589,8 @@ def write_file(full_path: Path, content: str) -> None:
 
     pages_template = 'title: "{name}"\n'
     md_template = "# {name}\n\n::: {py_path}\n"
-    no_docstring_template = (
-        md_template + f"{' ' * 4}options:\n{' ' * 6}show_if_no_docstring: true\n"
+    no_docstring_template_addition = (
+        f"{' ' * 4}options:\n{' ' * 6}show_if_no_docstring: true\n"
     )
 
     if docs_api_ref_dir.exists() and pre_clean:
@@ -555,7 +605,10 @@ def write_file(full_path: Path, content: str) -> None:
         print(f"Writing file: {docs_api_ref_dir / '.pages'}", flush=True)
     write_file(
         full_path=docs_api_ref_dir / ".pages",
-        content=pages_template.format(name="API Reference"),
+        content=(
+            pages_template.format(name="API Reference")
+            + (no_docstring_template_addition if "." in full_docs_folder else "")
+        ),
     )
 
     single_package = len(package_dirs) == 1
@@ -570,10 +623,10 @@ def write_file(full_path: Path, content: str) -> None:
                     dirnames.remove(unwanted)
 
             relpath = Path(dirpath).relative_to(
-                package if single_package else root_repo_path
+                package if single_package else package.parent
             )
             abspath = (
-                package / relpath if single_package else root_repo_path / relpath
+                package / relpath if single_package else package.parent / relpath
             ).resolve()
             if debug:
                 print("relpath:", relpath, flush=True)
@@ -594,8 +647,13 @@ def write_file(full_path: Path, content: str) -> None:
                     print(f"Writing file: {docs_sub_dir / '.pages'}", flush=True)
                 write_file(
                     full_path=docs_sub_dir / ".pages",
-                    content=pages_template.format(
-                        name=str(relpath).rsplit("/", maxsplit=1)[-1]
+                    content=(
+                        pages_template.format(name=relpath.name)
+                        + (
+                            no_docstring_template_addition
+                            if str(relpath) in full_docs_folder
+                            else ""
+                        )
                     ),
                 )
 
@@ -620,22 +678,38 @@ def write_file(full_path: Path, content: str) -> None:
                     package.relative_to(root_repo_path) if relative else package.name
                 )
                 py_path = (
-                    f"{py_path_root}/{relpath}/{filename.stem}".replace("/", ".")
-                    if str(relpath) != "."
-                    else f"{py_path_root}/{filename.stem}".replace("/", ".")
+                    f"{py_path_root}/{filename.stem}".replace("/", ".")
+                    if str(relpath) == "."
+                    or (str(relpath) == package.name and not single_package)
+                    else f"{py_path_root}/{relpath}/{filename.stem}".replace("/", ".")
                 )
                 if debug:
                     print("filename:", filename, flush=True)
                     print("py_path:", py_path, flush=True)
 
-                # For special folders we want to include EVERYTHING, even if it doesn't
+                relative_file_path = (
+                    str(filename) if str(relpath) == "." else str(relpath / filename)
+                )
+
+                # For special files we want to include EVERYTHING, even if it doesn't
                 # have a doc-string
-                template = (
-                    no_docstring_template
-                    if str(relpath) in full_docs_folder
-                    else md_template
+                template = md_template + (
+                    no_docstring_template_addition
+                    if relative_file_path in full_docs_file
+                    else ""
                 )
 
+                # Include special options, if any, for certain files.
+                if relative_file_path in special_options_files:
+                    template += (
+                        f"{' ' * 4}options:\n" if "options:\n" not in template else ""
+                    )
+                    template += "\n".join(
+                        f"{' ' * 6}{option}"
+                        for option in special_options_files[relative_file_path]
+                    )
+                    template += "\n"
+
                 if debug:
                     print("template:", template, flush=True)
                     print(

docs/hooks/docs_api_reference.md

@@ -45,6 +45,10 @@ Any of these options can be given through the `args` key when defining the hook.
 | `--unwanted-folder` | A folder to avoid including into the Python API reference documentation. If this is not supplied, it will default to `__pycache__`.</br></br>**Note**: Only folder names, not paths, may be included.</br></br>**Note**: All folders and their contents with these names will be excluded.</br></br>This input option can be supplied multiple times. | No | \_\_pycache\_\_ | _string_ |
 | `--unwanted-file` | A file to avoid including into the Python API reference documentation. If this is not supplied, it will default to `__init__.py`</br></br>**Note**: Only full file names, not paths, may be included, i.e., filename + file extension.</br></br>**Note**: All files with these names will be excluded.</br></br>This input option can be supplied multiple times. | No | \_\_init\_\_.py | _string_ |
 | `--full-docs-folder` | A folder in which to include everything - even those without documentation strings. This may be useful for a module full of data models or to ensure all class attributes are listed.</br></br>This input option can be supplied multiple times. | No | _Empty string_ | _string_ |
+| `--full-docs-file` | A full relative path to a file in which to include everything - even those without documentation strings. This may be useful for a file full of data models or to ensure all class attributes are listed.</br></br>This input option can be supplied multiple times. | No | _Empty string_ | _string_ |
+| `--special-option` | A combination of a relative path to a file and a fully formed mkdocstrings option that should be added to the generated MarkDown file. The combination should be comma-separated.</br>Example:
+`my_module/py_file.py,show_bases:false`.</br></br>Encapsulate the value in double quotation marks (`"`) if including spaces ( ).</br></br>**Important**: If multiple package-dir options are supplied, the relative path MUST include/start with the package-dir value, e.g., `"my_package/my_module/py_file.py,show_bases: false"`.</br></br>This input option can be supplied multiple times. The options will be accumulated
+for the same file, if given several times. | No | _Empty string_ | _string_ |
 | `--relative` | Whether or not to use relative Python import links in the API reference markdown files. See section [Using it together with CI/CD workflows](#using-it-together-with-cicd-workflows) above. | No | `False` | _boolean_ |
 | `--debug` | Whether or not to print debug statements. | No | `False` | _boolean_ |