Fix documentation builds (#1039)

* Remove documentation-only deps from `docker/conda/environments/cuda11.8_dev.yml`
* `morpeus_docs` cmake target removed from default build target
* documentation deps are now installed from conda-forge not pypi
* Set an env variable `MORPHEUS_IN_SPHINX_BUILD` during documentation builds allowing the code to bypass certain calls with side-effects.
* Sphinx now treats all warnings as fatal errors.
* Removed out of date `docs/source/control_message_guide.md` doc
* Fix numerous docstring issues & broken links
* Private symbols moved into anonymous namespaces to prevent them from being visible to doxygen.
* `CudfHelper::load()` avoids calling `import_morpheus___lib__cudf_helpers` if `MORPHEUS_IN_SPHINX_BUILD` is defined.
* `morpheus/utils/nvt/mutate.py` will now defined a stub `annotate` decorator if `MORPHEUS_IN_SPHINX_BUILD` is defined.
* `morpheus/utils/schema_transforms.py` will avoid calling `patch_numpy_dtype_registry` and `register_morpheus_extensions` if `MORPHEUS_IN_SPHINX_BUILD` is defined.
* Replace empty `__alll__` declarations in `__init__.py` files

fixes #1028

Authors:
  - David Gardner (https://github.com/dagardner-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Devin Robison (https://github.com/drobison00)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1039

ci/scripts/github/docs.sh

@@ -35,7 +35,7 @@ ${MORPHEUS_ROOT}/scripts/fetch_data.py fetch docs examples
 git submodule update --init --recursive
 
 rapids-logger "Configuring for docs"
-cmake -B build -G Ninja ${CMAKE_BUILD_ALL_FEATURES} -DMORPHEUS_BUILD_DOCS=ON .
+cmake -B build -G Ninja ${CMAKE_BUILD_ALL_FEATURES} -DMORPHEUS_PYTHON_BUILD_STUBS=OFF -DMORPHEUS_BUILD_DOCS=ON .
 
 rapids-logger "Building docs"
 cmake --build build --target morpheus_docs

docker/conda/environments/cuda11.8_dev.yml

@@ -62,7 +62,6 @@ dependencies:
     - librdkafka=1.9.2
     - mlflow>=2.2.1,<3
     - mrc=23.07
-    - myst-parser==1.0.0
     - networkx=3.1
     - ninja=1.10
     - nodejs=18.15.0
@@ -89,8 +88,6 @@ dependencies:
     - rapidjson=1.1.0
     - scikit-build=0.17.1
     - scikit-learn=1.2.2
-    - sphinx
-    - sphinx_rtd_theme
     - sysroot_linux-64=2.17
     - tritonclient=2.26 #  Required by NvTabular, force the version, so we get protobufs compatible with 4.21
     - tqdm=4

docs/CMakeLists.txt

@@ -20,13 +20,14 @@ find_package(Sphinx REQUIRED)
 
 set(SPHINX_SOURCE ${CMAKE_CURRENT_SOURCE_DIR}/source)
 set(SPHINX_BUILD ${CMAKE_CURRENT_BINARY_DIR}/html)
+set(SPHINX_ARGS -b html -j auto -T -W)
 
-add_custom_target(${PROJECT_NAME}_docs ALL
-                  COMMAND
-                  BUILD_DIR=${CMAKE_CURRENT_BINARY_DIR}
-                  ${SPHINX_EXECUTABLE} -b html -j auto -T
-                  ${SPHINX_SOURCE} ${SPHINX_BUILD}
-                  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
-                  COMMENT "Generating documentation with Sphinx")
+add_custom_target(${PROJECT_NAME}_docs
+   COMMAND
+      BUILD_DIR=${CMAKE_CURRENT_BINARY_DIR} ${SPHINX_EXECUTABLE} ${SPHINX_ARGS} ${SPHINX_SOURCE} ${SPHINX_BUILD}
+   WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
+   COMMENT "Generating documentation with Sphinx"
+   DEPENDS morpheus-package-outputs
+)
 
 list(POP_BACK CMAKE_MESSAGE_CONTEXT)

docs/README.md

@@ -27,6 +27,6 @@ mamba env update -f docs/conda_docs.yml
 
 ## Build Morpheus and Documentation
 ```
-CMAKE_CONFIGURE_EXTRA_ARGS="-DMORPHEUS_BUILD_DOCS=ON" ./scripts/compile.sh
+CMAKE_CONFIGURE_EXTRA_ARGS="-DMORPHEUS_BUILD_DOCS=ON" ./scripts/compile.sh --target morpheus_docs
 ```
 Outputs to `build/docs/html`

docs/conda_docs.yml

@@ -15,16 +15,13 @@
 
 name: morpheus
 channels:
-    - conda-forge
+  - conda-forge
 dependencies:
-    - doxygen=1.9.2
-    - pip
-    ####### Morpheus Pip Dependencies (keep sorted!) #######
-    - pip:
-        - breathe==4.34.0
-        - exhale==0.3.6
-        - ipython
-        - myst-parser==0.17.2
-        - nbsphinx
-        - sphinx
-        - sphinx_rtd_theme
+  - breathe=4.34.0
+  - doxygen=1.9.2
+  - exhale=0.3.6
+  - ipython
+  - myst-parser=0.17.2
+  - nbsphinx
+  - sphinx
+  - sphinx_rtd_theme

docs/source/conf.py

@@ -31,13 +31,28 @@
 import os
 import sys
 import textwrap
+import warnings
 
 import packaging
 
+# Ignore FutureWarnings coming from docutils remove this once we can upgrade to Sphinx 5.0
+# https://github.com/sphinx-doc/sphinx/issues/9777
+warnings.simplefilter(action='ignore', category=FutureWarning)
+
+# Get the morpheus root from the environment variable or default to finding it relative to this file
+morpheus_root = os.environ.get('MORPHEUS_ROOT', os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
+
+# Make sure we can access the digital fingerprinting example
+sys.path.append(os.path.join(morpheus_root, 'examples/digital_fingerprinting/production/morpheus'))
+
+# Add the Sphinx extensions directory to sys.path to allow for the github_link extension to be found
 sys.path.insert(0, os.path.abspath('sphinxext'))
 
 from github_link import make_linkcode_resolve  # noqa
 
+# Set an environment variable we can use to determine ifuncf we are building docs
+os.environ["MORPHEUS_IN_SPHINX_BUILD"] = "1"
+
 # -- Project information -----------------------------------------------------
 
 project = 'morpheus'
@@ -113,7 +128,7 @@
         BRIEF_MEMBER_DESC = YES
         BUILTIN_STL_SUPPORT = YES
         DOT_IMAGE_FORMAT = svg
-        EXCLUDE_PATTERNS = */tests/* */include/nvtext/* */__pycache__/*
+        EXCLUDE_PATTERNS = */tests/* */include/nvtext/* */__pycache__/* */doca/*
         EXCLUDE_SYMBOLS = "@*" "cudf*" "py::literals" "RdKafka" "mrc*" "std*"
         EXTENSION_MAPPING = cu=C++ cuh=C++
         EXTRACT_ALL = YES
@@ -148,11 +163,21 @@
 myst_heading_anchors = 4  # Generate links for markdown headers
 autodoc_mock_imports = [
     "cudf",  # Avoid loading GPU libraries during the documentation build
+    "cupy",  # Avoid loading GPU libraries during the documentation build
+    "merlin",
     "morpheus.cli.commands",  # Dont document the CLI in Sphinx
+    "morpheus.utils.nvt.mutate.annotate",
+    "nvtabular",
+    "pandas",  # Avoid documenting pandas for the purposes of the dfencoder.dataframe
     "tensorrt",
+    "torch",
     "tqdm",
 ]
 
+suppress_warnings = [
+    "myst.header"  # Allow header increases from h2 to h4 (skipping h3)
+]
+
 # Config numpydoc
 numpydoc_show_inherited_class_members = True
 numpydoc_class_members_toctree = False
@@ -174,7 +199,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = "en"
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.

docs/source/developer_guide/contributing.md

@@ -44,7 +44,7 @@ More information can be found at: [Contributor Code of Conduct](https://github.c
 
 1. Find an issue to work on. The best way is to search for issues with the [good first issue](https://github.com/nv-morpheus/Morpheus/issues) label.
 2. Comment on the issue stating that you are going to work on it.
-3. Code! Make sure to update unit tests! Ensure the [license headers are set properly](#Licensing).
+3. Code! Make sure to update unit tests! Ensure the [license headers are set properly](#licensing).
 4. When done, [create your pull request](https://github.com/nv-morpheus/Morpheus/compare).
 5. Wait for other developers to review your code and update code as needed.
 6. Once reviewed and approved, a Morpheus developer will merge your pull request.
@@ -91,15 +91,15 @@ git submodule update --init --recursive
 
 The large model and data files in this repo are stored using [Git Large File Storage (LFS)](https://git-lfs.github.com/). These files will be required for running the training/validation scripts and example pipelines for the Morpheus pre-trained models.
 
-By default only those files stored in LFS strictly needed for running Morpheus are included when the Morpheus repository is cloned. Additional datasets can be downloaded using the `scripts/fetch_data.py` script. Refer to the section [Git LFS](README.md#git-lfs) of the [README.md](README.md) file for details on this.
+By default only those files stored in LFS strictly needed for running Morpheus are included when the Morpheus repository is cloned. Additional datasets can be downloaded using the `scripts/fetch_data.py` script. Refer to the section [Git LFS](../getting_started.md#git-lfs) of the [getting_started.md](../getting_started.md) guide for details on this.
 
 ### Build in Docker Container
 
 This workflow utilizes a Docker container to set up most dependencies ensuring a consistent environment.
 
 #### Prerequisites
 
-1. Ensure all [requirements](README.md#requirements) from [README.md](README.md) are met.
+1. Ensure all [requirements](../getting_started.md#requirements) from [getting_started.md](../getting_started.md) are met.
 1. Build the development container
    ```bash
    ./docker/build_container_dev.sh
@@ -159,7 +159,7 @@ This workflow utilizes a Docker container to set up most dependencies ensuring a
    pip install -e /workspace
    ```
    Once Morpheus has been built, it can be installed into the current virtual environment.
-5. [Run Morpheus](./README.md#running-morpheus)
+5. [Run Morpheus](../getting_started.md#running-morpheus)
    ```bash
    morpheus run pipeline-nlp ...
    ```

docs/source/developer_guide/guides.md

@@ -44,7 +44,7 @@ Morpheus includes, as of version 23.03, a number of pre-defined module implement
 custom pipeline. Modules can be thought of as units of work, which exist at a lower level than stages. Modules can
 be defined, registered, chained, nested, and loaded at runtime. Modules can be written in Python or C++.
 
-- [List of available Morpheus modules](../modules/morpheus_modules.md)
+- [List of available Morpheus modules](../modules/index.md)
 
 There are likely going to be situations that require writing a custom module, either for creating your own
 reusable work units, or for creating a new compound module from a set of existing primitives. The following guides

docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md

@@ -428,7 +428,7 @@ For each MultiAEMessage received, containing a trained model, the function uploa
 associated metadata such as experiment name, run name, parameters, metrics, and the model signature. If the MLflow
 server is running on Databricks, the function also applies the required permissions to the registered model.
 
-For a complete reference, refer to: [MLflow Model Writer](./docs/source/modules/examples/digital_fingerprinting/mlflow_model_writer.md)
+For a complete reference, refer to: [MLflow Model Writer](../../modules/core/mlflow_model_writer.md)
 
 ```python
 @register_module(MLFLOW_MODEL_WRITER, MORPHEUS_MODULE_NAMESPACE)

docs/source/developer_guide/guides/2_real_world_phishing.md

@@ -932,4 +932,4 @@ class WriteToRabbitMQStage(SinglePortStage):
 ```
 
 ## Note
-For information about testing the `RabbitMQSourceStage` and `WriteToRabbitMQStage` stages refer to [`examples/developer_guide/2_2_rabbitmq/README.md`](../../../../examples/developer_guide/2_2_rabbitmq/README.md) in the root of the Morpheus repo.
+For information about testing the `RabbitMQSourceStage` and `WriteToRabbitMQStage` stages refer to `examples/developer_guide/2_2_rabbitmq/README.md` in the root of the Morpheus repo.