[RELEASE] Morpheus v23.01 #646
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[gpuCI] Forward-merge branch-22.11 to branch-23.01 [skip gpuci]
[gpuCI] Forward-merge branch-22.11 to branch-23.01 [skip gpuci]
[gpuCI] Forward-merge branch-22.11 to branch-23.01 [skip gpuci]
[gpuCI] Forward-merge branch-22.11 to branch-23.01 [skip gpuci]
[gpuCI] Forward-merge branch-22.11 to branch-23.01 [skip gpuci]
[gpuCI] Forward-merge branch-22.11 to branch-23.01 [skip gpuci]
Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #513
Remove references to personal forks from READMEs. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Pete MacKinnon (https://github.com/pdmack) - Michael Demoret (https://github.com/mdemoret-nv) URL: #511
Fixes #501 Authors: - Pete MacKinnon (https://github.com/pdmack) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #502
#502 incorrectly ingested kafka-docker dir which is documented as part of developer kafka setup. Authors: - Pete MacKinnon (https://github.com/pdmack) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #518
- Sphinx documentation builds are now a cmake target - exhale & breathe extensions integrate doxygen generated output into Sphinx docs - Marking this as breaking since this PR changes the way the documentation is built New documentation procedure is: 1. Configure cmake with: `MORPHEUS_BUILD_DOCS=ON` 1. `cmake --build build --target morpheus_docs` Closes #272 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #414
Fixes #401 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #514
- Include markdown documentation under the `examples` dir in Sphinx documentation builds by creating symlinks in `docs/source` to the md files in the examples dir. - Replace some ableist language appearing in the documentation. - Remove explcit calls to pipeline.build, this is no longer needed - Update github url for Morpheus relates to issue #495 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - https://github.com/raykallen URL: #519
closes #520 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - David Gardner (https://github.com/dagardner-nv) - Devin Robison (https://github.com/drobison00) URL: #526
Moved `docs/source/basics/img` to Git LFS Authors: - Bhargav Suryadevara (https://github.com/bsuryadevara) - https://github.com/bsuryadev Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #532
- Removed `buffer` stage usage - Updated graph images Authors: - Bhargav Suryadevara (https://github.com/bsuryadevara) - https://github.com/bsuryadev Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #528
* Move most of the content of `README.md` to `docs/source/getting_started.md` * `README.md` now directs readers to other sources of documentation. * `CONTRIBUTING.md` now included in doc builds * `guides.rst` is now a markdown file to ensure its viewable in github * Titles in developer guides updated to remove the prefixed numbers (hold-over from when these were all in one document) * code of conduct & license appear in table of contents * `docs/source/index.rst` contains links to getting started docs * Known issues file refers to open github issues. * Remove smart quote chars * Fix misc parsing errors/warnings * Myst doesn't like 'yml' as a syntax alias for 'yaml' * Fix links to images for developer examples relates to issue #495 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Devin Robison (https://github.com/drobison00) - Michael Demoret (https://github.com/mdemoret-nv) URL: #522
* Morpheus is an SDK, "Morpheus SDK" and "Morpheus SDK Client" are redundant * We do have a helm chart named `morpheus-sdk-client`, so references to the helm chart retain the "sdk client". * Update elevator pitch paragraph in `docs/source/index.rst` to match that of the `README.md` * Add license comment to `README.md` and `CONTRIBUTING.md` files * Includes changes from PR #522 relates to issue #495 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #534
* Includes changes from PRs #522 & #534 * Moves `docs/source/basics/examples.rst` into `examples/basic_usage` and reformat as markdown. This avoids the issue of having two "examples" in the table of contents. * Update remaining references to the quickstart guide with "cloud deployment". * Create Parallel `index.rst` for the developer guides and examples, as markdown doesn't have the equivalent of the `toctree` macro. * Restructure `docs/source/getting_started.md` making it clear which steps are and are not needed when using pre-built containers. * Highlight the examples from top-level docs relates to issue #495 Authors: - David Gardner (https://github.com/dagardner-nv) - https://github.com/bsuryadev - Bhargav Suryadevara (https://github.com/bsuryadevara) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #536
Closes #542 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Devin Robison (https://github.com/drobison00) URL: #546
Closes #489 Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Devin Robison (https://github.com/drobison00) URL: #537
Resolves #496 Authors: - Devin Robison (https://github.com/drobison00) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #530
- Update to PyTorch 1.12.
- Latest PyTorch is v1.13 but is built for CUDA 11.6/11.7. Morpheus and RAPIDS officially support up to CUDA 11.5.
- Fixes errors seen in training scripts/notebooks when using large batch sizes.
- Updates `dfencoder` from `22.09` to `22.11`
- Updated `phishing` and `root cause` inference scripts to move `cudf` imports before `torch` imports. Allows to get past this error which also occurs with PyTorch 1.10:
```
Traceback (most recent call last):
File "root-cause-inference.py", line 33, in <module>
import cudf
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/cudf/__init__.py", line 5, in <module>
validate_setup()
File "/opt/conda/envs/morpheus/lib/python3.8/site-packages/cudf/utils/gpu_utils.py", line 18, in validate_setup
from cuda.cudart import cudaDeviceAttr, cudaError_t
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /opt/conda/envs/morpheus/lib/python3.8/site-packages/cuda/cudart.cpython-38-x86_64-linux-gnu.so)
````
Close #491
Authors:
- Eli Fajardo (https://github.com/efajardo-nv)
Approvers:
- Michael Demoret (https://github.com/mdemoret-nv)
URL: #523
* Includes changes from PR #538 * Fixes casing for proper nouns (NVIDIA, Docker, Triton, Python, Conda) when not referring to a command * Document `StreamPair` (docstrings for globals need to appear after the definition) * Other fixes suggested by Zenobia * Brings grade up from an A to A+ * Fixes CSS theme issue #543 * Various fixes to the `getting_started.md` doc fixes #539 * Fixes issue with the `docker/run_container_*.sh` scripts for users who have the Nvidia Container Toolkit installed but does not have nvidia set as the default runtime * Documents launching a pre-built Morpheus container * Fix errant entry in docstring for `AppShieldSource` which was showing up in the command line help * Documentation work-around for #475 * Fix a few remaining references to 'srf' Authors: - David Gardner (https://github.com/dagardner-nv) - https://github.com/bsuryadev - Bhargav Suryadevara (https://github.com/bsuryadevara) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: #547
* Fix grammar issues resulting from the SRF->MRC rename * Update docs to reflect that stages no longer run in a fiber * Third-party isn't one word * Set up is two words when its a verb, one word when a noun * Replace usage of the word "look" * Misc spelling & grammar fixes Authors: - David Gardner (https://github.com/dagardner-nv) - https://github.com/bsuryadev - Bhargav Suryadevara (https://github.com/bsuryadevara) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: #560
Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Bhargav Suryadevara (https://github.com/bsuryadevara) URL: #571
Closes #541 Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #572
With the updated compilers and dependencies, a new CI image is needed. Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #556
* Remove `<MORPHEUS_ROOT>` making all bash commands copy/past-able * Remove polling flags from Triton command Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #570
Fix script paths in example production Azure/Duo DFP pipeline run commands. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #608
Depends on: https://github.com/nv-morpheus/utilities/pull/1/files#diff-9a3c07061febeba4d3e5191e87706e262644abf8514317eeb084e67137e7c807 This PR is branched from the base Morpheus conversion to MRC (#530). It may make sense to go in as a monolithic patch, or piecemeal. Shifts to using a shared `utilities` submodule for shared cmake, scripts, utilities, etc.. This shared repo provides a number of new `morpheus_utils_xxx` cmake macros/functions that are now used throughout the configuration process. Closes #87 Authors: - Devin Robison (https://github.com/drobison00) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - David Gardner (https://github.com/dagardner-nv) URL: #531
Remove hard-coded keys `username` and `timestamp` from production DFP stages. Allows source dataframe column names to be configured with `config.ae.userid_column_name` and `config.ae.timestamp_column_name`. Fixes #606 Authors: - Eli Fajardo (https://github.com/efajardo-nv) - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) URL: #607
- Removed DFP pipeline output files - Updated `.gitignore` to disregard dip pipeline output files. Authors: - Bhargav Suryadevara (https://github.com/bsuryadevara) - https://github.com/bsuryadev Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #633
This is part 1 of 2 for updates needed by #618. For now, we are explicitly specifying 520 in the driver but plan on using `latest` in the future. Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #634
Closes #619 Authors: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #629
…phinx footer (#635) We now have a significant amount of content in markdown and restructured text files, we should start ensuring they have copyright headers. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #635
Update `morpheus-visualizations` submodule to `branch-23.01`. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #639
* Adds docstrings to several public API classes & functions fixes #549 * Remove unreachable custreamz code in `Receiver` * Exclude `morpheus/cli` from coverage scans * Unrelated fix for DFP notebooks missing `validation_size` constructor arg Breaking changes: * `Pipeline._do_run` is now `Pipeline.run_async` * Remove `StreamWrapper.start` this method wasn't being used, and wasn't being called. * Some methods were marked as "private" by prepending a `_` to the name * `click_completion_tools` moved from `morpheus/utils` to `morpheus/cli` * Remove unused `morpheus.utils.type_utils.mirror_args` method * Remove duplicate `get_log_levels` & `parse_log_level` methods from `morpheus.utils.logger` which also exist in `morpheus.cli.utils` fixes #624 TODO: - [x] Build sphinx make sure new docs all look good - [x] Test/verify changes in DFP - [x] Test/verify changes in visualization Authors: - David Gardner (https://github.com/dagardner-nv) - Bhargav Suryadevara (https://github.com/bsuryadevara) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #628
* DFP guide split into two docs. * Remove usage of Latin acronyms (ie, eg) with "for example" * Replace & with "and" * Other misc grammar fixes Work performed by Zenobia. fixes #569 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #637
Updating the Code of Conduct to follow the rest of RAPIDS. Closes #521 Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: #640
raykallen
reviewed
Jan 27, 2023
efajardo-nv
reviewed
Jan 27, 2023
efajardo-nv
reviewed
Jan 27, 2023
efajardo-nv
reviewed
Jan 27, 2023
For HTML, move the ref from 22.09 to 23.01. Moved the DFP starter Dockerfile ARG to 22.11. Fixes #576 Authors: - Pete MacKinnon (https://github.com/pdmack) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) URL: #642
Fixes #644 Authors: - Pete MacKinnon (https://github.com/pdmack) Approvers: - Eli Fajardo (https://github.com/efajardo-nv) - Michael Demoret (https://github.com/mdemoret-nv) URL: #645
Merged
Closes #486 Closes #251 Authors: - Eli Fajardo (https://github.com/efajardo-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Devin Robison (https://github.com/drobison00) - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) URL: #617
Addresses PR comments in #646 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Devin Robison (https://github.com/drobison00) - Eli Fajardo (https://github.com/efajardo-nv) - Pete MacKinnon (https://github.com/pdmack) URL: #652
Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Devin Robison (https://github.com/drobison00) - Eli Fajardo (https://github.com/efajardo-nv) URL: #653
Fixes #647 Authors: - Pete MacKinnon (https://github.com/pdmack) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #648
Includes various fixes for the following: 1. Fixing copyright headers 2. Fixing python style checks which were missing file exclusions 3. Fixing python style 4. Fixing cpp style 5. Fixing release container build Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Devin Robison (https://github.com/drobison00) - David Gardner (https://github.com/dagardner-nv) - Bartley Richardson (https://github.com/BartleyR) - Tad ZeMicheal (https://github.com/tzemicheal) - https://github.com/raykallen - Bhargav Suryadevara (https://github.com/bsuryadevara) URL: #654
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-23.01andv23.01releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-23.01until release (merging of this PR).What is the purpose of this PR?
branch-23.01intomainfor the release