Chaffelson · Nov 29, 2021 · Nov 29, 2021 · May 11, 2022 · May 12, 2022 · May 13, 2022
Showing 645 changed files with 259,281 additions and 33,638 deletions.
diff --git a/.gitignore b/.gitignore
@@ -157,3 +157,9 @@ fabric.properties
 
 # Stupid OSX exclusions
 /.vscode
+
+.DS_Store
+# Coverage reports
+coverage*
+.coverage
+htmlcov/
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -0,0 +1,36 @@
+# Read the Docs configuration file for Sphinx projects
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+    # You can also specify other tool versions:
+    # nodejs: "20"
+    # rust: "1.70"
+    # golang: "1.20"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+  # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs
+  # builder: "dirhtml"
+  # Fail on all warnings to avoid broken references
+  # fail_on_warning: true
+
+# Optionally build your docs in additional formats such as PDF and ePub
+# formats:
+#   - pdf
+#   - epub
+
+# Optional but recommended, declare the Python requirements required
+# to build your documentation
+# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+    - requirements: requirements.txt
+    - requirements: requirements_dev.txt
diff --git a/README.rst b/README.rst
@@ -60,6 +60,9 @@ The easiest way to install NiPyApi is with pip::
 
     # in bash
     pip install nipyapi
+    
+    # or with docker support for demos
+    pip install nipyapi[demo]
 
 You can set the config for your endpoints in the central config file::
 
@@ -93,12 +96,20 @@ Background and Documentation
 NiFi Version Support
 --------------------
 
-| Currently we are testing against NiFi versions 1.1.2 - 1.13.2, and NiFi-Registry versions 0.1.0 - 0.8.0.
+| Currently we are testing against NiFi versions 1.9.2 - 1.28.1, and NiFi-Registry versions 0.3.0 - 1.28.1.
+
+| We have also tested against the latest NiFi-2.2.0 release using the 1.x SDK and have found that basic functionality works as expected.
+| Apache NiFi offers no compatibility guarantees between major versions, and while many functions may work the same, you should test carefully for your specific use case.
+| In future we will create a specific branch of NiPyAPI for NiFi 2.x SDKs, and maintain separate NiFi 1.x and NiFi 2.x clients.
+
 | If you find a version compatibility problem please raise an `issue <https://github.com/Chaffelson/nipyapi/issues>`_
 
-Python Requirements
--------------------
+Python Support
+--------------
+
+| Python 2.7 or 3.9-12 supported, though other versions may work. 
+| We will shortly stop supporting Python2.
+| OSX M1 chips have had various issues with Requests and Certificates.
 
-| Python 2.7 or 3.5-8 supported, though other versions may work. *We will shortly stop supporting Python2*
-| Tested on AL2 and OSX 10.14.x - Windows automated testing not attempted
+| Tested on AL2023, developed on OSX 14+ and Windows 10 with Docker Desktop.
 | Outside of the standard Python modules, we make use of lxml, DeepDiff, ruamel.yaml and xmltodict in processing, and Docker for demo/tests.
diff --git a/docs/devnotes.rst b/docs/devnotes.rst
@@ -17,7 +17,7 @@ Decision Points
 Testing Notes
 -------------
 
-When running tests on new code, you are advised to run 'test_default' first, then 'test_regression', then finally 'test_security'.
+When running tests on new code, you are advised to run 'test_default' first, then 'test_regression', then finally 'test_ldap' and/or 'test_mtls'.
 Because of the way errors are propagated you may have code failures which cause a teardown which then fails because of security controls, which can then obscure the original error.
 
 
@@ -28,11 +28,56 @@ There is an Apache NiFi image available on Dockerhub::
 
     docker pull apache/nifi:latest
 
-There are a couple of configuration files for launching various Docker environment configurations in ./test_env_config for convenience.
+There are a couple of configuration files for launching various Docker environment configurations in resources/docker for convenience.
+
+Remote testing on AWS:AL3 with Visual Studio Code on OSX
+--------------------------------------------------------
+
+Instructions::
+
+    Deploy a t2.xlarge on EC2, preferably with an elastic IP
+    Add the machine as a remote on Visual Studio Code and Connect
+    Open up the console and install git so VSCode can clone the repo `sudo dnf install -y git`
+    Use the VSCode Source Control plugin to clone nipyapi https://github.com/Chaffelson/nipyapi.git
+    You can then open these notes in VSCode with the terminal for easy execution
+    Now install dependencies `sudo dnf install -y docker && sudo dnf groupinstall "Development Tools" -y`
+    Now ensure docker starts with the OS and gives your user access `sudo systemctl start docker && sudo systemctl enable docker && sudo usermod -a -G docker $USER`
+    Restart your terminal, or run `newgrp docker` to get Docker access permissions active
+    Install Pip `sudo dnf install python3-pip -y`
+    Instal docker compose `sudo curl -L "https://github.com/docker/compose/releases/download/v2.26.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose`
+    I recommend you install PyEnv to manage Python versions `sudo curl https://pyenv.run | bash`
+    Follow the instructions to set up your .bashrc
+    To build various versions of Python for testing you may also need `sudo dnf install bzip2-devel openssl-devel libffi-devel zlib-devel readline-devel sqlite-devel -y`
+    Install the latest supported version of Python for your main dev environment `pyenv install 3.9 2.7 3.12`
+    Set these versions as global in pyenv so tox can see them. Use the actual versions with the command `pyenv global 3.9.16 2.7.18 3.12.2`
+    You'll want to stand up the two sets of NiFi containers for testing. resources/docker/tox-full for default and regression tests, and resources/docker/secure for tests under auth.
+    You can switch between the tests by changing flags in tests/conftest.py around line 17.
+    Python3 can be tested automatically using Tox.
+    Python2 can be tested using the following steps within a Python2 virtualenv:
+    1. Install requirements: pip install -r requirements.txt
+    2. Install dev requirements: `pip install -r requirements_dev.txt`
+    3. Install package in editable mode with test support: `pip install -e .[test]`
+    4. Run tests: `pytest -v -s tests --tb=long -W ignore::urllib3.exceptions.InsecureRequestWarning`
+
+Setup Code Signing
+------------------
+
+If you want to sign and push code from your EC2 instance, you'll need to set up code signing. 
+Ensuring security of your keys is important, so please protect them with a good secret passphrase
+
+Instructions::
+
+    On your AL2023 instance, replace the default minimal gnupg package with the full one `sudo dnf install --allowerasing gnupg2-full`
+    Generate signing keys `gpg --full-generate-key`
+    Use the long key ID as your signingkey `git config --global user.signingkey <key here>`
+    git config --global commit.gpgsign true
+    Add the tty setting for gpg to your ~/.bashrc `export GPG_TTY=$(tty)`
 
 Remote Testing on Centos7
 -------------------------
 
+**Deprecated. Instructions kept for legacy reference.**
+
 Deploy a 4x16 or better on EC2 running Centos 7.5 or better, ssh in as root::
 
     yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
@@ -72,7 +117,7 @@ The NiFi and NiFi Registry REST API clients are generated using swagger-codegen,
 
 - the package manager for your OS
 - github: https://github.com/swagger-api/swagger-codegen
-- maven: http://central.maven.org/maven2/io/swagger/swagger-codegen-cli/2.3.1/swagger-codegen-cli-2.3.1.jar
+- maven: https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/2.3.1/swagger-codegen-cli-2.4.41.jar
 - pre-built Docker images on DockerHub (https://hub.docker.com/r/swaggerapi/swagger-codegen-cli/)
 
 In the examples below, we'll use Homebrew for macOS::
@@ -147,15 +192,14 @@ This assumes you have virtualenvwrapper, git, and appropriate python versions in
     python setup.py develop
     tox
     python setup.py test
-    python setup.py build_sphinx
+    Run `make html` in the docs subdir
     # check docs in build/sphinx/html/index.html
     python setup.py sdist bdist_wheel
     mktmpenv  # or pyenv virtualenvwrapper mktmpenv if using pyenv
     pip install path/to/nipyapi-0.3.1-py2.py3-none-any.whl  # for example
     # Run appropriate tests, such as usage tests etc.
     deactivate
     Push changes to Github
-    Check build on TravisCI
     Check dockerhub automated build
     # You may have to reactivate your original virtualenv
     twine upload dist/*

diff --git a/docs/history.rst b/docs/history.rst
@@ -2,6 +2,77 @@
 History
 =======
 
+0.22.0 (2025-03-25)
+--------------------
+
+| Updated low level clients for NiFi & Registry 1.28.1 release
+| Updates to supported Python versions, handing Template deprecation in NiFi-2.x, and Windows development support
+| Add support for mTLS for Nifi and Registry
+
+0.21.0 (2024-07-14)
+-------------------
+
+| Updated client for NiFi & Registry 1.27.0 release
+
+* Fix API model generator by @michaelarnauts in https://github.com/Chaffelson/nipyapi/pull/356
+* issue-360: handle -9 error messages better by @ottobackwards in https://github.com/Chaffelson/nipyapi/pull/361
+* Handle plain text response types so json values are correctly returned by @michaelarnauts in https://github.com/Chaffelson/nipyapi/pull/358
+* update clients to 1.27.0 by @Chaffelson in https://github.com/Chaffelson/nipyapi/pull/365
+* Simplified the use of setattr in recurse_flow, flatten, and list_all_by_kind methods in nipyapi/canvas.py. 
+* Added support for key_password in the Configuration class and its usage in nipyapi/nifi/rest.py and nipyapi/registry/rest.py. 
+* Fixed the method to retrieve HTTP headers in nipyapi/nifi/rest.py and nipyapi/registry/rest.py. 
+* Fixed issue #326 where the latest flow version was not being deployed by default
+* Updated pylintrc to match more modern python standards
+* Fixes nipyapi.nifi.ProcessGroupsApi.upload_process_group_with_http_info() incomplete #310
+* VersionedReportingTask added with appropriate functions
+* Move docker requirement to extras to avoid dependency install during standard usage
+* Set latest python3 version to 3.12
+* Deprecate usage of py.test in favour of newer pytest.
+* Update readme
+
+0.20.0 (2024-04-14)
+-------------------
+
+| Updated client for NiFi & Registry 1.23.2 release
+
+* Fix ruamel.yaml finally deprecating safe_loader
+* Hexoplon added ReportingTaskEntity to registered filters
+* ottobackwards fixed the root logger being setup by the libary
+* Generally updated test setup and several updated libraries complaining since last release
+
+0.19.1 (2022-08-08)
+-------------------
+
+| Updated client for NiFi & Registry 1.17.0 release
+
+* Update docker resources to the latest 1.17 container image
+* Fix issue in docker volume mounts where certificates were not readable in new versions of Docker
+* Fix issue in test_get_processor_type where new Twitter processor broke existing logic assumptions
+* Update readme to note issues with Python 3.10 and removing reference to python 3.5 testing
+
+0.19.0 (2022-05-13)
+-------------------
+
+| Updated client for NiFi & Registry 1.16.0 release
+
+* Modify utils.check_version to accept a default version to check against, and remove nested error when version check fails as it is overzealous
+* Update test file paths in test_utils in case someone runs tests as root which actually can write to fake devices and therefore break the tests
+* Added pylint exclusions for known but unimportant complaints
+* Added workaround for NiFi 9470 by providing default inherited_parameter_contexts per advice from Chris Sampson in NiPyAPI Issue #305
+* Updated NiFi and Registry versions in Docker compose configs to use latest 1.16.1 builds
+
+
+0.18.0 (2021-11-29)
+-------------------
+
+| Updated client for NiFi & Registry 1.15.0 release
+
+* Moved testing to secured single user by default for NiFi
+* Consolidated clients to the 1.15.0 release version for both NiFi and Registry
+* Updated logic to support changed working modes in tests
+* Corrected minor functional issues for 1.15 in login handling, SSL certs, Parameter Updates and Testing
+* Backwards compatibility and Regression tested as requiring no breaking changes
+
 0.17.1 (2021-10-21)
 -------------------
 

diff --git a/docs/nipyapi-docs/nipyapi.nifi.models.rst b/docs/nipyapi-docs/nipyapi.nifi.models.rst
@@ -188,38 +188,6 @@ nipyapi.nifi.models.batch\_size module
     :undoc-members:
     :show-inheritance:
 
-nipyapi.nifi.models.bucket module
----------------------------------
-
-.. automodule:: nipyapi.nifi.models.bucket
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-nipyapi.nifi.models.bucket\_dto module
---------------------------------------
-
-.. automodule:: nipyapi.nifi.models.bucket_dto
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-nipyapi.nifi.models.bucket\_entity module
------------------------------------------
-
-.. automodule:: nipyapi.nifi.models.bucket_entity
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-nipyapi.nifi.models.buckets\_entity module
-------------------------------------------
-
-.. automodule:: nipyapi.nifi.models.buckets_entity
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 nipyapi.nifi.models.bulletin\_board\_dto module
 -----------------------------------------------
 
@@ -1036,14 +1004,6 @@ nipyapi.nifi.models.peers\_entity module
     :undoc-members:
     :show-inheritance:
 
-nipyapi.nifi.models.permissions module
---------------------------------------
-
-.. automodule:: nipyapi.nifi.models.permissions
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 nipyapi.nifi.models.permissions\_dto module
 -------------------------------------------
 
@@ -1388,30 +1348,6 @@ nipyapi.nifi.models.queue\_size\_dto module
     :undoc-members:
     :show-inheritance:
 
-nipyapi.nifi.models.registry\_client\_entity module
----------------------------------------------------
-
-.. automodule:: nipyapi.nifi.models.registry_client_entity
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-nipyapi.nifi.models.registry\_clients\_entity module
-----------------------------------------------------
-
-.. automodule:: nipyapi.nifi.models.registry_clients_entity
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-nipyapi.nifi.models.registry\_dto module
-----------------------------------------
-
-.. automodule:: nipyapi.nifi.models.registry_dto
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 nipyapi.nifi.models.relationship\_dto module
 --------------------------------------------
 
@@ -1916,14 +1852,6 @@ nipyapi.nifi.models.versioned\_controller\_service module
     :undoc-members:
     :show-inheritance:
 
-nipyapi.nifi.models.versioned\_flow module
-------------------------------------------
-
-.. automodule:: nipyapi.nifi.models.versioned_flow
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
 nipyapi.nifi.models.versioned\_flow\_coordinates module
 -------------------------------------------------------
 
@@ -1948,13 +1876,6 @@ nipyapi.nifi.models.versioned\_flow\_entity module
     :undoc-members:
     :show-inheritance:
 
-nipyapi.nifi.models.versioned\_flow\_snapshot module
-----------------------------------------------------
-
-.. automodule:: nipyapi.nifi.models.versioned_flow_snapshot
-    :members:
-    :undoc-members:
-    :show-inheritance:
 
 nipyapi.nifi.models.versioned\_flow\_snapshot\_entity module
 ------------------------------------------------------------

diff --git a/docs/nipyapi-docs/nipyapi.rst b/docs/nipyapi-docs/nipyapi.rst
@@ -30,6 +30,14 @@ Config
     :undoc-members:
     :show-inheritance:
 
+Parameters
+----------
+
+.. automodule:: nipyapi.parameters
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 Security
 --------
 

diff --git a/nipyapi/__init__.py b/nipyapi/__init__.py
@@ -9,7 +9,7 @@
 
 __author__ = """Daniel Chaffelson"""
 __email__ = 'chaffelson@gmail.com'
-__version__ = '0.17.1'
+__version__ = '0.22.0'
 __all__ = ['canvas', 'system', 'templates', 'config', 'nifi', 'registry',
            'versioning', 'demo', 'utils', 'security', 'parameters']