From ad5bba2a2a8fd27b47d7efc92993674cbf7fa6f8 Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Mon, 3 Feb 2025 16:44:41 +1300 Subject: [PATCH 01/10] test helper guide --- docs/docsite/extra-docs.yml | 1 + docs/docsite/rst/guide_test_helper.rst | 383 +++++++++++++++++++++++++ 2 files changed, 384 insertions(+) create mode 100644 docs/docsite/rst/guide_test_helper.rst diff --git a/docs/docsite/extra-docs.yml b/docs/docsite/extra-docs.yml index f73d0fe0128..349d090e804 100644 --- a/docs/docsite/extra-docs.yml +++ b/docs/docsite/extra-docs.yml @@ -20,3 +20,4 @@ sections: - guide_vardict - guide_cmdrunner - guide_modulehelper + - guide_test_helper diff --git a/docs/docsite/rst/guide_test_helper.rst b/docs/docsite/rst/guide_test_helper.rst new file mode 100644 index 00000000000..4eb56d74096 --- /dev/null +++ b/docs/docsite/rst/guide_test_helper.rst @@ -0,0 +1,383 @@ +.. + Copyright (c) Ansible Project + GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) + SPDX-License-Identifier: GPL-3.0-or-later + +.. _ansible_collections.community.general.docsite.guide_test_helper: + +Test Helper Guide +================= + +Introduction +^^^^^^^^^^^^ + +The test helper was written to reduce the boilerplate code used in unit tests for modules. +It was originally written to handle tests of modules that run external commands using ``AnsibleModule.run_command()``. +At the time of writing (Feb 2025) that remains the only type of tests you can use +test Helper for, but it aims to provide support for other types of interactions. + +Until now, there are many different ways to implement unit tests that validate a module based on the execution of external commands. See some examples: + +* `test_apk.py `_ - A very simple one +* `test_bootc_manage.py `_ - + This one has more test cases, but do notice how the code is repeated amongst them. +* `test_modprobe.py `_ - + This one has 15 tests in it, but to achieve that it declares 8 classes repeating quite a lot of code. + +As you can notice, there is no consistency in the way these tests are executed - +they all do the same thing eventually, but each one is written in a very distinct way. + +The test Helper aims to: + +* provide a consistent idiom to define unit tests +* reduce the code to a bare minimal ... +* ... and making tests defined mostly as data +* allow the test cases definition to be expressed not only as a Python data structure but also as YAML content + +Quickstart +"""""""""" + +To use test helper, your test module will need only a bare minimal of code: + +.. code-block:: python + + # tests/unit/plugin/modules/test_ansible_module.py + from ansible_collections.community.general.plugins.modules import ansible_module + from .helper import Helper, RunCommandMock + + + Helper.from_module(ansible_module, __name__, mocks=[RunCommandMock]) + +Then, in the test specification file, you have: + +.. code-block:: yaml + + # tests/unit/plugin/modules/test_ansible_module.yaml + test_cases: + - id: test_ansible_module + flags: + diff: true + input: + state: present + name: Roger the Shrubber + output: + shrubbery: + looks: notice + price: not too expensive + changed: true + diff: + before: + shrubbery: null + after: + shrubbery: + looks: notice + price: not too expensive + mocks: + run_command: + - command: [/testbin/shrubber, --version] + rc: 0 + out: "2.80.0\n" + err: '' + - command: [/testbin/shrubber, --make-shrubbery] + rc: 0 + out: 'Shrubbery created' + err: '' + +.. note:: + + If you prefer to pick a different YAML file for the test cases, or if you prefer to define them in plain Python, + you can use the convenience methods ``Helper.from_file()`` and ``Helper.from_spec()``, respectively. + See more details below. + + +Using Test Helper +^^^^^^^^^^^^^^^^^ + +Test Module +""""""""""" + +The test helper is **strictly for unit tests**. To use it, you import the ``Helper`` class. +As mentioned in different parts of this guide, there are three different mechanisms to load the test cases. + +.. seealso:: + + See the Helper class reference below for API details on the three different mechanisms. + + +The easies and most recommended way of using test Helper is literally the example shown. +See a real world example at +`test_gconftool2.py `_. + +The ``from_module()`` method will pick the filename of the test module up (in the example above, ``tests/unit/plugins/modules/test_gconftool2.py``) +and it will search for ``tests/unit/plugins/modules/test_gconftool2.yaml`` (or ``.yml`` if that is not found). +In that file it will expect to find the test specification expressed in YAML format, conforming to the structure described below LINK LINK LINK. + +If you prefer to read the test specifications a different file path, use ``from_file()`` passing the file handle for the YAML file. + +And, if for any reason you prefer or need to pass the data structure rather than dealing with YAML files, use +the ``from_spec()`` method. +A real world example for that can be found at +`test_snap.py `_. + + +Test Specification +"""""""""""""""""" + +The strucutre of the test specification, in YAML is (excerpt from ``test_gio_mime.yaml``): + +.. code-block:: yaml + + --- + anchors: + environ: &env-def {environ_update: {LANGUAGE: C, LC_ALL: C}, check_rc: true} + test_cases: + - id: test_set_handler + input: + handler: google-chrome.desktop + mime_type: x-scheme-handler/http + output: + handler: google-chrome.desktop + changed: true + mocks: + run_command: + - command: [/testbin/gio, --version] + environ: *env-def + rc: 0 + out: "2.80.0\n" + err: '' + - command: [/testbin/gio, mime, x-scheme-handler/http] + environ: *env-def + rc: 0 + out: '' + err: > + No default applications for “x-scheme-handler/http” + - command: [/testbin/gio, mime, x-scheme-handler/http, google-chrome.desktop] + environ: *env-def + rc: 0 + out: "Set google-chrome.desktop as the default for x-scheme-handler/http\n" + err: '' + - id: test_set_handler_check + input: + handler: google-chrome.desktop + mime_type: x-scheme-handler/http + output: + handler: google-chrome.desktop + changed: true + stdout: Module executed in check mode + diff: + before: + handler: null + after: + handler: google-chrome.desktop + flags: + check: true + diff: true + mocks: + run_command: + - command: [/testbin/gio, --version] + environ: *env-def + rc: 0 + out: "2.80.0\n" + err: '' + - command: [/testbin/gio, mime, x-scheme-handler/http] + environ: *env-def + rc: 0 + out: '' + err: > + No default applications for “x-scheme-handler/http” + +Top level +--------- + +At the top level there are two accepted keys: + +- ``anchors`` (*dict, Optional*) + Placeholder for you to define YAML anchors that can be repeated in the test cases. + Its contents are never accessed directly by test Helper. +- ``test_cases`` (*list, Mandatory*) + List of test cases, see below for definition. + +Test cases +---------- + +You write the test cases with five elements: + +- ``id`` (*str, Mandatory*) + Used to identify the test case. +- ``flags`` (*dict, Optional*) + Flags controling the behavior of the test case. Accepted flags: + + * ``check`` (*bool, Optional*): set to ``true`` if the module is to be executed in **check mode**. + * ``diff`` (*bool, Optional*): set to ``true`` if the module is to be executed in **diff mode**. + * ``skip`` (*str, Optional*): set the test case to be skipped, providing the message for ``pytest.skip()``. + * ``xfail`` (*str, Optional*): set the test case to expect failure, providing the message for ``pytest.xfail()``. +- ``input`` (*dict, Optional*) + Parameters for the Ansible module, it can be empty. +- ``output`` (*dict, Optional*) + Expected return values from the Ansible module. + All RV names are used here are expected to be found in the module output, but not all RVs in the output must be here. + It can include special RVs such as ``changed`` and ``diff``. + It can be empty. +* ``mocks`` (*dict, Optional*) + Mocked interactions, ``run_command`` being the only one supported for now. + Each key in this dictionary refers to one subclass of ``TestCaseMock`` (see more below) and contains a list of the interactions for that ``TestCaseMock``. + All keys are expected to be named using snake case, as in ``run_command``. + The Python class supporting the test case mock is constructed by converting the snake case name to a + camel case name with suffix ``Mock``, so for example ``run_command`` becomes ``RunCommandMock``. + The test will fail if the Ansible module make a number of interactions different from what is specififed in the test case. + The structure for that specification is dependent on the implementing class, see details below. + +TestCaseMocks Specifications +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +RunCommandMock Specification +"""""""""""""""""""""""""""" + +For each interaction the structure is as follows: + +- ``command`` (*list OR str, Mandatory*) + The command that is expected to be executed by the module. It corresponds to the parameter ``args`` of the ``AnsibleModule.run_command()`` call. + It can be either a list or a string, though the list form is generally recommended. +- ``environ`` (*dict, Mandatory*) + All other parameters passed to the ``AnsibleModule.run_command()`` call. + Most commonly used are ``environ_update`` and ``check_rc``. + Must include all parameters the Ansible module uses in the ``AnsibleModule.run_command()`` call, otherwise the test will fail. +- ``rc`` (*int, Mandatory*) + The return code for the command execution. + As per usual in bash scripting, a value of ``0`` means success, whereas any other number is an error code. +- ``out`` (*str, Mandatory*) + The *stdout* result of the command execution, as one single string containing zero or more lines. +- ``err`` (*str, Mandatory*) + The *stderr* result of the command execution, as one single string containing zero or more lines. + + +Test Helper Reference +^^^^^^^^^^^^^^^^^^^^^ + +.. py:module:: .helper + + .. py:class:: Helper + + A class to encapsulate unit tests. + + .. py:staticmethod:: from_spec(ansible_module, test_module, test_spec, mocks=None) + + Creates a Helper instance from a given test specification. + + :param ansible_module: The Ansible module to be tested. + :type ansible_module: module + :param test_module: The test module. + :type test_module: module + :param test_spec: The test specification. + :type test_spec: dict + :param mocks: List of ``TestCaseMocks`` to be used during testing. Currently only ``RunCommandMock`` exists. + :type mocks: list or None + :return: A test Helper instance. + :rtype: Helper + + Example usage of ``from_spec()``: + + .. code-block:: python + + import sys + + from ansible_collections.community.general.plugins.modules import ansible_module + from .helper import Helper, RunCommandMock + + TEST_SPEC = dict( + test_cases=[ + ... + ] + ) + + helper = Helper.from_spec(ansible_module, sys.modules[__name__], TEST_SPEC, mocks=[RunCommandMock]) + + .. py:staticmethod:: from_file(ansible_module, test_module, test_spec_filehandle, mocks=None) + + Creates a Helper instance from a test specification file. + + :param ansible_module: The Ansible module to be tested. + :type ansible_module: module + :param test_module: The test module. + :type test_module: module + :param test_spec_filehandle: A file handle to an file stream handle providing the test specification in YAML format. + :type test_spec_filehandle: file + :param mocks: List of ``TestCaseMocks`` to be used during testing. Currently only ``RunCommandMock`` exists. + :type mocks: list or None + :return: A test Helper instance. + :rtype: Helper + + Example usage of ``from_file()``: + + .. code-block:: python + + import sys + + from ansible_collections.community.general.plugins.modules import ansible_module + from .helper import Helper, RunCommandMock + + with open("test_spec.yaml", "r") as test_spec_filehandle: + helper = Helper.from_file(ansible_module, sys.modules[__name__], test_spec_filehandle, mocks=[RunCommandMock]) + + .. py:staticmethod:: from_module(ansible_module, test_module_name, mocks=None) + + Creates a test helper instance from a given Ansible module and test module. + + :param ansible_module: The Ansible module to be tested. + :type ansible_module: module + :param test_module_name: The name of the test module. It works if passed ``__name__``. + :type test_module_name: str + :param mocks: List of ``TestCaseMocks`` to be used during testing. Currently only ``RunCommandMock`` exists. + :type mocks: list or None + :return: A test Helper instance. + :rtype: Helper + + Example usage of ``from_module()``: + + .. code-block:: python + + from ansible_collections.community.general.plugins.modules import ansible_module + from .helper import Helper, RunCommandMock + + # Example usage + helper = Helper.from_module(ansible_module, __name__, mocks=[RunCommandMock]) + + +Creating TestCaseMocks +^^^^^^^^^^^^^^^^^^^^^^ + +To create a new ``TestCaseMock`` you must extend that class and implement the relevant parts: + +.. code-block:: python + class ShrubberyMock(TestCaseMock): + # this name is mandatory, it is the name used in the test specification + name = "shrubbery" + + def setup(self, mocker): + # perform setup, commonly using mocker to patch some other piece of code + ... + + def check(self, test_case, results): + # verify the tst execution met the expectations of the test case + # for example the function was called as many times as it should + + def fixtures(self): + # returns a dict mapping names to pytest fixtures that should be used for the test case + # for example, in RunCommandMock it creates a fixture that patches AnsibleModule.get_bin_path + + +Caveats +^^^^^^^ + +Known issues/opportunities for improvement: + +* Only one ``Helper`` per test module: Test Helper injects a test function with a fixed name into the module's namespace, + so placing a second ``Helper`` instance is going to overwrite the function created by the first one. +* Order of elements in module's namespace is not consistent across executions in Python 3.5, so if adding more tests to the test module + might make Test Helper add its function before or after the other test functions. + In the community.general collection the CI processes uses ``pytest-xdist`` to paralellize and distribute the tests, + and it requires the order of the tests to be consistent. + + + +.. versionadded:: 7.5.0 From df72bb70a9f651a4223806911628ad3643f0ba67 Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Mon, 3 Feb 2025 19:22:32 +1300 Subject: [PATCH 02/10] small fixes --- docs/docsite/rst/guide_test_helper.rst | 72 ++++++++++++++------------ 1 file changed, 38 insertions(+), 34 deletions(-) diff --git a/docs/docsite/rst/guide_test_helper.rst b/docs/docsite/rst/guide_test_helper.rst index 4eb56d74096..9b61157e312 100644 --- a/docs/docsite/rst/guide_test_helper.rst +++ b/docs/docsite/rst/guide_test_helper.rst @@ -123,68 +123,68 @@ A real world example for that can be found at Test Specification """""""""""""""""" -The strucutre of the test specification, in YAML is (excerpt from ``test_gio_mime.yaml``): +The strucutre of the test specification, in YAML is (excerpt from ``test_opkg.yaml``): .. code-block:: yaml --- anchors: - environ: &env-def {environ_update: {LANGUAGE: C, LC_ALL: C}, check_rc: true} + environ: &env-def {environ_update: {LANGUAGE: C, LC_ALL: C}, check_rc: false} test_cases: - - id: test_set_handler + - id: install_zlibdev input: - handler: google-chrome.desktop - mime_type: x-scheme-handler/http + name: zlib-dev + state: present output: - handler: google-chrome.desktop - changed: true + msg: installed 1 package(s) mocks: run_command: - - command: [/testbin/gio, --version] + - command: [/testbin/opkg, --version] environ: *env-def rc: 0 - out: "2.80.0\n" + out: '' err: '' - - command: [/testbin/gio, mime, x-scheme-handler/http] + - command: [/testbin/opkg, list-installed, zlib-dev] environ: *env-def rc: 0 out: '' - err: > - No default applications for “x-scheme-handler/http” - - command: [/testbin/gio, mime, x-scheme-handler/http, google-chrome.desktop] + err: '' + - command: [/testbin/opkg, install, zlib-dev] + environ: *env-def + rc: 0 + out: | + Installing zlib-dev (1.2.11-6) to root... + Downloading https://downloads.openwrt.org/releases/22.03.0/packages/mips_24kc/base/zlib-dev_1.2.11-6_mips_24kc.ipk + Installing zlib (1.2.11-6) to root... + Downloading https://downloads.openwrt.org/releases/22.03.0/packages/mips_24kc/base/zlib_1.2.11-6_mips_24kc.ipk + Configuring zlib. + Configuring zlib-dev. + err: '' + - command: [/testbin/opkg, list-installed, zlib-dev] environ: *env-def rc: 0 - out: "Set google-chrome.desktop as the default for x-scheme-handler/http\n" + out: | + zlib-dev - 1.2.11-6 err: '' - - id: test_set_handler_check + - id: install_zlibdev_present input: - handler: google-chrome.desktop - mime_type: x-scheme-handler/http + name: zlib-dev + state: present output: - handler: google-chrome.desktop - changed: true - stdout: Module executed in check mode - diff: - before: - handler: null - after: - handler: google-chrome.desktop - flags: - check: true - diff: true + msg: package(s) already present mocks: run_command: - - command: [/testbin/gio, --version] + - command: [/testbin/opkg, --version] environ: *env-def rc: 0 - out: "2.80.0\n" + out: '' err: '' - - command: [/testbin/gio, mime, x-scheme-handler/http] + - command: [/testbin/opkg, list-installed, zlib-dev] environ: *env-def rc: 0 - out: '' - err: > - No default applications for “x-scheme-handler/http” + out: | + zlib-dev - 1.2.11-6 + err: '' Top level --------- @@ -227,6 +227,7 @@ You write the test cases with five elements: The test will fail if the Ansible module make a number of interactions different from what is specififed in the test case. The structure for that specification is dependent on the implementing class, see details below. + TestCaseMocks Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -349,6 +350,7 @@ Creating TestCaseMocks To create a new ``TestCaseMock`` you must extend that class and implement the relevant parts: .. code-block:: python + class ShrubberyMock(TestCaseMock): # this name is mandatory, it is the name used in the test specification name = "shrubbery" @@ -360,10 +362,12 @@ To create a new ``TestCaseMock`` you must extend that class and implement the re def check(self, test_case, results): # verify the tst execution met the expectations of the test case # for example the function was called as many times as it should + ... def fixtures(self): # returns a dict mapping names to pytest fixtures that should be used for the test case # for example, in RunCommandMock it creates a fixture that patches AnsibleModule.get_bin_path + ... Caveats From 4ceafc90ac6ccc99f42680d09d28ccd6cdabfc05 Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Mon, 3 Feb 2025 19:33:52 +1300 Subject: [PATCH 03/10] add BOTMETA entry --- .github/BOTMETA.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/BOTMETA.yml b/.github/BOTMETA.yml index 1beb35c57b4..bcd90dfa9a7 100644 --- a/.github/BOTMETA.yml +++ b/.github/BOTMETA.yml @@ -1543,6 +1543,8 @@ files: maintainers: baldwinSPC nurfet-becirevic t0mk teebes docs/docsite/rst/guide_scaleway.rst: maintainers: $team_scaleway + docs/docsite/rst/guide_test_helper.rst: + maintainers: russoz docs/docsite/rst/guide_vardict.rst: maintainers: russoz docs/docsite/rst/test_guide.rst: From 84c1531be0582c69ce635d9dd020bf4a515aaafd Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Mon, 3 Feb 2025 19:51:03 +1300 Subject: [PATCH 04/10] another small fix --- docs/docsite/rst/guide_test_helper.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docsite/rst/guide_test_helper.rst b/docs/docsite/rst/guide_test_helper.rst index 9b61157e312..dd04db22b85 100644 --- a/docs/docsite/rst/guide_test_helper.rst +++ b/docs/docsite/rst/guide_test_helper.rst @@ -218,7 +218,7 @@ You write the test cases with five elements: All RV names are used here are expected to be found in the module output, but not all RVs in the output must be here. It can include special RVs such as ``changed`` and ``diff``. It can be empty. -* ``mocks`` (*dict, Optional*) +- ``mocks`` (*dict, Optional*) Mocked interactions, ``run_command`` being the only one supported for now. Each key in this dictionary refers to one subclass of ``TestCaseMock`` (see more below) and contains a list of the interactions for that ``TestCaseMock``. All keys are expected to be named using snake case, as in ``run_command``. From 5129bfcda55c16220f085c29a798e1af0c988e01 Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Mon, 3 Feb 2025 21:05:34 +1300 Subject: [PATCH 05/10] reformated parameters type for consistency with other guides --- docs/docsite/rst/guide_test_helper.rst | 60 +++++++++++++------------- 1 file changed, 29 insertions(+), 31 deletions(-) diff --git a/docs/docsite/rst/guide_test_helper.rst b/docs/docsite/rst/guide_test_helper.rst index dd04db22b85..82ab973a9f8 100644 --- a/docs/docsite/rst/guide_test_helper.rst +++ b/docs/docsite/rst/guide_test_helper.rst @@ -191,35 +191,35 @@ Top level At the top level there are two accepted keys: -- ``anchors`` (*dict, Optional*) - Placeholder for you to define YAML anchors that can be repeated in the test cases. +- ``anchors: dict`` + Optional. Placeholder for you to define YAML anchors that can be repeated in the test cases. Its contents are never accessed directly by test Helper. -- ``test_cases`` (*list, Mandatory*) - List of test cases, see below for definition. +- ``test_cases: list`` + Mandatory. List of test cases, see below for definition. Test cases ---------- You write the test cases with five elements: -- ``id`` (*str, Mandatory*) - Used to identify the test case. -- ``flags`` (*dict, Optional*) - Flags controling the behavior of the test case. Accepted flags: - - * ``check`` (*bool, Optional*): set to ``true`` if the module is to be executed in **check mode**. - * ``diff`` (*bool, Optional*): set to ``true`` if the module is to be executed in **diff mode**. - * ``skip`` (*str, Optional*): set the test case to be skipped, providing the message for ``pytest.skip()``. - * ``xfail`` (*str, Optional*): set the test case to expect failure, providing the message for ``pytest.xfail()``. -- ``input`` (*dict, Optional*) - Parameters for the Ansible module, it can be empty. -- ``output`` (*dict, Optional*) - Expected return values from the Ansible module. +- ``id: str`` + Mandatory. Used to identify the test case. +- ``flags: dict`` + Optional. Flags controling the behavior of the test case. All flags are optional. Accepted flags: + + * ``check: bool``: set to ``true`` if the module is to be executed in **check mode**. + * ``diff: bool``: set to ``true`` if the module is to be executed in **diff mode**. + * ``skip: str``: set the test case to be skipped, providing the message for ``pytest.skip()``. + * ``xfail: str``: set the test case to expect failure, providing the message for ``pytest.xfail()``. +- ``input: dict`` + Optional. Parameters for the Ansible module, it can be empty. +- ``output: dict`` + Optional. Expected return values from the Ansible module. All RV names are used here are expected to be found in the module output, but not all RVs in the output must be here. It can include special RVs such as ``changed`` and ``diff``. It can be empty. -- ``mocks`` (*dict, Optional*) - Mocked interactions, ``run_command`` being the only one supported for now. +- ``mocks: dict`` + Optional. Mocked interactions, ``run_command`` being the only one supported for now. Each key in this dictionary refers to one subclass of ``TestCaseMock`` (see more below) and contains a list of the interactions for that ``TestCaseMock``. All keys are expected to be named using snake case, as in ``run_command``. The Python class supporting the test case mock is constructed by converting the snake case name to a @@ -236,20 +236,20 @@ RunCommandMock Specification For each interaction the structure is as follows: -- ``command`` (*list OR str, Mandatory*) - The command that is expected to be executed by the module. It corresponds to the parameter ``args`` of the ``AnsibleModule.run_command()`` call. +- ``command: Union[list, str]`` + Mandatory. The command that is expected to be executed by the module. It corresponds to the parameter ``args`` of the ``AnsibleModule.run_command()`` call. It can be either a list or a string, though the list form is generally recommended. -- ``environ`` (*dict, Mandatory*) - All other parameters passed to the ``AnsibleModule.run_command()`` call. +- ``environ: dict`` + Mandatory. All other parameters passed to the ``AnsibleModule.run_command()`` call. Most commonly used are ``environ_update`` and ``check_rc``. Must include all parameters the Ansible module uses in the ``AnsibleModule.run_command()`` call, otherwise the test will fail. -- ``rc`` (*int, Mandatory*) - The return code for the command execution. +- ``rc: int`` + Mandatory. The return code for the command execution. As per usual in bash scripting, a value of ``0`` means success, whereas any other number is an error code. -- ``out`` (*str, Mandatory*) - The *stdout* result of the command execution, as one single string containing zero or more lines. -- ``err`` (*str, Mandatory*) - The *stderr* result of the command execution, as one single string containing zero or more lines. +- ``out: str`` + Mandatory. The *stdout* result of the command execution, as one single string containing zero or more lines. +- ``err: str`` + Mandatory. The *stderr* result of the command execution, as one single string containing zero or more lines. Test Helper Reference @@ -382,6 +382,4 @@ Known issues/opportunities for improvement: In the community.general collection the CI processes uses ``pytest-xdist`` to paralellize and distribute the tests, and it requires the order of the tests to be consistent. - - .. versionadded:: 7.5.0 From 99b717656817d62004da73e1a1c580f102cf5790 Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Sat, 15 Feb 2025 18:28:27 +1300 Subject: [PATCH 06/10] adjust for renaming the helper to UTHelper --- docs/docsite/extra-docs.yml | 2 +- ...ide_test_helper.rst => guide_uthelper.rst} | 83 +++++++++---------- 2 files changed, 42 insertions(+), 43 deletions(-) rename docs/docsite/rst/{guide_test_helper.rst => guide_uthelper.rst} (84%) diff --git a/docs/docsite/extra-docs.yml b/docs/docsite/extra-docs.yml index 349d090e804..156e93309d6 100644 --- a/docs/docsite/extra-docs.yml +++ b/docs/docsite/extra-docs.yml @@ -20,4 +20,4 @@ sections: - guide_vardict - guide_cmdrunner - guide_modulehelper - - guide_test_helper + - guide_uthelper diff --git a/docs/docsite/rst/guide_test_helper.rst b/docs/docsite/rst/guide_uthelper.rst similarity index 84% rename from docs/docsite/rst/guide_test_helper.rst rename to docs/docsite/rst/guide_uthelper.rst index 82ab973a9f8..ac6146a7ed9 100644 --- a/docs/docsite/rst/guide_test_helper.rst +++ b/docs/docsite/rst/guide_uthelper.rst @@ -3,18 +3,18 @@ GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) SPDX-License-Identifier: GPL-3.0-or-later -.. _ansible_collections.community.general.docsite.guide_test_helper: +.. _ansible_collections.community.general.docsite.guide_uthelper: -Test Helper Guide -================= +UTHelper Guide +============== Introduction ^^^^^^^^^^^^ -The test helper was written to reduce the boilerplate code used in unit tests for modules. +``UTHelper`` was written to reduce the boilerplate code used in unit tests for modules. It was originally written to handle tests of modules that run external commands using ``AnsibleModule.run_command()``. At the time of writing (Feb 2025) that remains the only type of tests you can use -test Helper for, but it aims to provide support for other types of interactions. +``UTHelper`` for, but it aims to provide support for other types of interactions. Until now, there are many different ways to implement unit tests that validate a module based on the execution of external commands. See some examples: @@ -27,26 +27,26 @@ Until now, there are many different ways to implement unit tests that validate a As you can notice, there is no consistency in the way these tests are executed - they all do the same thing eventually, but each one is written in a very distinct way. -The test Helper aims to: +``UTHelper`` aims to: * provide a consistent idiom to define unit tests -* reduce the code to a bare minimal ... -* ... and making tests defined mostly as data +* reduce the code to a bare minimal, and +* define tests as data instead * allow the test cases definition to be expressed not only as a Python data structure but also as YAML content Quickstart """""""""" -To use test helper, your test module will need only a bare minimal of code: +To use UTHelper, your test module will need only a bare minimal of code: .. code-block:: python # tests/unit/plugin/modules/test_ansible_module.py from ansible_collections.community.general.plugins.modules import ansible_module - from .helper import Helper, RunCommandMock + from .uthelper import UTHelper, RunCommandMock - Helper.from_module(ansible_module, __name__, mocks=[RunCommandMock]) + UTHelper.from_module(ansible_module, __name__, mocks=[RunCommandMock]) Then, in the test specification file, you have: @@ -62,7 +62,7 @@ Then, in the test specification file, you have: name: Roger the Shrubber output: shrubbery: - looks: notice + looks: nice price: not too expensive changed: true diff: @@ -70,7 +70,7 @@ Then, in the test specification file, you have: shrubbery: null after: shrubbery: - looks: notice + looks: nice price: not too expensive mocks: run_command: @@ -86,25 +86,25 @@ Then, in the test specification file, you have: .. note:: If you prefer to pick a different YAML file for the test cases, or if you prefer to define them in plain Python, - you can use the convenience methods ``Helper.from_file()`` and ``Helper.from_spec()``, respectively. + you can use the convenience methods ``UTHelper.from_file()`` and ``UTHelper.from_spec()``, respectively. See more details below. -Using Test Helper -^^^^^^^^^^^^^^^^^ +Using ``UTHelper`` +^^^^^^^^^^^^^^^^^^ Test Module """"""""""" -The test helper is **strictly for unit tests**. To use it, you import the ``Helper`` class. +``UTHelper`` is **strictly for unit tests**. To use it, you import the ``.uthelper.UTHelper`` class. As mentioned in different parts of this guide, there are three different mechanisms to load the test cases. .. seealso:: - See the Helper class reference below for API details on the three different mechanisms. + See the UTHelper class reference below for API details on the three different mechanisms. -The easies and most recommended way of using test Helper is literally the example shown. +The easies and most recommended way of using ``UTHelper`` is literally the example shown. See a real world example at `test_gconftool2.py `_. @@ -114,8 +114,7 @@ In that file it will expect to find the test specification expressed in YAML for If you prefer to read the test specifications a different file path, use ``from_file()`` passing the file handle for the YAML file. -And, if for any reason you prefer or need to pass the data structure rather than dealing with YAML files, use -the ``from_spec()`` method. +And, if for any reason you prefer or need to pass the data structure rather than dealing with YAML files, use the ``from_spec()`` method. A real world example for that can be found at `test_snap.py `_. @@ -252,18 +251,18 @@ For each interaction the structure is as follows: Mandatory. The *stderr* result of the command execution, as one single string containing zero or more lines. -Test Helper Reference -^^^^^^^^^^^^^^^^^^^^^ +``UTHelper`` Reference +^^^^^^^^^^^^^^^^^^^^^^ -.. py:module:: .helper +.. py:module:: .uthelper - .. py:class:: Helper + .. py:class:: UTHelper A class to encapsulate unit tests. .. py:staticmethod:: from_spec(ansible_module, test_module, test_spec, mocks=None) - Creates a Helper instance from a given test specification. + Creates an ``UTHelper`` instance from a given test specification. :param ansible_module: The Ansible module to be tested. :type ansible_module: module @@ -273,8 +272,8 @@ Test Helper Reference :type test_spec: dict :param mocks: List of ``TestCaseMocks`` to be used during testing. Currently only ``RunCommandMock`` exists. :type mocks: list or None - :return: A test Helper instance. - :rtype: Helper + :return: An ``UTHelper`` instance. + :rtype: UTHelper Example usage of ``from_spec()``: @@ -283,7 +282,7 @@ Test Helper Reference import sys from ansible_collections.community.general.plugins.modules import ansible_module - from .helper import Helper, RunCommandMock + from .uthelper import UTHelper, RunCommandMock TEST_SPEC = dict( test_cases=[ @@ -291,11 +290,11 @@ Test Helper Reference ] ) - helper = Helper.from_spec(ansible_module, sys.modules[__name__], TEST_SPEC, mocks=[RunCommandMock]) + helper = UTHelper.from_spec(ansible_module, sys.modules[__name__], TEST_SPEC, mocks=[RunCommandMock]) .. py:staticmethod:: from_file(ansible_module, test_module, test_spec_filehandle, mocks=None) - Creates a Helper instance from a test specification file. + Creates an ``UTHelper`` instance from a test specification file. :param ansible_module: The Ansible module to be tested. :type ansible_module: module @@ -305,8 +304,8 @@ Test Helper Reference :type test_spec_filehandle: file :param mocks: List of ``TestCaseMocks`` to be used during testing. Currently only ``RunCommandMock`` exists. :type mocks: list or None - :return: A test Helper instance. - :rtype: Helper + :return: An ``UTHelper`` instance. + :rtype: UTHelper Example usage of ``from_file()``: @@ -315,14 +314,14 @@ Test Helper Reference import sys from ansible_collections.community.general.plugins.modules import ansible_module - from .helper import Helper, RunCommandMock + from .uthelper import UTHelper, RunCommandMock with open("test_spec.yaml", "r") as test_spec_filehandle: - helper = Helper.from_file(ansible_module, sys.modules[__name__], test_spec_filehandle, mocks=[RunCommandMock]) + helper = UTHelper.from_file(ansible_module, sys.modules[__name__], test_spec_filehandle, mocks=[RunCommandMock]) .. py:staticmethod:: from_module(ansible_module, test_module_name, mocks=None) - Creates a test helper instance from a given Ansible module and test module. + Creates an ``UTHelper`` instance from a given Ansible module and test module. :param ansible_module: The Ansible module to be tested. :type ansible_module: module @@ -330,18 +329,18 @@ Test Helper Reference :type test_module_name: str :param mocks: List of ``TestCaseMocks`` to be used during testing. Currently only ``RunCommandMock`` exists. :type mocks: list or None - :return: A test Helper instance. - :rtype: Helper + :return: An ``UTHelper`` instance. + :rtype: UTHelper Example usage of ``from_module()``: .. code-block:: python from ansible_collections.community.general.plugins.modules import ansible_module - from .helper import Helper, RunCommandMock + from .uthelper import UTHelper, RunCommandMock # Example usage - helper = Helper.from_module(ansible_module, __name__, mocks=[RunCommandMock]) + helper = UTHelper.from_module(ansible_module, __name__, mocks=[RunCommandMock]) Creating TestCaseMocks @@ -375,8 +374,8 @@ Caveats Known issues/opportunities for improvement: -* Only one ``Helper`` per test module: Test Helper injects a test function with a fixed name into the module's namespace, - so placing a second ``Helper`` instance is going to overwrite the function created by the first one. +* Only one ``UTHelper`` per test module: UTHelper injects a test function with a fixed name into the module's namespace, + so placing a second ``UTHelper`` instance is going to overwrite the function created by the first one. * Order of elements in module's namespace is not consistent across executions in Python 3.5, so if adding more tests to the test module might make Test Helper add its function before or after the other test functions. In the community.general collection the CI processes uses ``pytest-xdist`` to paralellize and distribute the tests, From c40bc7bd6913151ecfec6fa9d7b4b5b895213c21 Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Sat, 15 Feb 2025 18:36:24 +1300 Subject: [PATCH 07/10] add botmeta entry --- .github/BOTMETA.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/BOTMETA.yml b/.github/BOTMETA.yml index bcd90dfa9a7..2f5d6be180c 100644 --- a/.github/BOTMETA.yml +++ b/.github/BOTMETA.yml @@ -1545,6 +1545,8 @@ files: maintainers: $team_scaleway docs/docsite/rst/guide_test_helper.rst: maintainers: russoz + docs/docsite/rst/guide_uthelper.rst: + maintainers: russoz docs/docsite/rst/guide_vardict.rst: maintainers: russoz docs/docsite/rst/test_guide.rst: From fb35428caeb9942993ebd8bb444d85af3849673b Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Sat, 15 Feb 2025 18:38:02 +1300 Subject: [PATCH 08/10] remove the old-named botmeta entry --- .github/BOTMETA.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.github/BOTMETA.yml b/.github/BOTMETA.yml index 2f5d6be180c..f0d1b21b8f5 100644 --- a/.github/BOTMETA.yml +++ b/.github/BOTMETA.yml @@ -1543,8 +1543,6 @@ files: maintainers: baldwinSPC nurfet-becirevic t0mk teebes docs/docsite/rst/guide_scaleway.rst: maintainers: $team_scaleway - docs/docsite/rst/guide_test_helper.rst: - maintainers: russoz docs/docsite/rst/guide_uthelper.rst: maintainers: russoz docs/docsite/rst/guide_vardict.rst: From c9aa18b46fd4a8cd27705bc067f3e641f69f84fb Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Sat, 15 Feb 2025 19:08:00 +1300 Subject: [PATCH 09/10] improved wording --- docs/docsite/rst/guide_uthelper.rst | 100 +++++++++++++++------------- 1 file changed, 55 insertions(+), 45 deletions(-) diff --git a/docs/docsite/rst/guide_uthelper.rst b/docs/docsite/rst/guide_uthelper.rst index ac6146a7ed9..6d78c9762ab 100644 --- a/docs/docsite/rst/guide_uthelper.rst +++ b/docs/docsite/rst/guide_uthelper.rst @@ -122,7 +122,58 @@ A real world example for that can be found at Test Specification """""""""""""""""" -The strucutre of the test specification, in YAML is (excerpt from ``test_opkg.yaml``): +The structure of the test specification data is described below. + +Top level +--------- + +At the top level there are two accepted keys: + +- ``anchors: dict`` + Optional. Placeholder for you to define YAML anchors that can be repeated in the test cases. + Its contents are never accessed directly by test Helper. +- ``test_cases: list`` + Mandatory. List of test cases, see below for definition. + +Test cases +---------- + +You write the test cases with five elements: + +- ``id: str`` + Mandatory. Used to identify the test case. + +- ``flags: dict`` + Optional. Flags controling the behavior of the test case. All flags are optional. Accepted flags: + + * ``check: bool``: set to ``true`` if the module is to be executed in **check mode**. + * ``diff: bool``: set to ``true`` if the module is to be executed in **diff mode**. + * ``skip: str``: set the test case to be skipped, providing the message for ``pytest.skip()``. + * ``xfail: str``: set the test case to expect failure, providing the message for ``pytest.xfail()``. + +- ``input: dict`` + Optional. Parameters for the Ansible module, it can be empty. + +- ``output: dict`` + Optional. Expected return values from the Ansible module. + All RV names are used here are expected to be found in the module output, but not all RVs in the output must be here. + It can include special RVs such as ``changed`` and ``diff``. + It can be empty. + +- ``mocks: dict`` + Optional. Mocked interactions, ``run_command`` being the only one supported for now. + Each key in this dictionary refers to one subclass of ``TestCaseMock`` and its + structure is dictated by the ``TestCaseMock`` subclass implementation. + All keys are expected to be named using snake case, as in ``run_command``. + The ``TestCaseMock`` subclass is responsible for defining the name used in the test specification. + The structure for that specification is dependent on the implementing class. + See more details below for the implementation of ``RunCommandMock`` + +Example using YAML +------------------ + +We recommend you use ``UTHelper`` reading the test specifications from a YAML file. +See an example below of how one actually looks like (excerpt from ``test_opkg.yaml``): .. code-block:: yaml @@ -185,55 +236,15 @@ The strucutre of the test specification, in YAML is (excerpt from ``test_opkg.ya zlib-dev - 1.2.11-6 err: '' -Top level ---------- - -At the top level there are two accepted keys: - -- ``anchors: dict`` - Optional. Placeholder for you to define YAML anchors that can be repeated in the test cases. - Its contents are never accessed directly by test Helper. -- ``test_cases: list`` - Mandatory. List of test cases, see below for definition. - -Test cases ----------- - -You write the test cases with five elements: - -- ``id: str`` - Mandatory. Used to identify the test case. -- ``flags: dict`` - Optional. Flags controling the behavior of the test case. All flags are optional. Accepted flags: - - * ``check: bool``: set to ``true`` if the module is to be executed in **check mode**. - * ``diff: bool``: set to ``true`` if the module is to be executed in **diff mode**. - * ``skip: str``: set the test case to be skipped, providing the message for ``pytest.skip()``. - * ``xfail: str``: set the test case to expect failure, providing the message for ``pytest.xfail()``. -- ``input: dict`` - Optional. Parameters for the Ansible module, it can be empty. -- ``output: dict`` - Optional. Expected return values from the Ansible module. - All RV names are used here are expected to be found in the module output, but not all RVs in the output must be here. - It can include special RVs such as ``changed`` and ``diff``. - It can be empty. -- ``mocks: dict`` - Optional. Mocked interactions, ``run_command`` being the only one supported for now. - Each key in this dictionary refers to one subclass of ``TestCaseMock`` (see more below) and contains a list of the interactions for that ``TestCaseMock``. - All keys are expected to be named using snake case, as in ``run_command``. - The Python class supporting the test case mock is constructed by converting the snake case name to a - camel case name with suffix ``Mock``, so for example ``run_command`` becomes ``RunCommandMock``. - The test will fail if the Ansible module make a number of interactions different from what is specififed in the test case. - The structure for that specification is dependent on the implementing class, see details below. - - TestCaseMocks Specifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The ``TestCaseMock`` subclass is free to define the expected data structure. + RunCommandMock Specification """""""""""""""""""""""""""" -For each interaction the structure is as follows: +``RunCommandMock`` expects a ``list`` in which elements follow the structure: - ``command: Union[list, str]`` Mandatory. The command that is expected to be executed by the module. It corresponds to the parameter ``args`` of the ``AnsibleModule.run_command()`` call. @@ -368,7 +379,6 @@ To create a new ``TestCaseMock`` you must extend that class and implement the re # for example, in RunCommandMock it creates a fixture that patches AnsibleModule.get_bin_path ... - Caveats ^^^^^^^ From d8262edd7c48eadffb423d31442518ee68ce1cea Mon Sep 17 00:00:00 2001 From: Alexei Znamensky Date: Sat, 15 Feb 2025 19:09:21 +1300 Subject: [PATCH 10/10] improved wording --- docs/docsite/rst/guide_uthelper.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docsite/rst/guide_uthelper.rst b/docs/docsite/rst/guide_uthelper.rst index 6d78c9762ab..657ced66cfd 100644 --- a/docs/docsite/rst/guide_uthelper.rst +++ b/docs/docsite/rst/guide_uthelper.rst @@ -244,7 +244,7 @@ The ``TestCaseMock`` subclass is free to define the expected data structure. RunCommandMock Specification """""""""""""""""""""""""""" -``RunCommandMock`` expects a ``list`` in which elements follow the structure: +``RunCommandMock`` mocks can be specified with the key ``run_command`` and it expects a ``list`` in which elements follow the structure: - ``command: Union[list, str]`` Mandatory. The command that is expected to be executed by the module. It corresponds to the parameter ``args`` of the ``AnsibleModule.run_command()`` call.