Merge pull request #10 from qtc-de/develop

Prepare v1.4.0 Release

CHANGELOG.md

@@ -6,6 +6,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [1.4.0] - May 27, 2021
+
+### Added
+
+* Add support for running commands in [shell mode](/docs/README.md#worth-knowing)
+* Add logfile attribute for [tests and testers](/docs/README.md#logging)
+* Add support for custom [success and error strings](/docs/README.md##custom-strings)
+
+
 ## [1.3.2] - May 24, 2021
 
 ### Changed

bin/tricot

@@ -10,7 +10,7 @@ import tricot
 import argparse
 
 
-parser = argparse.ArgumentParser(description='''tricot v1.1.0 - a trivial command tester that allows you to verify that certain
+parser = argparse.ArgumentParser(description='''tricot v1.4.0 - a trivial command tester that allows you to verify that certain
                                                 commands or executables behave as expected. It uses .yml files for test
                                                 definitions and can be used from the command line or as a python library.''')
 
@@ -94,8 +94,7 @@ def initialize_logger(args: argparse.Namespace) -> None:
         tricot.Logger.set_verbosity(3)
 
     if args.log:
-        args.log.close()
-        tricot.Logger.set_logfile(args.log.name)
+        tricot.Logger.add_logfile(args.log)
 
 
 def load(files: list[io.TextIOWrapper]) -> None:
@@ -208,8 +207,12 @@ def main():
             sys.exit(tricot.constants.CONDITION_FORMAT_ERROR)
 
         except tricot.TricotException as e:
-            tricot.Logger.print('Encountered an error during test execution.', e=True)
+            tricot.Logger.print('Encountered an unexpected error.', e=True)
             tricot.Logger.print_blue(str(e), e=True)
+
+            if e.path:
+                tricot.Logger.print_mixed_yellow('Configuration file:', e.path, e=True)
+
             sys.exit(tricot.constants.TRICOT_EXCEPTION)
 
         except yaml.parser.ParserError as e:

docs/README.md

@@ -16,7 +16,9 @@ on it's own.
 - [Nesting Variables](#nesting-variables)
 - [Conditionals](#conditionals)
 - [Reusing Output](#reusing-output)
-- [Additional Command Line Switches](#additional-command-line-switches)
+- [Logging](#logging)
+- [Custom Strings](#custom-strings)
+- [Worth Knowing](#worth-knowing)
 
 
 ### Validator and Plugin List
@@ -515,12 +517,78 @@ Updating conditions is only allowed within of *tests* and not within *testers*.
 when all validators have run successfully. The ``on_error`` action triggers, if one or more validators failed.
 
 
-### Additional Command Line Switches
+### Logging
 
 ----
 
-Here is some more detailed explanation on some of *tricots* command line switches:
+*tricot* supports logging on *global*, *tester* and *test* level. For global logging, you can use the ``--log <FILE>`` command
+line option and all output is mirrored to the specified logfile. Logging single *tests* or *testers* is possible by using the
+``logfile`` attribute:
 
-* ``--logfile`` - Mirrors all tricot output into a logfile
-* ``--debug`` - Show details on each tester that runs, even when successful. Furthermore, disable
-  exception handling and show each exception with full details.
+```yaml
+tester:
+  name: ExampleTester
+  title: Just an example test
+  logfile: /log/example-tester.log
+
+tests:
+  - title: Test curl
+    description: |-
+      Test that our curl installation is working
+
+    command:
+      - curl
+      - http://example.org
+    logfile: /log/curl-tester.log
+
+    validators:
+      - status: 0
+
+  - title: Test wget
+    description: |-
+      Test that our wget installation is working
+
+    command:
+      - wget
+      - http://example.org
+    logfile: /log/wget-tester.log
+
+    validators:
+      - status: 0
+```
+
+Log files are always written in verbose mode and contain the full details for each *test* or *tester*.
+This is also true, even if the corresponding *test* or *tester* run was successful.
+
+
+### Custom Strings
+
+----
+
+*Testers*, *tests* and *validators* accept an additional *yml attribute* ``output``. This attribute
+is expected to be structured like this:
+
+```yml
+output:
+  success_string: worked :)
+  success_color: cyan
+  failure_string: nope :(
+  failure_color: magenta
+```
+
+Output specifications in *tests* overwrite settings in *testers* and output specifications in *validators*
+overwrite settings in *tests*. *Validators* are only allowed to set the ``failure_string`` and ``failure_color``
+settings. Furthermore, support for *validators* is currently limited by their parameter type. The ``output`` attribute
+can only be specified for *validators* that have an dictionary parameter type.
+
+
+### Worth Knowing
+
+----
+
+The following list contains information on some smaller *tricot* features that did not receive their own section:
+
+* You can always use the ``--debug`` command line option to show details on each tester that runs, even when successful.
+  Furthermore, the switch disables exception handling and shows each exception with full details.
+* Each *test* can contain the special attribute ``shell`` with a boolean value. If ``True`` commands are executed in shell
+  mode.

docs/plugins/README.md

@@ -56,6 +56,7 @@ to be used for more complicated cleanup actions, that cannot be performed by the
 ```python
 param_type = dict
 inner_types = {
+                'shell': {'required': False, 'type': bool},
                 'ignore_error': {'required': False, 'type': bool},
                 'timeout': {'required': False, 'type': int},
                 'cmd': {'required': True, 'type': list}
@@ -191,13 +192,15 @@ The plugin allows some additional configuration by using different plugin keys:
   This gives the command some initialization time.
 * ``timeout``: Timeout the command after the specified amount of seconds. A timed
   out plugin will cause an error. Use ``init`` if you need a timeout without error.
+* ``shell``: Execute the command in shell mode.
 
 
 **Type Validation**:
 
 ```python
 param_type = dict
 inner_types = {
+                'shell': {'required': False, 'type': bool},
                 'ignore_error': {'required': False, 'type': bool},
                 'init': {'required': False, 'type': int},
                 'background': {'required': False, 'type': bool},

setup.py

@@ -11,7 +11,7 @@
     url='https://github.com/qtc-de/tricot',
     name='tricot',
     author='Tobias Neitzel (@qtc_de)',
-    version='1.3.2',
+    version='1.4.0',
     author_email='',
 
     description='Trivial Command Testser',

tests/pytest/Plugins/CleanupCommandPlugin/cleanup_command_test.py

@@ -2,7 +2,6 @@
 
 import tricot
 import pytest
-import timeit
 
 from pathlib import Path
 from functools import partial
@@ -144,6 +143,26 @@ def test_cleanup_command_timeout():
         plug._stop()
 
 
+def test_cleanup_command_shell():
+    '''
+    Test if plain command execution is working by creating a directory in shell mode.
+
+    Parameters:
+        None
+
+    Returns:
+        None
+    '''
+    config = {'cmd': ['echo', 'hi', '&&', 'mkdir', test_dir], 'shell': True}
+
+    plug = tricot.get_plugin(Path(__file__), 'cleanup_command', config, {})
+    plug.stop()
+
+    r = resolve(test_dir)
+    assert r.is_dir()
+    r.rmdir()
+
+
 @pytest.fixture(autouse=True)
 def resource():
     '''

tests/pytest/Plugins/OsCommandPlugin/command_test.py

@@ -176,6 +176,26 @@ def test_os_command_init():
     assert timer.timeit(number=1) > 2
 
 
+def test_os_command_shell():
+    '''
+    Test if plain command execution is working by creating a directory in shell mode.
+
+    Parameters:
+        None
+
+    Returns:
+        None
+    '''
+    config = {'cmd': ['echo', 'hi', '&&', 'mkdir', test_dir], 'shell': True}
+
+    plug = tricot.get_plugin(Path(__file__), 'os_command', config, {})
+    plug._run()
+
+    r = resolve(test_dir)
+    assert r.is_dir()
+    r.rmdir()
+
+
 @pytest.fixture(autouse=True)
 def resource():
     '''

tests/pytest/Validators/LineCountValidator/line_count_test.py

@@ -27,6 +27,7 @@
 
 results = [True, True, True, True, False, True, True, True, True, True, True]
 
+
 @pytest.mark.parametrize('output, config, result', zip(outputs, configs, results))
 def test_line_count_validator(output: int, config: dict, result: bool):
     '''

tests/tricot/test-cases/extra/Log.yml

@@ -0,0 +1,52 @@
+tester:
+  name: log
+  title: Log Test
+  description: |-
+    "Performs different checks on tricot's logfile feature."
+  logfile: ${tester-log}
+
+
+variables:
+  tester-log: /tmp/tricot-tester-logile.log
+  test-log: /tmp/tricot-test-logile.log
+  probe: 'This string should be contained within the logfile :)'
+
+
+tests:
+  - title: Log Test - Success
+    description: >
+      'Echo a string that should be contained within the logfiles'
+    command:
+      - echo
+      - ${probe}
+    logfile: ${test-log}
+    validators:
+      - contains:
+          values:
+            - ${probe}
+
+  - title: Log Verify - Success
+    description: >
+      'Checks whether the logfiles are present and contain the required content'
+    command:
+      - echo
+      - dummy
+    validators:
+      - file_contains:
+        - file: ${tester-log}
+          contains:
+            - ${probe}
+            - 'Log Test - Success... success'
+            - 'Command exit code: 0'
+            - 'Validator name: contains'
+        - file: ${test-log}
+          contains:
+            - ${probe}
+            - 'Log Test - Success... success'
+            - 'Command exit code: 0'
+            - 'Validator name: contains'
+      - file_exists:
+          cleanup: True
+          files:
+            - ${tester-log}
+            - ${test-log}

tests/tricot/test-cases/extra/Shell.yml

@@ -0,0 +1,62 @@
+tester:
+  name: shell
+  title: shell Tester
+  description: |-
+    'Check whether tricot handels shell mode correctly'
+
+tests:
+  - title: Echo Stderr - Success
+    description: >
+      'Echo to stderr'
+    command:
+      - echo
+      - 'Should appear in stderr'
+      - '1>&2'
+    shell: True
+    validators:
+      - contains:
+          stream: stderr
+          values:
+            - 'Should appear in stderr'
+      - line_count:
+          stream: stderr
+          count: 1
+
+  - title: Output Redirection - Success
+    description: >
+      'Redirect echo to file'
+    command:
+      - echo
+      - 'This is a test'
+      - '>'
+      - ./output_redirection_test.txt
+    shell: True
+    validators:
+      - file_contains:
+        - file: ./output_redirection_test.txt
+          contains:
+            - 'This is a test'
+      - file_exists:
+          cleanup: True
+          files:
+              - ./output_redirection_test.txt
+
+  - title: Pipe - Success
+    description: >
+      'Pipe echo to tee'
+    command:
+      - echo
+      - 'This is another test'
+      - '|'
+      - tee
+      - ./output_redirection_test.txt
+    shell: True
+    validators:
+      - file_contains:
+        - file: ./output_redirection_test.txt
+          contains:
+            - 'This is another test'
+      - file_exists:
+          cleanup: True
+          files:
+              - ./output_redirection_test.txt

tests/tricot/test-cases/plugin-tests/CleanupCommandPlugin.yml

@@ -15,3 +15,5 @@ variables:
 testers:
   - ./CleanupCommandPlugin/CleanupCommandPluginCleanup.yml
   - ./CleanupCommandPlugin/CleanupCommandPluginVerify.yml
+  - ./CleanupCommandPlugin/CleanupCommandPluginCleanupShell.yml
+  - ./CleanupCommandPlugin/CleanupCommandPluginVerifyShell.yml