Use implicit namespace plugins for import and export (#1216)

* behavior outline
* FIrst pass at allow external plugins
* remove template exporter
* Add listing of active plugins to '--version' output
* Documentation for plugins
* [Docs] add custom imports and exporters to site TOC
* [Docs] better linewrapping
* enforce positive initial linewrap
  Check column widths
  update gitignore
  throw error when linewrap too small
  simply check for large enough linewrap value
* delete unused error message
* PR feedback
  make exception more informative
  update check_linewrap signature in src and test
  make check_linewrap a free function
* delete unused function
* delete else..pass block
* newline for make format
* Include dates_exporter
* Use Base classes for importer and exporters.
* [Docs] improve documentation of custom Importers and Exporters
* [Testing] separate run with external plugin!
* basic behavior test
* prototype unittest for JSON Exporter
  test for unimplemented method
* make format
  delete unused imports
* Remove 'importer' or 'exporter' from filenames where not needed
* [Test] run different tests with or without the external plugins installed
* [Test] move test rot13 plugin into git tree
  from MinchinWeb/jrnl-rot13-exporter@0dc912a
* consolidate demo plugins to common package
* [Docs] name page for plugins
* [Docs] include the sample plug in code files directly
* style fixes
* [test] determine whether to run external plug in tests based on installed packages
* improved code documentation
* style fixes for GitHub actions
* Convert "short" and "pretty" (and "default") formaters to plugins
  further to #1177
* more code clean up
  tests pass locally...now for GitHub...
* [tests] dynamically determine jrnl version for plugin tests
* [GitHub Actions] direct install of testing plugins
* Remove template code
* [plugins] meta --> collector
* [Docs] create scripted entries using an custom importer
* (closer to) being able to run behave tests outside project root directory
* We already know when exporter to use
  Don't re-calculate it!
* [Tests] don't name test plugin 'testing"
  If so named, pip won't install it.
* [Test] run behave tests with test plugins outside project root
* [Test] behave tests pass locally
* [Docs] fix typo
* [GitHub Actions] run test commands from poetry's shell
* black-ify code
* [GitHub Actions] move downstream (rather than up) to run tests
* [GitHub Actions] set shell to poetry
* [GitHub Workflows] Manually activate virtual environment
* [GitHub Actions] Skip Windows & Python 3.8
  Can't seem to find Python exe?
* [GiotHub Actions] explicitly use virtual env
* [GitHub Actions] create virutal env directly
* [GitHub Actions] better activate of Windows virtual env
* [GitHub Actions] create virtual env on Mac
* [Github Actions] install wheel and upgrade pip
* [GitHub Actions] skip virtual environments altogether
* [GitHub Actions] change directory for behave test
* Remove Windows exclusions from CI as per note -- they should be working now

Co-authored-by: Suhas <sugas182@gmail.com>
Co-authored-by: Micah Jerome Ellison <micah.jerome.ellison@gmail.com>

.github/workflows/testing_external_plugins.yaml

@@ -0,0 +1,57 @@
+name: Testing
+
+on:
+  push:
+    branches: [ develop, release ]
+    paths:
+    - 'jrnl/**'
+    - 'features/**'
+    - 'tests/**'
+    - 'poetry.lock'
+    - 'pyproject.toml'
+  pull_request:
+    branches: [ develop ]
+    paths:
+    - 'jrnl/**'
+    - 'features/**'
+    - 'tests/**'
+    - 'poetry.lock'
+    - 'pyproject.toml'
+
+jobs:
+  test-namespace-plugins:
+    if: >
+      ! contains(github.event.head_commit.message, '[ci skip]')
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        python-version: [ 3.7, 3.8, 3.9 ]
+        os: [ ubuntu-latest, macos-latest, windows-latest ]
+        exclude: # Added for GitHub Actions PR problem 2020-12-19 -- remove later!
+          - os: windows-latest
+            python-version: 3.9
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install pip setuptools wheel --upgrade
+        python -m pip install .
+        python -m pip install ./tests/external_plugins_src/
+        python -m pip install pytest behave
+    # installed test plugins aren't recognized by "behave" if run from the
+    # project's root folder
+
+    - name: Test with pytest
+      if: success() || failure()
+      run: pytest --junitxml=reports/pytest/results.xml
+
+    - name: Test with behave
+      if: success() || failure()
+      run: cd features && behave --no-skipped --format progress2 --junit --junit-directory ../reports/behave

docs/formats.md

@@ -13,7 +13,10 @@ used alone (e.g. `jrnl --format json`) to display all entries from the selected
 
 This page shows examples of all the built-in formats, but since `jrnl` supports adding
 more formats through plugins, you may have more available on your system. Please see
-`jrnl --help` for a list of which formats are available on your system.
+`jrnl --version` for a list of which formats are available on your system. Note
+that plugins can also override built-in formats, so review your installed
+plugins if your output does not match what is listed here. You can also [write
+your own plugins](./plugins.md) to create custom formats.
 
 Any of these formats can be used interchangeably, and are only grouped into "display",
 "data", and "report" formats below for convenience.

docs/plugins.md

@@ -0,0 +1,181 @@
+<!-- Copyright (C) 2012-2021 jrnl contributors
+     License: https://www.gnu.org/licenses/gpl-3.0.html -->
+
+# Extending jrnl
+
+*jrnl* can be extended with custom importers and exporters.
+
+Note that custom importers and exporters can be given the same name as a
+built-in importer or exporter to override it.
+
+Custom Importers and Exporters are traditional Python packages, and are
+installed (into *jrnl*) simply by installing them so they are available to the
+Python interpreter that is running *jrnl*.
+
+Exporter are also used as "formatters" when entries are written to the command
+line.
+
+## Rational
+
+I added this feature because *jrnl* was overall working well for me, but I
+found myself maintaining a private fork so I could have a slightly customized
+export format. Implementing (import and) export plugins was seen as a way to
+maintain my custom exporter without the need to maintaining my private fork.
+
+This implementation tries to keep plugins as light as possible, and as free of
+boilerplate code as reasonable. As well, internal importers and exporters are
+implemented in almost exactly the same way as custom importers and exporters,
+and so it is hoped that plugins can be moved from "contributed" to "internal"
+easily, or that internal plugins can serve as a base and/or a demonstration for
+external plugins.
+
+-- @MinchinWeb, May 2021
+
+## Entry Class
+
+Both the Importers and the Exporters work on the `Entry` class. Below is a
+(selective) description of the class, it's properties and functions:
+
+- **Entry** (class) at `jrnl.Entry.Entry`.
+    - **title** (string): a single line that represents a entry's title.
+    - **date** (datetime.datetime): the date and time assigned to an entry.
+    - **body** (string): the main body of the entry. Can be basically any
+      length. *jrnl* assumes no particular structure here.
+    - **starred** (boolean): is an entry starred? Presumably, starred entries
+      are of particular importance.
+    - **tags** (list of strings): the tags attached to an entry. Each tag
+      includes the pre-facing "tag symbol".
+    - **\_\_init\_\_(journal, date=None, text="", starred=False)**: contractor
+      method
+        - **journal** (*jrnl.Journal.Journal*): a link to an existing Journal
+          class. Mainly used to access it's configuration.
+        - **date** (datetime.datetime)
+        - **text** (string): assumed to include both the title and the body.
+          When the title, body, or tags of an entry are requested, this text
+          will the parsed to determine the tree.
+        - **starred** (boolean)
+
+Entries also have "advanced" metadata if they are using the DayOne backend, but
+we'll ignore that for the purposes of this demo.
+
+## Custom Importer
+
+If you have a (custom) datasource that you want to import into your jrnl
+(perhaps like a blog export), you can write a custom importer to do this.
+
+An importer takes the source data, turns it into Entries and then appends those
+entries to a Journal. Here is a basic Importer, assumed to be provided with a
+nicely formatted JSON file:
+
+~~~ python
+{%
+  include-markdown "../tests/external_plugins_src/jrnl/contrib/importer/simple_json.py"
+  comments=false
+%}
+~~~
+
+Note that the above is very minimal, doesn't do any error checking, and doesn't
+try to import all possible entry metadata.
+
+Another potential use of a custom importer is to effectively create a scripted
+entry creator. For example, maybe each day you want to create a journal entry
+that contains the answers to specific questions; you could create a custom
+"importer" that would ask you the questions, and then create an entry containing
+the answers provided.
+
+Some implementation notes:
+
+- The importer class must be named **Importer**, and should sub-class
+  **jrnl.plugins.base.BaseImporter**.
+- The importer module must be within the **jrnl.contrib.importer** namespace.
+- The importer must not have any `__init__.py` files in the base directories
+  (but you can have one for your importer base directory if it is in a
+  directory rather than a single file).
+- The importer must be installed as a Python package available to the same
+  Python interpreter running jrnl.
+- The importer must expose at least the following the following members:
+    - **version** (string): the version of the plugin. Displayed to help the
+      user debug their installations.
+    - **names** (list of strings): these are the "names" that can be passed to
+      the CLI to involve your importer. If you specify one used by a built-in
+      plugin, it will overwrite it (effectively making the built-in one
+      unavailable).
+    - **import_(journal, input=None)**: the actual importer. Must append
+      entries to the journal passed to it. It is recommended to accept either a
+      filename or standard input as a source.
+
+## Custom Exporter
+
+Custom exporters are useful to make *jrnl*'s data available to other programs.
+One common usecase would to generate the input to be used by a static site
+generator or blogging engine.
+
+An exporter take either a whole journal or a specific entry and exports it.
+Below is a basic JSON Exporter; note that a more extensive JSON exporter is
+included in *jrnl* and so this (if installed) would override the built in
+exporter.
+
+~~~ python
+{%
+  include-markdown "../tests/external_plugins_src/jrnl/contrib/exporter/custom_json.py"
+  comments=false
+%}
+~~~
+
+Note that the above is very minimal, doesn't do any error checking, and doesn't
+export all entry metadata.
+
+Some implementation notes:
+
+- the exporter class must be named **Exporter** and should sub-class
+  **jrnl.plugins.base.BaseExporter**.
+- the exporter module must be within the **jrnl.contrib.exporter** namespace.
+- The exporter must not have any `__init__.py` files in the base directories
+  (but you can have one for your exporter base directory if it is in a
+  directory rather than a single file).
+- The exporter must be installed as a Python package available to the same
+  Python interpreter running jrnl.
+- the exporter should expose at least the following the following members
+  (there are a few more you will need to define if you don't subclass
+  `jrnl.plugins.base.BaseExporter`):
+    - **version** (string): the version of the plugin. Displayed to help the
+      user debug their installations.
+    - **names** (list of strings): these are the "names" that can be passed to
+      the CLI to invole your exporter. If you specific one used by a built-in
+      plugin, it will overwrite it (effectively making the built-in one
+      unavailable).
+    - **extension** (string): the file extention used on exported entries.
+    - **export_entry(entry)**: given an entry, returns a string of the formatted,
+      exported entry.
+    - **export_journal(journal)**: (optional) given a journal, returns a string
+      of the formatted, exported entries of the journal. If not implemented,
+      *jrnl* will call **export_entry()** on each entry in turn and then
+      concatenate the results together.
+
+### Special Exporters
+
+There are a few "special" exporters, in that they are called by *jrnl* in
+situations other than a traditional export. They are:
+
+- **short** -- called by `jrnl --short`. Displays each entry on a single line.
+  The default is to print the timestamp of the entry, followed by the title.
+  The built-in (default) plugin is at `jrnl.plugins.exporter.short`.
+- **default** -- called when a different format is not specified. The built-in
+  (default) plugin is at `jrnl.plugins.exporter.pretty`.
+
+## Development Tips
+
+- Editable installs (`pip install -e ...`) don't seem to play nice with
+  the namespace layout. If your plugin isn't appearing, try a non-editable
+  install of both *jrnl* and your plugin.
+- If you run *jrnl* from the main project root directory (the one that contains
+  *jrnl*'s source code), namespace plugins won't be recognized. This is (I
+  suspect) because the Python interpreter will find your *jrnl* source directory
+  (which doesn't contain your namespace plugins) before it find your
+  "site-packages" directory (i.e. installed packages, which will recognize
+  namespace packages).
+- Don't name your plugin file "testing.py" or it won't be installed (at least
+  automatically) by pip.
+- For examples, you can look to the *jrnl*'s internal importers and exporters.
+  As well, there are some basic external examples included in *jrnl*'s git repo
+  at `tests/external_plugins_src` (including the example code above).

docs_theme/requirements.txt

@@ -1 +1,2 @@
-mkdocs==1.1
+mkdocs==1.1.2
+mkdocs-include-markdown-plugin==2.8.0

features/environment.py

@@ -1,9 +1,17 @@
 import os
+from pathlib import Path
 import shutil
 
 from jrnl.os_compat import on_windows
 
+try:
+    from jrnl.contrib.exporter import flag as testing_exporter
+except ImportError:
+    testing_exporter = None
+
 CWD = os.getcwd()
+HERE = Path(__file__).resolve().parent
+TARGET_CWD = HERE.parent  # project root folder
 
 # @see https://behave.readthedocs.io/en/latest/tutorial.html#debug-on-error-in-case-of-step-failures
 BEHAVE_DEBUG_ON_ERROR = False
@@ -15,6 +23,8 @@ def setup_debug_on_error(userdata):
 
 
 def before_all(context):
+    # always start in project root directory
+    os.chdir(TARGET_CWD)
     setup_debug_on_error(context.config.userdata)
 
 
@@ -27,10 +37,10 @@ def before_all(context):
 
 
 def clean_all_working_dirs():
-    if os.path.exists("test.txt"):
-        os.remove("test.txt")
+    if os.path.exists(HERE / "test.txt"):
+        os.remove(HERE / "test.txt")
     for folder in ("configs", "journals", "cache"):
-        working_dir = os.path.join("features", folder)
+        working_dir = HERE / folder
         if os.path.exists(working_dir):
             shutil.rmtree(working_dir)
 
@@ -46,20 +56,28 @@ def before_feature(context, feature):
         feature.skip("Skipping on Windows")
         return
 
+    if "skip_only_with_external_plugins" in feature.tags and testing_exporter is None:
+        feature.skip("Requires test external plugins installed")
+        return
+
+    if "skip_no_external_plugins" in feature.tags and testing_exporter:
+        feature.skip("Skipping with external plugins installed")
+        return
+
 
 def before_scenario(context, scenario):
     """Before each scenario, backup all config and journal test data."""
     # Clean up in case something went wrong
     clean_all_working_dirs()
     for folder in ("configs", "journals"):
-        original = os.path.join("features", "data", folder)
-        working_dir = os.path.join("features", folder)
+        original = HERE / "data" / folder
+        working_dir = HERE / folder
         if not os.path.exists(working_dir):
             os.mkdir(working_dir)
         for filename in os.listdir(original):
-            source = os.path.join(original, filename)
+            source = original / filename
             if os.path.isdir(source):
-                shutil.copytree(source, os.path.join(working_dir, filename))
+                shutil.copytree(source, (working_dir / filename))
             else:
                 shutil.copy2(source, working_dir)
 
@@ -73,11 +91,22 @@ def before_scenario(context, scenario):
         scenario.skip("Skipping on Windows")
         return
 
+    if (
+        "skip_only_with_external_plugins" in scenario.effective_tags
+        and testing_exporter is None
+    ):
+        scenario.skip("Requires test external plugins installed")
+        return
+
+    if "skip_no_external_plugins" in scenario.effective_tags and testing_exporter:
+        scenario.skip("Skipping with external plugins installed")
+        return
+
 
 def after_scenario(context, scenario):
     """After each scenario, restore all test data and remove working_dirs."""
-    if os.getcwd() != CWD:
-        os.chdir(CWD)
+    if os.getcwd() != TARGET_CWD:
+        os.chdir(TARGET_CWD)
 
     # only clean up if debugging is off and the scenario passed
     if BEHAVE_DEBUG_ON_ERROR and scenario.status != "failed":

features/format.feature

@@ -26,6 +26,7 @@ Feature: Custom formats
         |   basic_folder    |
         |   basic_dayone    |
 
+    @skip_no_external_plugins
     Scenario Outline: JSON format
         Given we use the config "<config>.yaml"
         And we use the password "test" if prompted
@@ -48,6 +49,7 @@ Feature: Custom formats
         | basic_folder    |
         | basic_dayone    |
 
+    @skip_no_external_plugins
     Scenario: Exporting dayone to json
         Given we use the config "dayone.yaml"
         When we run "jrnl --export json"
@@ -91,6 +93,7 @@ Feature: Custom formats
         | basic_folder    |
         | basic_dayone    |
 
+    @skip_no_external_plugins
     Scenario Outline: Exporting using filters should only export parts of the journal
         Given we use the config "<config>.yaml"
         And we use the password "test" if prompted
@@ -112,6 +115,7 @@ Feature: Custom formats
         | basic_folder    |
         | basic_dayone    |
 
+    @skip  # template exporters have been removed
     Scenario Outline: Exporting using custom templates
         Given we use the config "<config>.yaml"
         And we load template "sample.template"