Tweak dependency versions for newer Pythons (#199)

* Tweak dependency versions for newer Pythons

To ensure compatibility with newer Pythons unpin some requirements
when building in environments with later Pythons.

This resolves compability issues with new Pythons and is OK as
charms can now be build on the Ubuntu release that they will also
run on.

* Fixup branch for actions operator

* Further updates to get layer-basic to work with py3.10

The 'split' is at py3.8 to enable greater testing.  This commit adds
further pins for pre py3.8 and relaxations for >= py3.8 to allow new
versions of python to build charms.

* Add comments to README on build/run bases

This patch updates the README to include information that the
layer-basic is now aware of what it is being built on and the modules in
the wheelhouse.txt (for layer-basic) are dependent on whether they are
pre- or post-inclusive of Python 3.8

* Fix linting issue by removing commented-out code

* Use latest version of charm-tools for testing

* Correct egg name for charm-tools

Co-authored-by: Alex Kavanagh <alex.kavanagh@canonical.com>

.github/workflows/main.yml

@@ -35,7 +35,7 @@ jobs:
         run: |
           pip install tox
       - name: Setup operator environment
-        uses: charmed-kubernetes/actions-operator@master
+        uses: charmed-kubernetes/actions-operator@main
       - name: Run test
         run: tox -e func
       - name: Show Status

README.md

@@ -12,5 +12,19 @@ Go read the [layer-basic documentation][] for more info on how to use this
 layer. It is now hosted together with the charms.reactive documentation in order
 to reduce the amount of places a charmer needs to search for info.
 
+# Python Versions, Built charms and what they can run on.
+
+Due to major backwards incompatibilities between Python 3.10 and previous
+versions, there is a compatibility break between Python 3.8 (focal) and earlier
+versions of Python.  Why Python 3.8 rather than 3.10?  Mostly due to all of the
+incompatibilities being deprecated and available in Python 3.8, so the ability
+of authors to test at that version.
+
+As the charmhub.io now offers a place to build reactive charms on a per series
+('base') and architecture basis, layer-basic supports *at least* Python 3.5 to
+Python 3.10.  However, a charm *built* on Python 3.5 won't work on Python 3.8
+and vice-versa.  The objective going forwards is to build the charm for the
+*base* that the charm will run on.
+
 [charm-helpers]: https://pythonhosted.org/charmhelpers/
 [layer-basic documentation]: https://charmsreactive.readthedocs.io/en/latest/layer-basic.html

lib/charms/layer/basic.py

@@ -201,17 +201,24 @@ def bootstrap_charm_deps():
         # conflicts)
         _versions = _load_wheelhouse_versions()
         _pkgs = _versions.keys() - set(pre_install_pkgs)
-        # add back the versions such that each package in pkgs is
-        # <package_name>==<version>.
-        # This ensures that pip 20.3.4+ will install the packages from the
-        # wheelhouse without (erroneously) flagging an error.
-        pkgs = _add_back_versions(_pkgs, _versions)
-        reinstall_flag = '--force-reinstall'
-        if not cfg.get('use_venv', True) and pre_eoan:
-            reinstall_flag = '--ignore-installed'
-        check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
-                    '--no-cache-dir', '-f', 'wheelhouse'] + list(pkgs),
-                   env=_get_subprocess_env())
+        # Jinja2 3+ relies on MarkupSafe actually being installed prior to
+        # attempting to be installed from the wheelhouse.  Thus, if MarkupSafe
+        # and/or wheel are in _pkgs, then install them first.
+        _pre_packages = [p for p in _pkgs if p in ('wheel', 'MarkupSafe')]
+        _pkgs = [p for p in _pkgs if p not in _pre_packages]
+        for _pkgs_set in (_pre_packages, _pkgs):
+            # add back the versions such that each package in pkgs is
+            # <package_name>==<version>.
+            # This ensures that pip 20.3.4+ will install the packages from the
+            # wheelhouse without (erroneously) flagging an error.
+            pkgs = _add_back_versions(_pkgs_set, _versions)
+            reinstall_flag = '--force-reinstall'
+            # if not cfg.get('use_venv', True) and pre_eoan:
+            if not cfg.get('use_venv', True):
+                reinstall_flag = '--ignore-installed'
+            check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
+                        '--no-cache-dir', '-f', 'wheelhouse'] + list(pkgs),
+                       env=_get_subprocess_env())
         # re-enable installation from pypi
         os.remove('/root/.pydistutils.cfg')
 

test-requirements.txt

@@ -1,2 +1,3 @@
-charm-tools
+# Use latest so that layer-basic tests against main branch
+git+https://github.com/juju/charm-tools.git#egg=charm-tools
 git+https://github.com/openstack-charmers/zaza.git#egg=zaza

tests/bundles/minimal.yaml

@@ -1,32 +1,34 @@
+# Disable all bundles that aren't 'latest', currently focal
+# See readme for details
 applications:
-  minimal-trusty:
-    series: trusty
-    charm: /tmp/charm-builds/minimal
-    num_units: 1
-  minimal-xenial:
-    series: xenial
-    charm: /tmp/charm-builds/minimal
-    num_units: 1
-  minimal-bionic:
-    series: bionic
-    charm: /tmp/charm-builds/minimal
-    num_units: 1
+  #minimal-trusty:
+    #series: trusty
+    #charm: /tmp/charm-builds/minimal
+    #num_units: 1
+  #minimal-xenial:
+    #series: xenial
+    #charm: /tmp/charm-builds/minimal
+    #num_units: 1
+  #minimal-bionic:
+    #series: bionic
+    #charm: /tmp/charm-builds/minimal
+    #num_units: 1
   minimal-focal:
     series: focal
     charm: /tmp/charm-builds/minimal
     num_units: 1
-  minimal-no-venv-trusty:
-    series: trusty
-    charm: /tmp/charm-builds/minimal-no-venv
-    num_units: 1
-  minimal-no-venv-xenial:
-    series: xenial
-    charm: /tmp/charm-builds/minimal-no-venv
-    num_units: 1
-  minimal-no-venv-bionic:
-    series: bionic
-    charm: /tmp/charm-builds/minimal-no-venv
-    num_units: 1
+  #minimal-no-venv-trusty:
+    #series: trusty
+    #charm: /tmp/charm-builds/minimal-no-venv
+    #num_units: 1
+  #minimal-no-venv-xenial:
+    #series: xenial
+    #charm: /tmp/charm-builds/minimal-no-venv
+    #num_units: 1
+  #minimal-no-venv-bionic:
+    #series: bionic
+    #charm: /tmp/charm-builds/minimal-no-venv
+    #num_units: 1
   minimal-no-venv-focal:
     series: focal
     charm: /tmp/charm-builds/minimal-no-venv

tests/charm-minimal/wheelhouse.txt

@@ -1,2 +1,3 @@
 # We add a wheel that needs to be built from source to validate that this works
-psycopg2
+psycopg2;python_version < '3.8'
+psycopg;python_version >= '3.8'

tox.ini

@@ -24,8 +24,8 @@ commands =
     /bin/mkdir -p /tmp/charm-builds/_tmp/layers
     /bin/bash -c '/bin/ln -sf $(readlink --canonicalize {toxinidir}) /tmp/charm-builds/_tmp/layers/layer-basic'
     /bin/bash -c '/bin/ln -sf $(readlink --canonicalize {toxinidir}/tests/charm-minimal) /tmp/charm-builds/_tmp/layers/charm-minimal'
-    charm-build tests/charm-minimal
-    charm-build tests/charm-minimal-no-venv
+    charm-build --log-level DEBUG tests/charm-minimal
+    charm-build --log-level DEBUG tests/charm-minimal-no-venv
     functest-run-suite --keep-model
 
 

wheelhouse.txt

@@ -1,16 +1,29 @@
 # pip is pinned to <19.0 to avoid https://github.com/pypa/pip/issues/6164
 # even with installing setuptools before upgrading pip ends up with pip seeing
 # the older setuptools at the system level if include_system_packages is true
-pip>=18.1,<19.0
+pip>=18.1,<19.0;python_version < '3.8'
+pip;python_version >= '3.8'
 # pin Jinja2, PyYAML and MarkupSafe to the last versions supporting python 3.5
 # for trusty
-Jinja2<=2.10.1
-PyYAML<=5.2
-MarkupSafe<2.0.0
-setuptools<42
-setuptools-scm<=1.17.0
-charmhelpers>=0.4.0,<1.0.0
+Jinja2==2.10;python_version >= '3.0' and python_version <= '3.4' # py3 trusty
+Jinja2==2.11;python_version == '2.7' or python_version == '3.5'  # py27, py35
+Jinja2;python_version >= '3.6' # py36 and on
+
+PyYAML==5.2;python_version >= '3.0' and python_version <= '3.4' # py3 trusty
+PyYAML<5.4;python_version == '2.7' or python_version >= '3.5'  # all else
+
+MarkupSafe<2.0.0;python_version < '3.6'
+MarkupSafe<2.1.0;python_version == '3.6' # Just for python 3.6
+MarkupSafe;python_version >= '3.7' # newer pythons
+
+setuptools<42;python_version < '3.8'
+setuptools;python_version >= '3.8'
+setuptools-scm<=1.17.0;python_version < '3.8'
+setuptools-scm;python_version >= '3.8'
+flit_core;python_version >= '3.8'
+charmhelpers>=0.4.0,<2.0.0
 charms.reactive>=0.1.0,<2.0.0
-wheel<0.34
+wheel<0.34;python_version < '3.8'
+wheel;python_version >= '3.8'
 # pin netaddr to avoid pulling importlib-resources
 netaddr<=0.7.19