Update Django to next LTS #2734

phillxnet · 2023-11-08T16:34:45Z

This represents a blocker for our Py3.10 Milestone (https://github.com/rockstor/rockstor-core/milestone/25) and we have issue (#2696) which may limit us to Py3.10 for now. Plus our current Py3.9 is itself a blocker for our Tumbleweed-aarch64 target (see: #2693) preventing our next Stable release.

Canonical issue for Milestone: https://github.com/rockstor/rockstor-core/milestone/24

From: https://docs.djangoproject.com/en/4.2/faq/install/#what-python-version-can-i-use-with-django

we have:

Django version	Python versions
3.2	3.6, 3.7, 3.8, 3.9, 3.10 (added in 3.2.9)

Where-as our current Django is at its end re Python compatibility, representing a Python update blocker as-is:
From: https://docs.djangoproject.com/en/3.2/faq/install/#what-python-version-can-i-use-with-django

Django version	Python versions
2.2	3.5, 3.6, 3.7, 3.8 (added in 2.2.8), 3.9 (added in 2.2.17)

phillxnet · 2023-11-10T13:31:36Z

Initial pyproject.toml change against current testing branch poetry.lock (after poetry install) resulted in:

lbuildvm:/opt/rockstor # poetry update
Updating dependencies
Resolving dependencies... (0.7s)

Writing lock file

Package operations: 0 installs, 3 updates, 0 removals

  • Updating wrapt (1.15.0 -> 1.16.0)
  • Updating charset-normalizer (3.3.1 -> 3.3.2)
  • Updating django (2.2.28 -> 3.2.23)

And given:

lbuildvm:/opt/rockstor # poetry show --tree
...
dbus-python 1.2.18 Python bindings for libdbus
distro 1.8.0 Distro - an OS platform information API
django 3.2.23 A high-level Python Web framework that encourages rapid development and clean, pragmatic design.
├── asgiref >=3.3.2,<4
│   └── typing-extensions >=4 
├── pytz *
└── sqlparse >=0.2.2
...

We have two additional, incidental, updates along the way.

Wrapt

Changelog: https://wrapt.readthedocs.io/en/latest/changes.html
And looks to be a second level dependency of Django-oath-toolkit:

django-oauth-toolkit 2.3.0 OAuth2 Provider for Django
├── django >=2.2,<4.0.0 || >4.0.0
│   ├── asgiref >=3.3.2,<4 
│   │   └── typing-extensions >=4 
│   ├── pytz * 
│   └── sqlparse >=0.2.2 
├── jwcrypto >=0.8.0
│   ├── cryptography >=3.4 
│   │   └── cffi >=1.12 
│   │       └── pycparser * 
│   └── deprecated * 
│       └── wrapt >=1.10,<2

Charset-normalizer

Changelog: https://github.com/Ousret/charset_normalizer/blob/master/CHANGELOG.md
A dependency of requests:

└── requests >=2.13.0
    ├── certifi >=2017.4.17 
    ├── charset-normalizer >=2,<4 
    ├── idna >=2.5,<4 
    └── urllib3 >=1.21.1,<3

phillxnet · 2023-11-10T14:03:37Z

Post Django update we have 78 new WARNINGS:
All akin to:

smart_manager.CPUMetric: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
        HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
...
storageadmin.User: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
        HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.

System check identified 78 issues (0 silenced).

And 5 new test failures:

With the following repeated many times:

Traceback (most recent call last):
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/defaulttags.py", line 1037, in find_library
    return parser.libraries[name]
KeyError: 'staticfiles'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/storageadmin/tests/test_shares.py", line 141, in test_get
    self.get_base(self.BASE_URL)
  File "/opt/rockstor/src/rockstor/storageadmin/tests/test_api.py", line 60, in get_base
    response1 = self.client.get('%s/invalid' % baseurl)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/rest_framework/test.py", line 289, in get
    response = super().get(path, data=data, **extra)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/rest_framework/test.py", line 206, in get
    return self.generic('GET', path, **r)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/rest_framework/test.py", line 234, in generic
    return super().generic(
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/test/client.py", line 473, in generic
    return self.request(**r)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/rest_framework/test.py", line 286, in request
    return super().request(**kwargs)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/rest_framework/test.py", line 238, in request
    request = super().request(**kwargs)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/test/client.py", line 714, in request
    response = self.handler(environ)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/test/client.py", line 145, in __call__
    response = self.get_response(request)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/rest_framework/test.py", line 258, in get_response
    return super().get_response(request)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 130, in get_response
    response = self._middleware_chain(request)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 49, in inner
    response = response_for_exception(request, exc)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 114, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 153, in handle_uncaught_exception
    return callback(request)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/views/defaults.py", line 97, in server_error
    return HttpResponseServerError(template.render())
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 170, in render
    return self._render(context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/test/utils.py", line 100, in instrumented_test_render
    return self.nodelist.render(context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 938, in render
    bit = node.render_annotated(context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 905, in render_annotated
    return self.render(context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/loader_tags.py", line 127, in render
    compiled_parent = self.get_parent(context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/loader_tags.py", line 124, in get_parent
    return self.find_template(parent, context)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/loader_tags.py", line 103, in find_template
    template, origin = context.template.engine.find_template(
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/engine.py", line 125, in find_template
    template = loader.get_template(name, skip=skip)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/loaders/cached.py", line 58, in get_template
    template = super().get_template(template_name, skip)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/loaders/base.py", line 29, in get_template
    return Template(
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 155, in __init__
    self.nodelist = self.compile_nodelist()
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 193, in compile_nodelist
    return parser.parse()
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 478, in parse
    raise self.error(token, e)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/base.py", line 476, in parse
    compiled_result = compile_func(self, token)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/defaulttags.py", line 1094, in load
    lib = find_library(parser, name)
  File "/opt/rockstor/.venv/lib/python3.9/site-packages/django/template/defaulttags.py", line 1039, in find_library
    raise TemplateSyntaxError(
django.template.exceptions.TemplateSyntaxError: 'staticfiles' is not a registered tag library. Must be one of:
admin_list
admin_modify
admin_urls
cache
i18n
l10n
log
pipeline
rest_framework
static
tz

...

Ran 278 tests in 29.091s

FAILED (errors=5)

phillxnet · 2023-11-10T14:06:34Z

https://docs.djangoproject.com/en/dev/releases/3.0/#features-removed-in-3-0

Looks to be our culprit, with the above suggested new format:
https://docs.djangoproject.com/en/dev/releases/2.1/#deprecated-features-2-1

> {% load staticfiles %} and {% load admin_static %} are deprecated in favor of {% load static %}, which works the same.
~~django.contrib.staticfiles.templatetags.static() is deprecated in favor of django.templatetags.static.static().~~
> HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
> /opt/rockstor/src/rockstor/urls.py:80: RemovedInDjango40Warning: django.conf.urls.url() is deprecated in favor of django.urls.re_path().
url(r"^$", home, name="home"),

The 'RemovedInDjango40Warning' warnings are extremely numerous as they pertain to our entire url definitions, attempting to address here to avoid a large amount of warning logging spam.

phillxnet · 2023-11-10T14:13:04Z

We may also be subject to: https://docs.djangoproject.com/en/dev/releases/3.0/#features-removed-in-3-0

The field_name keyword argument of QuerySet.earliest() and latest() is removed.

Working on removing the forest of warnings and staticfiles messages first.

phillxnet · 2023-11-10T16:21:52Z

See: https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys

All warnings etc can be view via:

cd /opt/rockstor/
export DJANGO_SETTINGS_MODULE=settings
poetry run python -Wa .venv/bin/django-admin test

for our:

HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.

phillxnet · 2023-11-10T17:12:55Z

Thus far we now have, from django-admin test:

lbuildvm:/opt/rockstor # poetry run python -Wa .venv/bin/django-admin test
System check identified no issues (0 silenced).

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

And we have a full test suite pass:

export DJANGO_SETTINGS_MODULE=settings
cd /opt/rocksotr/src/rockstor/
poetry run django-admin test -v 2
...
----------------------------------------------------------------------
Ran 278 tests in 29.885s

OK

Update to Django 3.2.23, with incidental minor update to latest wrapt & charset-normalizer. # Includes - Required move to "load static" from "load staticfiles" in base/setup.html. - Settings option for DEFAULT_AUTO_FIELD: models.W042 - Project wide move from url to re_path: RemovedInDjango40Warning.

Update Django to next LTS #2734

phillxnet · 2023-11-11T13:41:49Z

Closing as:
Fixed by #2737

phillxnet added this to the Django 3.2 milestone Nov 8, 2023

phillxnet self-assigned this Nov 8, 2023

phillxnet mentioned this issue Nov 9, 2023

Reduce JS libs dependencies via HTMX 2 #2735

Open

1 task

phillxnet mentioned this issue Nov 10, 2023

Update Django to next LTS #2734 #2737

Merged

phillxnet added a commit that referenced this issue Nov 11, 2023

Merge pull request #2737 from phillxnet/2734-Update-Django-to-next-LTS

0f0f422

Update Django to next LTS #2734

phillxnet closed this as completed Nov 11, 2023

phillxnet mentioned this issue Nov 13, 2023

DeprecationWarning: distutils re Py3.10 #2696

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Django to next LTS #2734

Update Django to next LTS #2734

phillxnet commented Nov 8, 2023 •

edited

Loading

phillxnet commented Nov 10, 2023

phillxnet commented Nov 10, 2023

phillxnet commented Nov 10, 2023 •

edited

Loading

phillxnet commented Nov 10, 2023

phillxnet commented Nov 10, 2023 •

edited

Loading

phillxnet commented Nov 10, 2023 •

edited

Loading

phillxnet commented Nov 11, 2023

Update Django to next LTS #2734

Update Django to next LTS #2734

Comments

phillxnet commented Nov 8, 2023 • edited Loading

phillxnet commented Nov 10, 2023

Wrapt

Charset-normalizer

phillxnet commented Nov 10, 2023

...

phillxnet commented Nov 10, 2023 • edited Loading

phillxnet commented Nov 10, 2023

phillxnet commented Nov 10, 2023 • edited Loading

phillxnet commented Nov 10, 2023 • edited Loading

phillxnet commented Nov 11, 2023

phillxnet commented Nov 8, 2023 •

edited

Loading

phillxnet commented Nov 10, 2023 •

edited

Loading

phillxnet commented Nov 10, 2023 •

edited

Loading

phillxnet commented Nov 10, 2023 •

edited

Loading