Update Huey task queue library

[phillxnet]

We have a number of usefull fixes from when we initially introduced (and pinned) Huey to 2.3.0. It is proposed that we update this pinning or adopt a major version only pin instead.
PyPi: https://pypi.org/project/huey/#history

Noteworthy additions from the upstream changelog: https://github.com/coleifer/huey/blob/master/CHANGELOG.md

• 2.5.0 Fix bug in SIGINT and SIGTERM behavior for gevent users.
• Add an on_commit_task() decorator for Django extension that will enqueue the task after any database changes have been committed. This eliminates a common race condition where a task is enqueued and executed before the corresponding database changes have been committed.
• Check to ensure the gevent monkeypatch was applied when running the consumer with greenlet workers, log warning if it is not.
• 2.4.1 Attempt to reconnect to database if connection becomes unusable (e.g. due to a server restart). See: huey.contrib.sql_huey.SqlHuey.
• 2.4.0 Task expiration: https://huey.readthedocs.io/en/latest/guide.html#task-expiration - we have had reports of task ongoing indefinitely that may benefit here.

We also have a number of other bug-fixes, and Django enhancements in the interim releases.

[phillxnet]

By removing the pinning, at least for now, we have:

lbuildvm:/opt/rockstor # poetry update
Updating dependencies
Resolving dependencies... (10.0s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating huey (2.3.0 -> 2.5.0)

[phillxnet]

Testing

Balance

[11/Nov/2023 17:28:57] INFO [storageadmin.tasks:64] Task [start_balance], id: edc582b8-22e3-4ff8-98dd-840ef999a190 completed OK
[11/Nov/2023 17:28:57] INFO [storageadmin.tasks:67] Updating end_time accordingly to 2023-11-11 17:28:57.093087+00:00
[11/Nov/2023 17:28:57] INFO [storageadmin.tasks:56] Now executing Huey task [update_end_time], id: bca8a0af-ecdd-4e00-99cc-2be52b8aa360.
[11/Nov/2023 17:28:57] INFO [storageadmin.tasks:64] Task [update_end_time], id: bca8a0af-ecdd-4e00-99cc-2be52b8aa360 completed OK

With the corresponding contents in huey's dedicated log file: /opt/rockstor/var/log/huey.log

[2023-11-11 17:14:02,002] INFO:huey.consumer:MainThread:Huey consumer started with 2 thread, PID 13447 at 2023-11-11 17:14:02.002631
[2023-11-11 17:14:02,003] INFO:huey.consumer:MainThread:Scheduler runs every 1 second(s).
[2023-11-11 17:14:02,003] INFO:huey.consumer:MainThread:Periodic tasks are enabled.
[2023-11-11 17:14:02,003] INFO:huey.consumer:MainThread:The following commands are available:
+ fs.btrfs.start_resize_pool
+ fs.btrfs.start_balance
+ smart_manager.views.ztask_helpers.ztask_helpers.restart_rockstor
+ storageadmin.views.pool_balance.update_end_time
+ storageadmin.views.rockon_helpers.start
+ storageadmin.views.rockon_helpers.stop
+ storageadmin.views.rockon_helpers.update
+ storageadmin.views.rockon_helpers.install
+ storageadmin.views.rockon_helpers.uninstall
+ storageadmin.views.config_backup.restore_rockons
+ storageadmin.views.config_backup.restore_config
[2023-11-11 17:27:59,307] INFO:huey:Worker-2:Executing fs.btrfs.start_balance: edc582b8-22e3-4ff8-98dd-840ef999a190
[2023-11-11 17:28:57,092] INFO:huey:Worker-2:fs.btrfs.start_balance: edc582b8-22e3-4ff8-98dd-840ef999a190 executed in 57.784s
[2023-11-11 17:28:57,094] INFO:huey:Worker-2:Executing storageadmin.views.pool_balance.update_end_time: bca8a0af-ecdd-4e00-99cc-2be52b8aa360
[2023-11-11 17:28:57,214] INFO:huey:Worker-2:storageadmin.views.pool_balance.update_end_time: bca8a0af-ecdd-4e00-99cc-2be52b8aa360 executed in 0.120s

Rock-on install:

rockstor.log

[11/Nov/2023 17:43:14] INFO [storageadmin.tasks:56] Now executing Huey task [install], id: 94a94b9b-ad71-4737-8d63-277c234208f4.
[11/Nov/2023 17:43:14] ERROR [system.osi:261] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'netdata_official']. output: [''] error: ['Error response from daemon: No such container: netdata_official', '']
[11/Nov/2023 17:43:14] ERROR [system.osi:261] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'netdata_official']. output: [''] error: ['Error response from daemon: No such container: netdata_official', '']
[11/Nov/2023 17:43:45] INFO [storageadmin.tasks:64] Task [install], id: 94a94b9b-ad71-4737-8d63-277c234208f4 completed OK

huey.log

[2023-11-11 17:43:14,516] INFO:huey:Worker-2:Executing storageadmin.views.rockon_helpers.install: 94a94b9b-ad71-4737-8d63-277c234208f4
[2023-11-11 17:43:45,215] INFO:huey:Worker-2:storageadmin.views.rockon_helpers.install: 94a94b9b-ad71-4737-8d63-277c234208f4 executed in 30.698s

Rock-on stop

rockstor.log

[11/Nov/2023 17:58:09] INFO [storageadmin.tasks:56] Now executing Huey task [stop], id: d5dbacc0-8b9d-408d-a0a9-e037342ed6ce.
[11/Nov/2023 17:58:13] INFO [storageadmin.tasks:64] Task [stop], id: d5dbacc0-8b9d-408d-a0a9-e037342ed6ce completed OK

huey.log

[2023-11-11 17:58:09,960] INFO:huey:Worker-2:Executing storageadmin.views.rockon_helpers.stop: d5dbacc0-8b9d-408d-a0a9-e037342ed6ce
[2023-11-11 17:58:13,785] INFO:huey:Worker-2:storageadmin.views.rockon_helpers.stop: d5dbacc0-8b9d-408d-a0a9-e037342ed6ce executed in 3.824s

Rock-on uninstall

rockstor.log

[11/Nov/2023 18:00:50] INFO [storageadmin.tasks:56] Now executing Huey task [uninstall], id: 6e3bff2a-ca30-4f84-bc74-5ddacf51b716.
[11/Nov/2023 18:00:50] INFO [storageadmin.tasks:64] Task [uninstall], id: 6e3bff2a-ca30-4f84-bc74-5ddacf51b716 completed OK

huey.log

[2023-11-11 18:00:50,118] INFO:huey:Worker-1:Executing storageadmin.views.rockon_helpers.uninstall: 6e3bff2a-ca30-4f84-bc74-5ddacf51b716
[2023-11-11 18:00:50,336] INFO:huey:Worker-1:storageadmin.views.rockon_helpers.uninstall: 6e3bff2a-ca30-4f84-bc74-5ddacf51b716 executed in 0.218s

config restore

A config was saved (no rock-ons installed) and then directly there after restored.
rockstor.log

[11/Nov/2023 18:04:17] INFO [storageadmin.tasks:56] Now executing Huey task [restore_config], id: f54c451d-4f68-4c4b-86d6-0033f9e58fd9.
...
... N.B. we have some entries of:
Exception: Exception while setting access_token for url(https://localhost). Make sure credentials are correct: 'access_token'. Moving on.
... But they are outside the scope of this issues.
...
[11/Nov/2023 18:04:18] INFO [storageadmin.tasks:64] Task [restore_config], id: f54c451d-4f68-4c4b-86d6-0033f9e58fd9 completed OK
[11/Nov/2023 18:04:18] INFO [storageadmin.tasks:56] Now executing Huey task [restore_rockons], id: 6bb00698-08a8-40db-9afe-be66cfbdf8d6.
...
N.B. again with the same access_token issue
...
[11/Nov/2023 18:04:18] INFO [storageadmin.tasks:64] Task [restore_rockons], id: 6bb00698-08a8-40db-9afe-be66cfbdf8d6 completed OK

huey.log

[2023-11-11 18:04:17,016] INFO:huey:Worker-1:Executing storageadmin.views.config_backup.restore_config: f54c451d-4f68-4c4b-86d6-0033f9e58fd9
[2023-11-11 18:04:18,402] INFO:huey:Worker-1:storageadmin.views.config_backup.restore_config: f54c451d-4f68-4c4b-86d6-0033f9e58fd9 executed in 1.385s
[2023-11-11 18:04:18,405] INFO:huey:Worker-1:Executing storageadmin.views.config_backup.restore_rockons: 6bb00698-08a8-40db-9afe-be66cfbdf8d6
[2023-11-11 18:04:18,639] INFO:huey:Worker-1:storageadmin.views.config_backup.restore_rockons: 6bb00698-08a8-40db-9afe-be66cfbdf8d6 executed in 0.234s

[phillxnet]

Closing as:
Fixed by #2741