Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rockstor-build systemd service #2793

Closed
phillxnet opened this issue Jan 31, 2024 · 6 comments
Closed

Add rockstor-build systemd service #2793

phillxnet opened this issue Jan 31, 2024 · 6 comments
Assignees

Comments

@phillxnet
Copy link
Member

It is proposed that we move the executing of our build.sh shell script, from the rpm %posttrans scriptlet, to a one-shot systemd service that runs if of our .venv directory is not found. Our Python virtual environment is a pre-requisite of our other services and as such we can more easily user surface & develop this scripts failure/function if it is more properly located within our host OS's main orchestration.

Our current use of rpm's %posttrans scriptlet has surfaced a number of sensitivities/constraint, indicating that we are likely stretching the intended use of this somewhat constrained post rpm transaction scriptlet: i.e. contents are executed after the rpm package transaction has completed (we need our new build.sh to be in place after all) but we then face the constraints of a failed %posttrans execution and it's consequent failure to our other systemd services (no .venv). Moving/delaying build.sh's execution to the systemd services phase, which are auto-started post rpm install, means we can more properly place our required early OS arrangement under OS orchestration, rather than squeezing them into the constrained/limited last step of the packaging system.

It would also be significantly easier to develop further on build.sh it it were run from a more configurable environment such as systemd is designed to provide. As-is we have had a number of failures that pertain only to rpm's scriptlet env that otherwise are not seen in a regular terminal. And build.sh testing is then trivially accomplished by exercising it's proposed dedicated systemd script, rather than having to build an rpm and test that rpms's install/update function just to ensure we maintain function in that env.

We are also likely over-stepping the intended boundaries of %posttrans when we install additional programs; e.g our uninstall/install/upgrade maintenance of our build system Poetry: such as we do in build.sh. And additionally we are, more recently, using OS provided pipx to do this. Further complicating our requirements of a packaging system.

There are also odd corner cases to our current approach such as has recently been discovered in our latest installer build developments. See:
"Tumbleweed: build.sh fails as no /etc/locale.conf": rockstor/rockstor-installer#156

However this would dictate that our installer will have to have internet access in order that the first invocation of rockstor-build.service could download from PyPi our required .venv dependencies. But that is assumed for a modern OS instance anyway for all other OS updates. It would also remove the special case of our installer having a pre-build .venv when no other rpm has this: our rpm install/update auto-wipes the install location prior .venv directory: hence using this as an flag for build.sh's systemd service execution.

@phillxnet
Copy link
Member Author

Dev notes

Our 'flag' condition: we only run if the .venv does not exist. Such as after an rpm upgrade; where we are then required to re-establish the .venv to the new specifications defined by the freshly installed package.

systemctl status rockstor-build.service 
○ rockstor-build.service - Build Rockstor
     Loaded: loaded (/usr/lib/systemd/system/rockstor-build.service; enabled; preset: disabled)
     Active: inactive (dead)
  Condition: start condition unmet at Thu 2024-02-01 15:44:01 WET; 7min ago

Feb 01 15:44:01 installer systemd[1]: Build Rockstor was skipped because of an unmet condition check (ConditionPathIsDirectory=!/opt/rockstor/.venv).

@phillxnet
Copy link
Member Author

rockstor-build.service

With no .venv dir, and in this test case also no Poetry, we have the following status output from the proposed rockstor-build service:

systemctl status rockstor-build.service 
● rockstor-build.service - Build Rockstor
     Loaded: loaded (/usr/lib/systemd/system/rockstor-build.service; enabled; preset: disabled)
     Active: active (exited) since Thu 2024-02-01 17:56:05 WET; 2min 17s ago
   Main PID: 1460 (code=exited, status=0/SUCCESS)
      Tasks: 5
        CPU: 17.598s
     CGroup: /system.slice/rockstor-build.service
             ├─1770 gpg-agent --homedir /root/.gnupg --use-standard-socket --daemon
             └─1772 scdaemon --multi-server

Feb 01 17:56:05 installer build.sh[1460]: ROCKSTOR BUILD SCRIPT COMPLETED
Feb 01 17:56:05 installer build.sh[1460]: If installing from source, from scratch, for development; i.e. NOT via RPM:
Feb 01 17:56:05 installer build.sh[1460]: Note GnuPG & password-store ExecStartPre steps in /opt/rockstor/conf/rockstor-pre.ser>
Feb 01 17:56:05 installer build.sh[1460]: 1. Run 'cd /opt/rockstor'.
Feb 01 17:56:05 installer build.sh[1460]: 2. Run 'systemctl start postgresql'.
Feb 01 17:56:05 installer build.sh[1460]: 3. Run 'export DJANGO_SETTINGS_MODULE=settings'.
Feb 01 17:56:05 installer build.sh[1460]: 4. Run 'export PASSWORD_STORE_DIR=/root/.password-store'.
Feb 01 17:56:05 installer build.sh[1460]: 5. Run 'poetry run initrock' as root (equivalent to rockstor-pre.service ExecStart).
Feb 01 17:56:05 installer build.sh[1460]: 6. Run 'systemctl enable --now rockstor-bootstrap'.
Feb 01 17:56:05 installer systemd[1]: Finished Build Rockstor.

@phillxnet
Copy link
Member Author

No .venv so rockstor-build is run:

full-build-service-run-2793

subsequent boot

We have our .venv so the build service is not executed:

subsequent-reboot-so-no-build-2793

@phillxnet
Copy link
Member Author

Poetry-install-txt

Contents generated from build.sh's execution from within systemd via the proposed (in pending PR) rockstor-build.service during a boot where our pipx installed Poetry was not installed, and there was no existing flag/venv dir of .venv:

cat /opt/rockstor/poetry-install.txt 
PIPX_MAN_DIR=/usr/local/share/man
PIPX_BIN_DIR=/usr/local/bin
PWD=/opt/rockstor
SYSTEMD_EXEC_PID=1460
LANG=en_GB.UTF-8
INVOCATION_ID=2cc4e1c64460439aad8ea5db878e5269
DJANGO_SETTINGS_MODULE=settings
PIPX_HOME=/opt/pipx
SHLVL=1
JOURNAL_STREAM=8:9217
PATH=/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PASSWORD_STORE_DIR=/root/.password-store
_=/usr/bin/env
Poetry (version 1.7.1)
Loading configuration file /opt/rockstor/poetry.toml
Creating virtualenv rockstor in /opt/rockstor/.venv
Using virtualenv: /opt/rockstor/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 50 installs, 0 updates, 0 removals, 1 skipped

  • Installing pycparser (2.21)
  • Installing cffi (1.16.0)
  • Installing h11 (0.14.0)
  • Installing cryptography (41.0.7)
  • Installing jeepney (0.8.0)
  • Installing more-itertools (10.1.0)
  • Installing wrapt (1.16.0)
  • Installing wsproto (1.2.0)
  • Installing zipp (3.17.0)
  • Installing asgiref (3.7.2)
  • Installing certifi (2023.11.17)
  • Installing charset-normalizer (3.3.2)
  • Installing deprecated (1.2.14)
  • Installing idna (3.6)
  • Installing importlib-metadata (6.8.0)
  • Installing jaraco-classes (3.3.0)
  • Installing secretstorage (3.3.3)
  • Installing simple-websocket (1.0.0)
  • Installing sqlparse (0.4.4)
  • Installing urllib3 (2.1.0)
  • Installing bidict (0.22.1)
  • Installing django (4.2.7)
  • Installing greenlet (3.0.1)
  • Installing jwcrypto (1.5.0)
  • Installing keyring (23.13.1)
  • Installing oauthlib (3.2.2)
  • Installing packaging (23.2)
  • Installing python-engineio (4.8.0)
  • Installing pytz (2023.3.post1)
  • Installing requests (2.31.0)
  • Installing typing-extensions (4.8.0)
  • Installing zope-event (5.0)
  • Installing zope-interface (6.1)
  • Installing dbus-python (1.3.2)
  • Installing distro (1.8.0)
  • Installing django-pipeline (2.1.0)
  • Installing django-oauth-toolkit (2.3.0)
  • Installing djangorestframework (3.14.0)
  • Installing gevent (23.9.1)
  • Installing gunicorn (21.2.0)
  • Installing huey (2.5.0)
  • Installing keyring-pass (0.8.1)
  • Installing psutil (5.9.4)
  • Installing psycogreen (1.0)
  • Installing psycopg (3.1.13)
  • Installing python-socketio (5.9.0)
  • Installing pyzmq (25.1.1)
  • Installing setuptools (69.0.2): Skipped for the following reason: Already installed
  • Installing six (1.16.0)
  • Installing supervisor (4.2.4)
  • Installing urlobject (2.1.1)

Installing the current project: rockstor (5.0.7)
  - Building package rockstor in editable mode
  - Adding rockstor.pth to /opt/rockstor/.venv/lib/python3.11/site-packages for /opt/rockstor
  - Adding the backup-config script to /opt/rockstor/.venv/bin
  - Adding the bootstrap script to /opt/rockstor/.venv/bin
  - Adding the data-collector script to /opt/rockstor/.venv/bin
  - Adding the debug-mode script to /opt/rockstor/.venv/bin
  - Adding the delete-api-key script to /opt/rockstor/.venv/bin
  - Adding the delete-rockon script to /opt/rockstor/.venv/bin
  - Adding the flash-optimize script to /opt/rockstor/.venv/bin
  - Adding the initrock script to /opt/rockstor/.venv/bin
  - Adding the mnt-share script to /opt/rockstor/.venv/bin
  - Adding the ovpn-client-gen script to /opt/rockstor/.venv/bin
  - Adding the ovpn-client-print script to /opt/rockstor/.venv/bin
  - Adding the ovpn-initpki script to /opt/rockstor/.venv/bin
  - Adding the prep_db script to /opt/rockstor/.venv/bin
  - Adding the pwreset script to /opt/rockstor/.venv/bin
  - Adding the qgroup-clean script to /opt/rockstor/.venv/bin
  - Adding the qgroup-maxout-limit script to /opt/rockstor/.venv/bin
  - Adding the replicad script to /opt/rockstor/.venv/bin
  - Adding the send-replica script to /opt/rockstor/.venv/bin
  - Adding the st-pool-scrub script to /opt/rockstor/.venv/bin
  - Adding the st-snapshot script to /opt/rockstor/.venv/bin
  - Adding the st-system-power script to /opt/rockstor/.venv/bin
  - Adding the rockstor-5.0.7.dist-info directory to /opt/rockstor/.venv/lib/python3.11/site-packages

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Feb 1, 2024
Move build.sh execution from within rpm %posttrans script to
it's own dedicated rockstor-build.service. Enabling greater
fidelity and control over the environment and timing; and eases
development and user feedback on build.sh failures in the future.

Partnered with rockstor.spec changes in rockstor-rpmbuild repo.

## Includes
- New rockstor-build.service file.
- After= & Requires= entries in rockstor-pre on rockstor-build,
to extend our service cascade.
- Trivial build.sh and pkg_mgmt.py comment updates.
- Add the new rockstor-build.service to initrock.py to assist
in asserting the service akin to all other rockstor services.
Mostly redundant given our rpm service management, but nice-to-have.
@Hooverdan96
Copy link
Member

good stuff!

phillxnet added a commit that referenced this issue Feb 2, 2024
…md-service

Add rockstor-build systemd service #2793
@phillxnet
Copy link
Member Author

Closing as:
Fixed by #2795

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants