Adopt dedicated secrets management library #2728
Comments
|
An alternative, but a little more complex/heavy-weight approach here, could be to 'standardise' on keyring: https://github.com/jaraco/keyring
This library is more naively concerned with desktop associated keystores, i.e. from the README.rst again:
However we have also the option of:
Among these indicated "third-party-backends" we have a potential candidate of:
Where from the linked (in PyPi) GitHub homepage of this project we have: https://github.com/frispete/keyrings.cryptfile#notes
So all a little tricky as we ultimately have to have a de-crypt key (password) for any encryption we rely-upon such as the encrypted local file storage, but I like the idea that we could, without only superficial extensions, switch out the secrets backend used by keyring. Maybe a half way step here is to at lease adopt the maturity of keyring, to gain our standard interface, and initially side step the tricky encryption requirement for interaction, i.e. entry of a decrypt secret (our host OS has no appropriate user logged in) and initially go with say: (again from https://github.com/jaraco/keyring#third-party-backends):
PyPi page: https://pypi.org/project/keyrings.alt/ But:
However this is the exact same scenario we face with the use of say python-dotenv: i.e. plain text passwords/secrets !! But we can iterate to more advanced back-ends as we progress. But this route does seem like a lot of dependencies to move plain-text secrets from one file to another !! Suggestions welcome here. We are, after all, sitting on a base OS with all required options available: but we ultimately still need to open a keystore of some sorts automatically give we have no OS login as-such. |
|
With all the reading I did this morning, it seems to me that the keyring stuff is all good especially for centralizing logins/passphrases, etc., however, since it does require the using of some (master) password at some point AND we don't have (as you pointed out) a login process during startup, this would still result in some plaintext password somewhere (or decryptable password by looking at the coding). https://github.com/jaraco/keyring#using-keyring-on-headless-linux-systems I assume the Gnome backend will continue to live on for a long time with continuous security improvements, so even if we eventually figure out a process/configuration that allows for "password entry required" upon startup somehow (or read a dongle, HW config, or some quantum physics related entity :)), then the backend piece will be up-to-date and secure. But, of course all that I have concluded from above might be incorrect ... |
|
@Hooverdan96 Thanks, yes this is all quite the challenge.
Hence me thinking the crypt file backend would be lighter, and have the same capability, and the same complexity re feeding a password from presumably an env we setup via Poetry or something. The gnome keyring may be a better bet, given it's native support in the keyring module, than the third party module indicated above. But it may also have more weight/surface area. Tricky. |
|
you are probably right. I think (if I saw this correctly for a test install, disregarding any version pinning that might be required, or other dependencies/already existing items) for a few of the mentioned backends, the sizes are not small: gnome-keyring - 2.4 MiB. With dependencies (330 packages) installed 513.6 MiB Installing the keyrings cryptfile 1.3-9 with dependencies, seems to be ~17MiB: so, in either case, not totally trivial if you want to keep the footprint relatively small... |
|
@Hooverdan96 Thanks, I was wondering about the actual sizes. I've used the gnome-keyring before, but that was on a desktop so no extra was required (gnome back then). Re the base:
That's nice and small, plus we already have cffi and pycparser and cryptography I think (some incoming via the Django-oath-toolkit I think). |
|
A Poetry update introduces more flexibility re env variables: noting here in case this helps us out re establishing install persistent secrets that are, if not found, randomly generated and established via Poetry. |
|
We likely have at least 2 regression following on from "Update django-oauth-toolkit #2710" #2727
|
|
Regarding a follow-up on:
We are now, post: #2755 using the latest Poetry: however without an unofficial plugin (risky for such a critical part of our setup) Poetry does not support environmental variables natively, and it is definitely not favoured as in-scope by the main developers of Poetry: |
PyPI
|
|
Potential extension of OS dependency re keyrings.cryptfile: Quick start guide in https://pypi.org/project/keyrings.cryptfile/
|
|
With just the (insufficient) addition of:
We have the following changes: With the first two likely incidental. Adding:
Results in the following error: lbuildvm:/opt/rockstor # poetry update |
|
From: python-poetry/poetry#6858 Some context, apparently Poetry itself uses keyring: https://blog.frank-mich.com/python-poetry-1-0-0-private-repo-issue-fix/ So likely the self-add is relevant there. |
|
Unrelated to this issue, but wanted to note here for reference: |
|
@FroggyFlox Re:
That good to know, especially given the semi-maintained nature of python-dbus that was a bit of a shock. As a follow-up on this issue; I'm now looking into OS keyring as likely the better place to have this resource located anyway; i.e.: But still fathoming our options here. We will likely have to have some related changes in rpmbuild and our installer as a result: but we are then not carrying such sensitive packages that we should be depending on our OS to provide, especially give the final hurdle of needing an on filesystem password !!! I.e. we ultimately have to unlock our passwords without a login !!! |
|
My current though on establishing the cryptfile password is to set it to a long random during install, via a one-shot systemd service that embeds into /root a secured file with the master pass within it (in plain text). Ideas on how we get around this ultimate requirement: bar having folks type it in on every boot, are welcome. |
Re keyring keyrings.alt versions from OS:As above we have python311-keyring: 24.2.0-150400.5.3.1 = fairly up-to-date. But the package: python3-keyrings.alt is a little old at 4.0.2 ish: With a PyPi version: https://pypi.org/project/keyrings.alt/ at 5.0.0 https://github.com/jaraco/keyrings.alt/blob/main/NEWS.rst#v412
https://github.com/jaraco/keyrings.alt/blob/main/NEWS.rst#v420
|
OS python311-keyringWe also have the spin-off repo: https://github.com/jaraco/keyrings.alt from the same maintainer where we find our proposed cryptfile backend option as part of a bundle. And as referenced before: https://pypi.org/project/keyring/#third-party-backends
PyPi: https://pypi.org/project/keyrings.alt/ Using OS keyring package as above, plus poetry dependency: Results in the same:
|
|
N.B. keyrings.alt does not contain the keyrings.cryptfile backend. |
|
Another way we can introduce the master-unlock password, for what-ever we settle on here re backend keystore, is via: -EnvironmentFile= https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#EnvironmentFile= which will expose the contents of a file as environment variables, but they are more secure than "Environment=".
|
|
It seems we have no clear way to install via Poetry our initially proposed keyrings.cryptfile, and it is not available to the OS. Keyring very much looks to be desktop orientated, but I'm still considering other options re the backend as the gnome.keyring is massive and pulls in such things as Mesa !!! and x keyboard stuff. We have the following 3rd party keystores as hopefully more appropriate for our headless approach and more amenable to Poetry or OS install: From: https://keyring.readthedocs.io/en/latest/#third-party-backends
|
keyring_pass** N.B. password-store itself looks to be authored by Jason A. Donenfeld, the author of WireGuard !! ** So the only potentially weak link here is the the interface project of keyring_pass, recent but few releases. But this choice would at least get us going again on this issue and instantiate in rockstor-core the keyring client: where the backend could be up for swap-out. But still - we really don't what to have to switch this out later. But we could if need be. Or end up maintaining keyring_pass, or an fork, which is not necessarily a show-stopper. Keyring_pass does look to have Py3.11 compatibiliy currently, and password-store is the secure side of things anyway so we may have our answer on this relatively simple approach - accessible via python or the shell. |
|
The result of adding to pyproject.toml just:
results in: Also pulls in keyring, althought it is not the latest at Dec 2022 !! But the secretstorage is latest, as is jaroco-classes and (@FroggyFlox ) jeepney dependency is also latest. Dependenncies via poetry: |
'password-store' from OS packageas the final backend Where removing --no-recommends results in the same package dependencies. |
|
As password-store relies on GPG we need to generate a public/private GPG key pair for use in encryption/decryption of our password-store. Manual: https://www.gnupg.org/documentation/manuals/gnupg/
But from: https://www.gnupg.org/documentation/manuals/gnupg/The-quick-key-manipulation-interface.html
And finally from: https://www.gnupg.org/documentation/manuals/gnupg/Programmatic-use-of-GnuPG.html
However from: We have a very old and likely inappropriate package: https://pypi.python.org/pypi/pygpgme |
|
GPGME stands for GnuPG made easy: https://www.gnupg.org/documentation/manuals/gpgme/Generating-Keys.html so really orientated towards programmatic interface only. So falling back to using the GPG interface created to GPGME: https://www.gnupg.org/documentation/manuals/gnupg/OpenPGP-Key-Management.html#OpenPGP-Key-Management To generate gpg keys for the root user (UID 0) - asks for password at terminal so we can't script as is. But if no/empty pass, and we rely on default filesystem security we can do: And our root user (GID 0) then has: |
|
As we use the --yes option above we do not have idempotency !! and end up rewriting our pgp info: Resetting via: lists no secrets. |
Proposed initial creation command (from systemd) via idempotent command:POC Status: Re-runnning the same setup command again: |
|
N.B. the user-id is not a UID as indicated above - but an identifier for the key holder !!! |
|
Test to see if we have sufficent pgp config in place with our new key/config created.
And the creation/addition of a password for our CLIENT_SECRET, via cli could be:
Attempted re-creation the same named password gives terminal interaction: Remove via: We can also specify the length of the generated password i.e. to replace our SECRET_KEY: We will likely create via python and the python-keyring, but we may have to script within services or in rpmbuild and the above establishes pass function before moving on to the pass-to-python-keyring shim (keyring-pass) to enable our python-keyring access to password-store managed secrets. |
Quick .venv test of autoconfig re keyring backend |
venv store passcli retrieve:with contents via: python venv retrieval of both cli and python created passwords next !! |
Keyring backed infokeyring has a command line option that we can utilise within our .venv: Help (keyring cli) |
keyring cli setsetgetpass shell retrieval: |
Summary
|
set in SHELL via passget_password within .venv via python-keyringforce generate/refresh password (SHELL):I.e. to cycle our SECRET_KEY predominantly session related Django secret on each update, or boot. |
Silence output of passWe need not log/expose the "pass generate ..." password: ergo stdout to /dev/null [EDIT] curiously this was not practicable/workable as-is within systemd ExecStartPre, so for now I've nulled the stdout from rockstor-pre, we log to rockstor.log anyway and stderr is still defaulted to journal. |
Adopt password-store by Jason A. Donenfeld of wireguard fame as our base OS password store; under the root user. Employ as backed to poetry-keyring via adapter shim keyring-pass. Enabling lightweight/secure (via GPG encryption) secrets management from both an OS level and from within our Python Poetry venv. # Includes: - Keyring-pass additional dependency, with secondary dependency on python-keyring. - ExecStartPre additions to initialise GNUPG, pass, and rotate/generate Django's SECRET_KEY. - PASSWORD_STORE_DIR environmental variable in all main rockstor systemd services. - Add Django's new-in-4.1 SECRET_KEY_FALLBACKS setting. - Incidental update to cryptography: 41.0.5 to 41.0.7. - Incidental update to idna: 3.4 to 3.6. - TODO: re CLIENT_SECRET in settings.py as work-in-progress.
- Set CLIENT_SECRET in keyring via initrock.py, and maintain as install instance persistent. Set only during db initialisation, or if it does not exist: i.e. updating from pre keyring install. - Get CLIENT_SECRET (settings.py) from keyring. - Update build.sh with developer instructions re GnuPG & pass. - Incidental addition of Poetry 1.2 style dev dependencies group, with the addition of `black` as an optional install. - Incidental black re-format of initrock.py - Simplify logging within initrock.py.
- Additional minimal setup of GNUPG & pass in build.sh as we need a valid Django config for collectstatic, and we now store SECRET_KEY in OS provided 'pass'.
Adopt password-store by Jason A. Donenfeld of wireguard fame as our base OS password store; under the root user. Employ as back-end to python-keyring via adapter shim keyring-pass. Enabling lightweight/secure (via GPG encryption) secrets management from both an OS level and from within our Python Poetry venv. # Includes: - Keyring-pass additional dependency, with secondary dependency on python-keyring. - ExecStartPre additions to initialise GNUPG, pass, and rotate/generate Django's SECRET_KEY. - PASSWORD_STORE_DIR environmental variable in all main rockstor systemd services. - Add Django's new-in-4.1 SECRET_KEY_FALLBACKS setting. - Incidental update to cryptography: 41.0.5 to 41.0.7. - Incidental update to idna: 3.4 to 3.6. - Set CLIENT_SECRET in keyring via initrock.py, and maintain as install instance persistent. Set only during db initialisation, or if this key does not exist: i.e. updating from pre keyring install. - Get CLIENT_SECRET (settings.py) from keyring. - Update build.sh with developer instructions re GnuPG & pass. - Additional minimal setup of GNUPG & pass in build.sh as we need a valid Django config for collectstatic, and we now store SECRET_KEY in OS provided 'pass'. - Incidental addition of Poetry 1.2 style dev dependencies group, with the addition of `black` as an optional install. - Incidental black re-format of initrock.py - Simplify logging within initrock.py.
…management-library Adopt dedicated secrets management library #2728
|
Closing as: |
We have an ongoing need to centralise our secrets management. This need has more recently been highlighted within the following PR/Issue: "Update django-oauth-toolkit #2710" #2727 where we had need to establish a new means to maintain an install-persistent OAUTH_INTERNAL_APP client secret. As this function was deemed to be outside the scope of the linked issue/PR the proposal is being made in this issue. The temporary work-around in #2727 was to establish a per-boot/rockstor-bootstrap.service persistent secret. This however is inappropriate for such facilities as replication where we require install persistence for secrets.
Popular candidates would be:
Proposed initial focus would be our settings.py file.
The text was updated successfully, but these errors were encountered: