feat(secretstores): Add systemd-credentials plugin

[srebhan]

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

The PR adds a secret-store plugin allowing to access systemd credentials from Telegraf (requires systemd 254+). This is helpful when starting Telegraf as a service without manual interaction. There the new plugin can serve as a initial secret-store to unlock others or to provide the secrets itself.

Please checkout https://systemd.io/CREDENTIALS/ for details.

Tested on ArchLinux and Ubuntu Server 23.04.

[jcpunk]

I have some limited interest in this.

[jdstrand]

The code and process looks fine. I've approved but added a number of comments to improve the documentation, but feel free to incorporate them and commit with another review from me.

[jdstrand]

I forgot to mention, systemd 250 is pretty new. I looked at various Enterprise distros and found:

• Ubuntu 22.04 LTS has 249 (22.10 (non-LTS) has 252 so next Ubuntu LTS (24.04) will)
• Debian Bookworm has 252
• RHEL 9.2 has 252
• SUSE Enterprise Server 15 SP5 has 249
• Amazon Linux 2023 has 252
• Oracle Linux 9 has 252

I then used lxd to create a bookworm VM:

$ lxc launch images:debian/bookworm/amd64 debian-bookworm --vm -c limits.cpu=2 -c limits.memory=4GiB --console=vga
$ lxc exec debian-bookworm --user=0 --group=0 -- mkdir -m 0700 /root/.ssh # may already exist
$ lxc file push $HOME/.ssh/id_rsa.pub debian-bookworm/root/.ssh/authorized_keys --mode=0600
$ lxc exec debian-bookworm --user=0 --group=0 -- chown root:root /root/.ssh/authorized_keys
$ lxc shell debian-bookworm
$ sudo apt-get update && sudo apt-get install openssh-server
# now can ssh with 'ssh root@debian-bookworm.lxd'

I could then add a remove a virtual tpm device with:

$ lxc config device add debian-bookworm vtpm tpm path=/dev/tpm0   # shutdown the vm before doing this
$ lxc config device remove debian-bookworm vtpm                   # shutdown the vm before doing this

This got me to see:

$ sudo systemd-creds has-tpm2
yes
+firmware
+driver
+system
+subsystem

which was cool, but it appears I needed to do more to actually use it:

$ echo foo | sudo systemd-creds encrypt - /etc/telegraf/credentials/testme
Credential secret file '/var/lib/systemd/credential.secret' is not located on encrypted media, using anyway.
TPM2 support not installed: Operation not supported
TPM2 sealing didn't work, continuing without TPM2: Operation not supported

but I was able to solve that with:

$ sudo apt-get install tpm2-tools
$ echo foo | sudo systemd-creds encrypt - /etc/telegraf/credentials/testme
Credential secret file '/var/lib/systemd/credential.secret' is not located on encrypted media, using anyway.
$ sudo systemd-creds decrypt /etc/telegraf/credentials/testme
foo

If I shutdown the VM, remove the tpm, start it and try to access the secret, I see (simulates decryption key in /var and /etc/telegraf/credentials being stolen and used on another machine):

$ sudo systemd-creds has-tpm2
partial
-firmware
-driver
+system
+subsystem
$ sudo systemd-creds decrypt /etc/telegraf/credentials/testme
ERROR:tcti:src/tss2-tcti/tcti-device.c:452:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: No such file or directory 
Failed to initialize TCTI context: tcti:IO failure

Cool!

[jdstrand]

We should consider documenting the use of PrivateMounts= as described in https://systemd.io/CREDENTIALS/:

"It is a good idea to also enable mount namespacing for services that process credentials configured this way. If so, the runtime credential directory of the specific service is not visible to any other service. Use PrivateMounts= as minimal option to enable such namespacing. Note that many other sandboxing settings (e.g. ProtectSystem=, ReadOnlyPaths= and similar) imply PrivateMounts=, hence oftentimes it’s not necessary to set this option explicitly."

[jdstrand]

This should be safe since the system mounts should be visible to telegraf but new mounts (like the one for the decrypted files) are not propagated outside of telegraf's private mount namespace, but will need testing. From https://manpages.ubuntu.com/manpages/lunar/en/man5/systemd.exec.5.html

PrivateMounts=
           Takes a boolean parameter. If set, the processes of this unit will be run in their own
           private file system (mount) namespace with all mount propagation from the processes
           towards the host's main file system namespace turned off. This means any file system
           mount points established or removed by the unit's processes will be private to them
           and not be visible to the host. However, file system mount points established or
           removed on the host will be propagated to the unit's processes. See
           [mount_namespaces](https://manpages.ubuntu.com/manpages/lunar/en/man7/mount_namespaces.7.html)(7) for details on file system namespaces. Defaults to off.

[srebhan]

I don't think we can do this as it also means that Telegraf cannot read files in the file-system which is required by various plugins...

[jdstrand]

That's the thing; my interpretation of the man page is that the system's private mounts all propagate into the container, but mounts within the container do not propagate into the host. This is also in the man page:

           When turned on, this executes three operations for each invoked process: a new
           CLONE_NEWNS namespace is created, after which all existing mounts are remounted to
           MS_SLAVE to disable propagation from the unit's processes to the host (but leaving
           propagation in the opposite direction in effect). Finally, the mounts are remounted
           again to the propagation mode configured with MountFlags=, see below.

This is why the flag is meaningful for this feature: without it, the mount is available to others but with it, it is not available to other services (that use PrivateMounts).

I did look into this further and found that:

• when the service is running and LoadCredentialsEncrypted=... is used, $CREDENTIALS_DIRECTORY is /run/credentials/telegraf.service in the service, and this directory has 0500 permissions with telegraf as the owner (decrypted files are 0400 with telegraf as owner)
• when the service is running and LoadCredentialsEncrypted=... is used, the decrypted credentials are in the root-protected /run/credentials/... directory on the host
• when the service is running and LoadCredentialsEncrypted=... is used and PrivateMounts IS NOT set, everything in /run/credentials on the host is available in /run/credentials in the service
• when the service is running and LoadCredentialsEncrypted=... is used and PrivateMounts IS set, only /run/credentials/telegraf.service on the host is available in /run/credentials in the service
• when services that specify LoadCredentialsEncrypted=... aren't running, then the decrypted credentials for those services are NOT present in /run/credentials on the host (or any running services)

Importantly, this has a few implications:

1. root on the host will always be able to see all the credentials via /run/credentials (but also by running the systemd-creds command, entering mount namespaces, adjusting unit files, etc). There is no protection against unrestricted root processes on the host
2. with or without PrivateMounts being set, the decrypted credentials have DAC permissions set to the User= in the systemd unit. Specifically for telegraf, only processes as the telegraf user can see the decrypted credentials in /run/credentials/telegraf.service on the host or within the service. Other non-root processes cannot see the credentials
3. with PrivateMounts NOT SET in a service's unit file that runs as ROOT, the root user can see all currently decrypted credentials for running services (eg, the foo.service doesn't specify User=foo (or specifies User=root), this service can see /run/credentials/foo.service/*, /run/credentials/telegraf.service/* and any other decrypted credentials for running services)
4. with PrivateMounts NOT SET in a service's unit file that runs as NON-ROOT, this non-root user can see all currently decrypted credentials for running services that are accessible to this non-root user
5. with PrivateMounts SET in a service's unit file that runs as ROOT, the root user can only see its own decrypted credentials (eg, the foo.service doesn't specify User=foo (or specifies User=root), this service can see /run/credentials/foo.service/* but not /run/credentials/telegraf.service/* or any other decrypted credentials for running services)
6. with PrivateMounts SET in a service's unit file that runs as NON-ROOT, this service can only see /run/credentials/foo.service/* but not /run/credentials/telegraf.service/* or any other decrypted credentials for running services regardless of if they run as the same user or not)

Practically speaking, this feature does not protect against attacker-controlled root processes or attacker-controlled root running services that don't set PrivateMounts as these processes can always see the decrypted credentials. PrivateMounts adds some protection to the telegraf service against attacker-controlled non-root running services. It adds considerable protection against attacker-controlled root running services when those root running services also specify PrivateMounts in their unit files (assuming they can't rewrite the unit file; see ProtectSystem=full in the man page).

[jdstrand]

Based on my investigation of PrivateMounts, I'm now changing this to 'Request changes' so the documentation can be updated to recommend using PrivateMounts. Feel free to adjust the unit file in the deb and rpm packaging to specify it by default (but that is not a blocker for this PR and can come at a later time). Please also mention in the docs why it is recommended (and ideally that it can offer protections when other units also specify it).

[srebhan]

@jdstrand thanks for the review. I hope I addressed all your points.
@powersj I added PrivateMounts = true to the service as requested by Jamie and I agree that this should (from documentation) work. If we see issues with not being able to access files we likely need to revert that. Just as a heads-up.

[jdstrand]

Thanks for the updates! I've given my approval so you're not blocked on me, but I gave some more suggestions to the readme.

[srebhan]

@jdstrand thanks for the feedback. I used all your suggestions so you might or might not want to take another look and approve!?

[jdstrand]

Thanks for the changes!

[powersj]

Thank you for the work on this and sorry this took so long. Mainly questions for my education and to verify we are on the same page.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[powersj]

Let's take it for v1.29! Thank you for the work and to @jdstrand for the reviews.