[Feature] Add smartmontools to Telegraf Docker Image

[mikekuzak]

Hi

I'm using telegraf:1.20.2-alpine and I'm starting it up with the following commands:
command: bash -c "apk add smartmontools lm-sensors-detect && telegraf"

The newer docker container is not running as root anymore so I can't add myself the needed packages.
There is a 'smart' section in the conf file but without having the binaries in the images this fuction is useless.

Thanks

[powersj]

The containers were updated to only run telegraf as the telegraf user. Users should still be able to install packages now, but you may need to update to the latest docker image.

If you find this is not the case, please do let us know which image you are using and the exact way you are launching the image. Thanks!

[phillipjf]

@powersj any recommendations for achieving this without building a custom image on the latest images (At time of writing this is 1.21.4)? The permissions on the telegraf user no longer allow this:

$ docker run -it 'telegraf:alpine' /bin/bash -c 'apk update && apk add ipmitool && telegraf'
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied

$ docker run -it 'telegraf' /bin/bash -c 'apt-get update && apt-get install ipmitool && telegraf'
Reading package lists... Done
E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)

[powersj]

Hi,

When we originally changed to running as the telegraf user it prevented any additional software install, even with a custom image. This was clearly not our intention. The reference to the latest image was that we fixed this to allow users to still connect to the container as root and make changes.

That said, with the configuration change of running as non-root, the cleanest solution is to create a custom image with whatever software is required. You could exec post-launch and install software, but that is not as clean.

We realize that making this change does prevent users from doing everything up-front like you have in your message, but the security concerns of running telegraf as root all the time were too great.

Hope that helps

[phillipjf]

This is very cumbersome for usages like Unraid where templates are provided through Community Applications. The ability to update a container quickly based on an "authoritative" source is lost. The closest workaround I've found is to the previous method is to copy the entrypoint.sh from this repository and modify it to include the appropriate install command (apk/apt-get). I mount this as a volume at the same path as the ENTRYPOINT in the Dockerfile (/entrypoint.sh). Less than ideal but easier than maintaining my own build.

[kennymc-c]

@phillipjf Where exactly did you put the install command inside entrypoint.sh? I tried it at the beginning and after the first if function. But the tools are still missing when I start the container.

[phillipjf]

@kennymc-c if you don't have it already, I believe you'd need to run the container with --user=root. This, obviously, nullifies the changes the influxdata team have made to make the container more secure, so use at your own risk.

entrypoint.sh:

#!/bin/bash
set -e

apt-get update -y && apt-get install -y ipmitool

if [ "${1:0:1}" = '-' ]; then
...
# rest of `entrypoint.sh`

$ docker run --user=root -v "$(pwd)"/entrypoint.sh:/entrypoint.sh -it 'telegraf'

[kennymc-c]

@phillipjf Unfortunately this does not work for me. I use the alpine image and therefore apk instead of apt-get. With --user=root as extra parameter in Unraid 6.10.2 I get the following error message at startup:

ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied

I then tried the normal image with the matching entrypoint.sh and used apt-get instead. Here it runs fine at first but then I get the following message in the log which probably indicates that Telegraf itself is not running as root user after all

2022-05-31T17:54:30Z E! [inputs.ipmi_sensor] Error in plugin: failed to run command /usr/bin/ipmitool sdr: exit status 1 - Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
2022-05-31T17:54:30Z E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied
2022-05-31T17:54:30Z E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0": dial unix /var/run/docker.sock: connect: permission denied 

[phillipjf]

right -- this is another drawback. you can read more here on how to run the container to monitor docker:
https://www.influxdata.com/blog/docker-run-telegraf-as-non-root/

The only suggestion I have for being able to do both monitoring is to either build your own container or to run two containers configured for the respective plugins.

[kennymc-c]

Ok, i had this as a parameter before but of course this does not work together with the other --user=root parameter. It is a pity that you are forced to go this workaround for what I think is a very common use case for Telegraf. So many will probably stay with the old version. I hope at least for the release of security updates for these version.

[EcceGratum]

I use the default container on Lubuntu and need to follow a few steps after each container update to make it work again.
It would be nice if it was already included.

I describe the whole process i do after a container update in my last comment on #7743.

To sum it up, "apt get sudo smartmontools" and "echo telegraf ALL=NOPASSWD:/usr/sbin/smartctl >> /etc/sudoers" as root through console in portainer.

[davidnewhall]

fwiw, the correct way to do this is to bake your own image. example: https://github.com/golift/telegraf-docker

This approach is documented on the docker hub repo readme at the bottom under Install Additional Packages. You can use this "different" docker uri into your unRAID or compose template without a problem.

[kennymc-c]

Unfortunately, neither of these are a nice solution. Maintaining your own image is not exactly the best option and this has already been mentioned here and cannot be the final solution to the problem in my opinion.
Especially if you are not a paying customer of Docker Hub, the image cannot update itself automatically through your own Github repo, if I understood it correctly. Why doesn't Telegraf offer such images themselves that already include the tools? At least the most frequently used ones.
I will now test it with the linked image, which already contains all relevant additional tools for me.

[kennymc-c]

Just tried it and wonder how to get ipmitools to run without sudo since it is included. Sudo is not installed in this custom image or included in the base image. The trick with adding the telegraf user to the group that belongs to /dev/ipmi0 doesn't work here either, since it only works with the root account itself or sudo.

[davidnewhall]

Maintaining your own image is not exactly the best option

We can agree to disagree. I prefer to maintain my own images, but I also have a lot of experience doing so and can understand why someone would not want to do that.

I did not use sudo to get ipmi to work. I'm running my container on unraid, and what I did was this. On the docker host (unraid):

chgrp docker /dev/ipmi0
chmod g+wr /dev/ipmi
ls -l /dev/ipmi0
crw-rw---- 1 root docker 245, 0 Dec 25 19:12 /dev/ipmi0

And then on my container config, I added extra parameter of --user telegraf:281 where 281 is the group id of my unraid docker user. I also added a bind mount for /dev/ipmi0:/dev/ipmi0 rw.

IPMI data is now being collected.

The problem I'm having is collecting smartctl data from a mega raid controller. The character device that unraid creates in /dev for the mega raid card is not accessible by any user except root, even after doing the chgrp/chmod shown above. I have some more things to try, but this use case really sucks.

[mikekuzak]

I came across the same you need to be root for smartctl to get data

[davidnewhall]

smartctl works fine for every drive except those connected to the mega raid controller.
EDIT: I take that back. I do not have any drives directly connected that have smart data. I'm able to run smartctl against these non-smart drives in the container as the telegraf user and I get no errors. I also get no real data since the disks don't have it. I assume this would work on a disk with smart capabilities, but I cannot currently test it.

EDIT2: I will figure this out. I have to. :) I may even modify the dockerfile in golift to run the damn thing as root. We'll see..

EDIT3: My quick "make it work" hack is to (ugh) log into the container and run:

addgroup --gid 281 docker
usermod -G docker,disk telegraf

I'll probably wind up putting something like this into the docker file.

[davidnewhall]

I added sudo to the container. This seems to fix all the problems. I also added an auto builder in Docker Cloud and a dependency auto updater in GitHub, so the image should always be up to date. Feel free to open an issue on the repo if you need different features or packages. I built it for me, but happy to share it. Details in repo readme: https://github.com/golift/telegraf-docker

Good luck to all those in this thread!