Issues with temporary base_directory permissions when running as root in Docker

[Lucas-C]

What action do you want to perform

Simply use pytest-mysql, but in a Docker container

What are the results

Because the user executing py.test is root, the mysqld process does not have the rights to read/write in the temporary directory created.
Specifically, this base_directory is created in factories/process.py. Because py.test tmpdir_factory relies on Python native tempfile.mkdtemp, the following extract from the doc applies:

The directory is readable, writable, and searchable only by the creating user ID.

This explains the errors I get:

E       TimeoutExpired: Executor <mirakuru.tcp.TCPExecutor: "
E                   /usr/bin/mysqld_safe --datadir=/tmp/pytest-mysql-w_4ZZi/mysqldata_31589 --pid-file=/tmp/pytest-mysql-w_4ZZi/mysql-server.31589.pid
E                   --port=31589 --socket=/tmp/pytest-mysql-w_4ZZi/mysql.31589.sock --log-error=/tmp/mysql-server.31589.log
E                   --tmpdir=/tmp/pytest-mysql-w_4ZZi --skip-syslog
E                   "> timed out after 60 seconds
/usr/lib/python2.7/site-packages/mirakuru/base.py:341: TimeoutExpired

And in /tmp/mysql-server.31589.log:

180205 16:24:10 mysqld_safe Starting mysqld daemon with databases from /tmp/pytest-mysql-w_4ZZi/mysqldata_31589
2018-02-05 16:24:10 140169925409024 [Note] /usr/libexec/mysqld (mysqld 10.1.30-MariaDB) starting as process 241 ...
2018-02-05 16:24:10 140169925409024 [ERROR] mysqld: Can't create/write to file '/tmp/pytest-mysql-w_4ZZi/mysqldata_31589/aria_log_control' (Errcode: 13 "Permission denied")
2018-02-05 16:24:10 140169925409024 [ERROR] mysqld: Got error 'Can't create file' when trying to use aria control file '/tmp/pytest-mysql-w_4ZZi/mysqldata_31589/aria_log_control'
2018-02-05 16:24:10 140169925409024 [ERROR] Plugin 'Aria' init function returned error.
2018-02-05 16:24:10 140169925409024 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB

.

2018-02-05 16:24:10 140169925409024 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: The InnoDB memory heap is disabled
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: Compressed tables use zlib 1.2.7
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: Using Linux native AIO
2018-02-05 16:24:10 140169925409024 [Note] InnoDB: Using SSE crc32 instructions
2018-02-05 16:24:10 140169925409024 [ERROR] mysqld: Can't create/write to file '/tmp/pytest-mysql-w_4ZZi/ibqvuAQT' (Errcode: 13 "Permission denied")
2018-02-05 16:24:10 7f7bda9cb900  InnoDB: Error: unable to create temporary file; errno: 13
2018-02-05 16:24:10 140169925409024 [ERROR] Plugin 'InnoDB' init function returned error.
2018-02-05 16:24:10 140169925409024 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-02-05 16:24:10 140169925409024 [Note] Plugin 'FEEDBACK' is disabled.
2018-02-05 16:24:10 140169925409024 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2018-02-05 16:24:10 140169925409024 [ERROR] Unknown/unsupported storage engine: InnoDB
2018-02-05 16:24:10 140169925409024 [ERROR] Aborting

180205 16:24:10 mysqld_safe mysqld from pid file /tmp/pytest-mysql-w_4ZZi/mysql-server.31589.pid ended

What are the expected results

No errors

[Lucas-C]

This does not necessarily requires a fix, but my question is: how could I use pytest-mysql in a Docker container without hitting this issue ?

[Lucas-C]

One last bit of evidence:

$ ll /tmp/pytest-mysql-*
drwx------ 5 root root 4096 Feb  5 16:24 mysqldata_31589

[fizyk]

@Lucas-C could you try master and see if the problem persist? Or it's the place where MariaDB differs from MySQL? Master contains this work: #7 / #56

[Lucas-C]

Thanks a lot for your help !

Using this repo current master branch, I got the following (different) error:

CalledProcessError: Command 'mysqld --initialize-insecure --datadir=/tmp/pytest-of-root/pytest-0/pytest-mysql0/mysqldata_17881 --tmpdir=/tmp/pytest-of-root/pytest-0/pytest-mysql0 --log-error=/tmp/mysql-server.17881.log' returned non-zero exit status 127

Also:

[root@c037fd158857 app]# ll /tmp/pytest-of-root/pytest-0/pytest-mysql0/
total 4
drwxr-xr-x 2 root root 4096 Feb  6 08:23 mysqldata_17881

[Lucas-C]

From your doc, I thought that mysqld_safe would be called instead of myslqd, which is not in the $PATH: https://pypi.python.org/pypi/pytest-mysql

[fizyk]

For that you'd have to follow master's readme. That version is not yet packaged though. And you'd need to pass both mysqld and mysqld_safe.

That's just a wild guess with the master branch, I recall I had similar issues with directory access when adjusting code to MySQL >= 5.7.6

[Lucas-C]

For that you'd have to follow master's readme. That version is not yet packaged though. And you'd need to pass both mysqld and mysqld_safe.

Sorry, I didn't understand what you meant there.
What is the "master's readme" ? This repo README.rst ? What should I follow in it ?
What do you mean by "this version is not yet packaged" ? How is the packaged version on pypi different from the master branch of this repo in terms of default configuration ?
Why and where should I pass "both mysqld and mysqld_safe" ?

[Lucas-C]

I solved my issue with mysqld not found by adding /usr/libexec/ to the $PATH.

Now I have this error, using the current master branch of this repo:

2018-02-06 14:55:54 140533405866240 [Note] mysqld (mysqld 10.1.30-MariaDB) starting as process 238 ...
mysqld: Please consult the Knowledge Base to find out how to run mysqld as root!
2018-02-06 14:55:54 140533405866240 [ERROR] Aborting

[fizyk]

See https://github.com/ClearcodeHQ/pytest-mysql#configuration

master branch will eventually be tagged and released as v2.0.0, and it makes use of mysqld --initialize ( https://github.com/ClearcodeHQ/pytest-mysql/blob/master/src/pytest_mysql/executor.py#L63 ) to initialise mysql for tests, and mysqld_safe to run the server. While the 1.1.1 version uses now deprecated/removed mysql_install_db

[Lucas-C]

The command executed was mysqld --initialize-insecure --datadir=/tmp/pytest-of-root/pytest-0/pytest-mysql0/mysqldata_31562 --tmpdir=/tmp/pytest-of-root/pytest-0/pytest-mysql0 --log-error=/tmp/mysql-server.31562.log
I added --user=root and got the following error:

2018-02-06 15:04:23 140223936502016 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2018-02-06 15:04:23 140223936502016 [ERROR] mysqld: unknown option '--initialize-insecure'
2018-02-06 15:04:23 140223936502016 [ERROR] Aborting

[Lucas-C]

Thanks for you answer. Seems like --initialize-insecure is not supported by my version:

# mysqld --version
mysqld  Ver 10.1.30-MariaDB for Linux on x86_64 (MariaDB Server)

[Lucas-C]

mysql_install_db is present so I guess my issue is with pytest-mysql 1.1.1

[fizyk]

does mysql_install_db allow for switching user? OR can you use docker with other user? Alternatively, does it work if you try to run it locally without docker?

[Lucas-C]

Sorry, I cannot invest more time in trying to make this work right now.
I hoped for an "out of the box" solution for my problem, and wanted to experiment this handy lib.
I don't have a major need for a MySQL pytest fixture, so I'm going to opt for another solution.

Thanks a lot for your help !

For others hitting the same issue, I would recommend to follow @fizyk suggestion to run your tests in Docker under another user.

Feel free to close this issue if you want.

[mmaslowskicc]

pytest-mysql tests succeed in Docker when running as a non-root user; my Dockerfile:

FROM python:3.6-jessie

ENV DEBIAN_FRONTEND=noninteractive
RUN echo exit 101 > /usr/sbin/policy-rc.d && \
    chmod +x /usr/sbin/policy-rc.d && \
    echo mysql-apt-config mysql-apt-config/select-server select mysql-5.7 | debconf-set-selections && \
    wget http://dev.mysql.com/get/mysql-apt-config_0.7.3-1_all.deb && \
    apt-get update -q && \
    apt-get install -q -y lsb-release && \
    dpkg --install mysql-apt-config_0.7.3-1_all.deb && \
    rm mysql-apt-config_0.7.3-1_all.deb && \
    apt-get update -q && \
    apt-get install -q -y --allow-unauthenticated -o Dpkg::Options::=--force-confnew mysql-server

WORKDIR /usr/src/app
COPY requirements-test.txt ./
RUN pip install --no-cache-dir -r requirements-test.txt
COPY . .
RUN pip install .
RUN chown -R nobody .
USER nobody

CMD pytest -x || cat /tmp/mysql-server.*.log

Before Docker, it was very unusual to run such tests as root: I think it's ok to declare running pytest-mysql as root as unsupported and close this issue (there are others for running with MariaDB).

[fizyk]

Okay, I'll close it after we'll land a readme section on how to run it in docker.

[88d52bdba0366127fffca9dfa93895]

Hi @fizyk, I know this thread is old but I want to mention that another way to run pytest-mysql in Docker is very simple, just add user=root in section [mysqld] in file /etc/my.cnf, like this:

echo -e '[mysqld]\nuser=root' > /etc/my.cnf

That's all and we can run our app as root (maybe other processes need root permission). It is easy and useful, so I think it's worth it to mention in README.