Severe Docker 1.12.1 performance regression with DB2 images (~10x slower)

[malduarte]

Expected behavior

Performance equal or better than 1.12.0

Actual behavior

Docker 1.12.1 is much slower on pretty much all DB2 activity. Not sure if it is DB2 only, but I'm only experiencing it on DB2. For example empty database creation takes ~24 seconds on 1.12.0 but on 1.12.1 it takes 10 times more, around ~253 seconds

docker stats seems to indicate lower cpu usage and lower I/O activity on 1.12.1

Information

• Di276C02C2-7AEC-4A1B-9F2C-9E6DAD3241CB
• Create a Dockerfile with the following content

FROM ibmcom/db2express-c:latest

ENV DB2INST1_PASSWORD=password
ENV LICENSE=accept
USER root
RUN /bin/bash -c 'su - db2inst1 -c "db2start && db2 create database test"'

Steps to reproduce the behavior

1. Install 1.12.0
2. Pull official db2 image docker pull ibmcom/db2express-c
3. Make sure you're running 1.12.0 docker --version should output Docker version 1.12.0, build 8eab29e
4. Create an image with the docker file above with the time command time docker build -t test . Consider running docker stats while it is running
5. Record the output
6. Remove the recently created image docker rmi <imageid>
7. Install 1.12.1
8. Make sure you're running 1.12.1 docker --version should output Docker version 1.12.1, build 6f9534c
9. Create an image with the attached docker file with the time command time docker build -t test .. Consider running docker stats while it is running
10. Compare the times.

12:05 $ docker --version
Docker version 1.12.0, build 8eab29e
✔ ~/docker_bench
12:05 $ time docker build -t test .
Sending build context to Docker daemon 66.56 kB
Step 1 : FROM ibmcom/db2express-c:latest
 ---> 7aa154d9b73c
Step 2 : ENV DB2INST1_PASSWORD password
 ---> Using cache
 ---> 8abd69a10768
Step 3 : ENV LICENSE accept
 ---> Using cache
 ---> 271672f5dc75
Step 4 : USER root
 ---> Using cache
 ---> 3f3ab0f72686
Step 5 : RUN /bin/bash -c 'su - db2inst1 -c "db2start && db2 create database test"'
 ---> Running in bd1c14701b68
libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory
SQL1063N  DB2START processing was successful.
DB20000I  The CREATE DATABASE command completed successfully.
 ---> 6bb79e3c1592
Removing intermediate container bd1c14701b68
Successfully built 6bb79e3c1592

real    0m23.646s
user    0m0.011s
sys 0m0.014s


12:10 $ docker --version
Docker version 1.12.1, build 6f9534c
✔ ~/docker_bench
12:11 $ time docker build -t test .
Sending build context to Docker daemon 66.56 kB
Step 1 : FROM ibmcom/db2express-c:latest
 ---> 7aa154d9b73c
Step 2 : ENV DB2INST1_PASSWORD password
 ---> Using cache
 ---> 8abd69a10768
Step 3 : ENV LICENSE accept
 ---> Using cache
 ---> 271672f5dc75
Step 4 : USER root
 ---> Using cache
 ---> 3f3ab0f72686
Step 5 : RUN /bin/bash -c 'su - db2inst1 -c "db2start && db2 create database test"'
 ---> Running in ebbf3069fb53
libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory
SQL1063N  DB2START processing was successful.
DB20000I  The CREATE DATABASE command completed successfully.
 ---> 032da4506157
Removing intermediate container ebbf3069fb53
Successfully built 032da4506157

real    4m23.451s
user    0m0.010s
sys 0m0.015s

[dave-tucker]

ping @dsheets

[justincormack]

I am going to test if this is an issue just on Linux, as there is not much visible Docker for Mac specific stuff here (eg no volumes or network)

[malduarte]

@justincormack I didn't try Linux, but I did try with docker 1.12.1 on docker-machine (virtualbox driver). Works fine there for this particular test, virtually the same time as docker 1.12.0 for mac

[justincormack]

@malduarte yes it may be Docker for Mac specific, but it is not obvious why, it could be for example related to the Linux runtime or something...

[justincormack]

Sounds like @djs55 has an idea of what may have caused this

[justincormack]

So it appears this may be because in previous versions it was not fully flushing blocks on fsync (and nor was Docker toolbox), so when we fixed this to add the full flush through to the OSX filesystem it slowed down. This would probably only be visible in database workloads where fsync is being used a lot.

[JanBednarik]

I have similar issue with PostgreSQL. It's approximately 10x slower after Docker for Mac upgrade to 1.12.1

[malduarte]

@justincormack : That makes sense. I understand current sync behaviour should be the default (event with the slow performance). Correctness should trump fast.

That being said Is there some kind of configuration option to disable fsync? It would be a boom for ephemeral container use cases. Example: container is spun up for a CI build and then thrown away.

[djs55]

Thanks very much for the easy to follow repro instructions. I ran 2 experiments:

• unmodified 1.12.1: time reports 5 minutes
• remove the call to fsync: time reports 30 seconds

It seems the VM calls the virtio-blk implementation of flush in hyperkit approximately 25000 times. Each of these is implemented currently by an fsync(F_FULLFSYNC) to avoid writes being partially written or re-ordered over a power loss (as recommended by the Apple docs). Unfortunately each fsync(F_FULLFSYNC) seems to take about 10ms which accounts for the slowdown.

I agree that there are important use-cases where data persistence is less important than throughput, especially on developer setups or CI builds where containers are ephemeral. I'll investigate the possibility of a configuration option.

[malduarte]

ooops, closed accidentally. Sorry folks

[timc13]

+1

[djs55]

I've added a configuration option to the master branch and it should be released in the next beta (27) later this week. It's not wired up to the UI yet. I'll post instructions explaining how to change the setting when the beta is ready.

[c0bra]

@kingbuzzman I'm am running 9.5.5. I tried using 9.5.5-alpine with the same results.

UPDATE: Appears to be due to volume mapping. This takes a few seconds:

docker run -it --rm -p 5434:5432 postgres:9.5.5-alpine postgres --version

This takes 1 min 30s:

docker run -it --rm -v ~/postgres_data:/var/lib/postgresql/data -p 5434:5432 postgres:9.5.5-alpine postgres --version

[kingbuzzman]

@c0bra how big is ~/postgres_data ? I have a couple of node apps that has 4gb+ inside them and i map them in just fine (2-5s tops).. there has to be something with your config.

[jack0]

if you don't stop the postgresql container correctly, the next time you start the container postgresql will do a validation check on all your data. use docker run without --rm and do a docker stop container_name before you do docker rm container_name. this ensures postgresql is stopped correctly so the next time you start the container it won't need validate your data

[c0bra]

@kingbuzzman 644MB.

@jack0 I'm using docker compose. I tried using the :delegated volume flag from here: https://docs.docker.com/docker-for-mac/osxfs-caching/ but no luck.

NOTE: It doesn't seem to matter whether I use docker-compose up/down or stop/start it still takes a while for the connections to work. I actually get no output at all in the log.

[EugenMayer]

In case i missed it, do you guys host-mount the /var/lib/postfgres ( or whereever your postgres data folder is ) or do you use a named-volume / anon volume for it?

what i see is this
-v ~/postgres_data:/var/lib/postgresql/data
And that is one of the worse ideas you can have. Delegated will not help you, rather cached if at all. Do not host-mount anything which is ever frequently read from or written on when you are using docker-for-mac. It is slow, it is utterly slow - and that is a known one.

If you insist on doing that, use http://docker-sync.io (i am biased) and you get native speed back - but straight volume mounts will slow down enormously

[c0bra]

@EugenMayer I guess I'm doing the bad one then.

I did try to get docker-sync working for a bit today; I can to finish that up and see if it improves. Thanks for the suggestion.

[EugenMayer]

@c0bra docker-sync will bring it back to what you get without volume mounts, simple as that ( its just native speed ) - we did quiet some benchmarking here: https://github.com/EugenMayer/docker-sync/wiki/4.-Performance

[c0bra]

@EugenMayer yep that did it. Blazing fast now!

Thank you so much. If I ever come across the pond I'll buy you a beer :)

[alinturbut]

Is this still actual? When i go to ~/Library/Containers/com.docker.docker/Data/database/disk there are just 2 files: path and size. My Docker is still very slow and I have sync=os on ps uax. Running about 15 containers with very slow startup time and no volumes.

Later edit: apparently, my docker was setup to only allow 2 GB of memory for containers, which I hadn't noticed from the beginning. Setting it to 8 turned everything back to Linux normal (from Docker GUI -> Preferences -> Advanced). So... don't forget to allocate enough RAM for the containers to be able to run :)

Oh, and btw, I didn't had to run the solutions presented here, it works from the box.
Local docker version Docker version 17.06.1-ce, build 874a737

[robertoandrade]

if the container doesn't have mission critical data in it, but still require high throughput I/O, is it safe to set this sync option to none?

[agis]

I'm on 17.09.0-ce-mac33 (19543) and ~/Library/Containers/com.docker.docker/Data/database. On the other hand, ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ does exist and has various files:

~/Library/Containers/com.docker.docker/Data $ ls com.docker.driver.amd64-linux/
Docker.qcow2    console-ring.0  console-ring.2  console-ring.4  lock            mac.0           pid             tty
console-ring    console-ring.1  console-ring.3  hypervisor.pid  log/            nic1.uuid       syslog

So maybe in recent versions the right path to create the disk folder in, along with those files is
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/ instead of
~/Library/Containers/com.docker.docker/Data/database/com.docker.driver.amd64-linux/?

cc @booi

[leepowelldev]

Not sure if I'm suffering from this problem - very new to docker. But I am composing a simple apache/php container with a mysql container volume mapped to a local directory for data storage. When I run my application through the containers I get intermittent connection problems to the MySQL container. Could this be down to the I/O speeds?

[EugenMayer]

@leepowellcouk no this is something else, see the purpose of scripts like https://github.com/vishnubob/wait-for-it - in short ( to not drag offroad here ): your app starts faster then your database, thus the db is "not yet there", so people use those scripts to "wait for the db" before starting the actual app app-container entrypoint

[leepowelldev]

@EugenMayer - Thanks, I'm not sure that's it as it happens across browser refreshes. I'll take a look elsewhere and see if I can work out whats going on.

[barat]

@leepowellcouk your problem is

mysql container volume mapped to a local directory for data storage

Volume bind-mount in mac/windows is very slow:
#77
#1592

For persistence just use docker named volumes. Don't mount it to local folesystem. Such volumes are almost persistent (you can delete them by docker system prune). If you need local copy as well just periodically use "docker cp" :)

Problem starts, when you want your codebase to be locally mounted (obvious usecase). Then if it's something like Symfony php app (lot of writes on cache generation) it will be sluggish. That's why tools like docker-sync exists until docker team will solve this more then one year old issue with bind mounting. But this is offtopic ;)

[leepowelldev]

@barat - Thank you, I don't want to derail this issue. I have tried to use docker named volumes and get the same problem.

[vampolo]

Tried @djs55 way and it works for me as well!

Docker version 17.09.0-ce, build afdb6d4

Wonder if this setting can just be the default one for Mac.

[djs55]

@vampolo thanks for the update.

For reference the current edge version (17.11.0-ce-mac40 (20561)) of Docker for Mac (and the future stable) will use raw format by default which is quite quick

DB20000I  The CREATE DATABASE command completed successfully.
Removing intermediate container 6cf22ef3647d
 ---> d0eb220a122b
Successfully built d0eb220a122b
Successfully tagged test:latest

real	0m25.518s
user	0m0.056s
sys	0m0.045s

Systems using the older qcow format should use the faster mode by default too.

Since this is fixed in edge, I'll close this issue. If there are future problems feel free to open fresh issues!

Thanks for your patience and for using Docker for Mac.

[mingp]

@djs55

That is great news to hear. As always, thanks to the Docker team for continuing to improve the product offering.

Are there any additional steps that Docker for Mac users need to take, other than upgrading to the latest version, in order to get this new performance improvement, such as setting a configuration value? How can we verify that our setup is correct and that we are getting the new and improved version? Thanks in advance.

[for0231]

My docker version:

doe:com.docker.driver.amd64-linux doe$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:13:02 2018
OS/Arch: darwin/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:22:38 2018
OS/Arch: linux/amd64
Experimental: true

There is not exist disk directory, and anything.
My path is /Users/doe/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux,
I see previous tips, the disk directory is git version, how can I solve this?

[markddavidoff]

@for0231 try ~/Library/Containers/com.docker.docker/Data/database/com.docker.driver.amd64-linux/disk :)

[thehappycoder]

I have the same question as @for0231
~/Library/Containers/com.docker.docker/Data/database/ doesn't exist
~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux exists
But there is not .git folder anywhere.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked