docker-compose up creates directories on host system

[strk]

Running docker-compose up results in creation of directories specified via volumes directive on the host system. I think it would be better to error out if the host directory does not exist.

[strk]

besides, the directories are created by the "root" user, even if docker-compose up was called by an unprivileged user (other than being allowed to use the docker sockets)

[dnephin]

The directories are created by docker engine, not by compose itself, so this change would need to be made in the engine.

#16072 was the only related issue I could find , but it's for named volumes, not host volumes.

@thaJeztah do you know if there is any talk of this?

[thaJeztah]

Yes, we marked that "feature" as deprecated in 1.9, and it's targeted for removal in 1.11; https://github.com/docker/docker/blob/master/docs/deprecated.md#auto-creating-missing-host-paths-for-bind-mounts

Auto-creating missing host paths for bind mounts

Deprecated in Release: v1.9

Target for Removal in Release: 1.11

When creating a container with a bind-mounted volume-- docker run -v /host/path:/container/path -- docker was automatically creating the /host/path if it didn't already exist.

This auto-creation of the host path is deprecated and docker will error out if the path does not exist.

[dnephin]

Excellent. Closing this issue since it's being fixed by engine.

[dnephin]

Thanks @thaJeztah !

[strk]

Thanks everyone!

[mitar]

Nooo. This was great feature. How can one then make sure paths exist if they do not yet? You have to wrap docker-compose then into a script?

[mitar]

I think this signals well a need for the hooks: #1341

[verglor]

Hi @dnephin and @thaJeztah, this was un-deprecated so the problem still exists.
It was solved by a new --mount flag instead of --volume (see #13121).
Is it possible to use this new flag for mounting volumes in docker-comopse ?
Can we reopen this issue, if it is not possible yet ?

[thaJeztah]

@verglor If I'm not mistaken, the "long form" volumes syntax will use the mount API, thus not create local directories (unless the daemon is too old to support that API); see #5490

[michaelarnauts]

I'm not sure if there is an existing issue for the mount syntax in docker-compose. There are a lot of issues that specify "mount", but most handle issues with the volumes.

[thaJeztah]

The "volumes" key in the compose-file is used both for (named) "volumes" and for bind-mounts (if you specify a host path instead of a volume name, or, in the long-form when specifying type: bind). Using the long form syntax should use the mount API, which prevents paths on the host from being created by the daemon;

    volumes:
      - type: bind
        source: ./some/path/on/host
        target: /some/path/in/container

[speller]

So, how a volume should be defined in docker-compose.yml file to allow a host directory to be automatically created without any additional custom magic scripts?

[thaJeztah]

@speller the shorthand syntax will do that; https://docs.docker.com/compose/compose-file/#short-syntax-3

volumes:
  - ./some/path/on/host:/some/path/in/container

Be aware though, that:

• the path will be created on the host where the daemon runs (which might be a remote host, or a VM), not on the client. (Docker Desktop makes that largely "transparent", but other setups might not have the same)
• if the path on the host points to a file, and that file is missing, that docker will create a directory in that location

Those were the reasons for not auto-creating the paths when using the long-hand syntax, as they've been cause for lots of confusion (and unexpected behavior).

[speller]

@thaJeztah It doesn't in my case. That's why I've found this topic on google. I have a stack deployed with the docker-compose file. I'm using the short syntax. And docker fails to start containers if their volumes' host directories do not exist. The parent directory of the volume directory exists (and have the root:root ownership). But docker can't create a subdir for the volume. If I create the volume directory with sudo (the volume directory has the root:root ownership) container starts successfully and change directory ownership to polkitd:root.

[thaJeztah]

@speller probably better to open a new ticket for that with details (information about the version of compose you're using, as well as engine version (output of docker version and docker info); if possible with minimal steps to reproduce. Are you only seeing that issue when using compose, or also when doing the same with docker run ?

[vaishnoavi]

@thaJeztah : I am also facing the same issue. In my Dockerfile, I am changing the permission of some folder to non-root user.
chown -R non-root-user /usr/local/app_name
When i try to start the container from docker-compose up or docker run, I am mapping the logs folder from the container to the host
-v /path/already/present/this_folder_does_not_exists:/usr/local/app_name/logs.
When i start the container, this_folder_does_not_exists gets created with the root user rather than non-root-user user. Can you provide me some info ?
My docker version : 17.05.0-ce

[volkflo]

@thaJeztah I tried the long volume syntax as you mentioned above, but is there a way to print a warning if the file or directory on the host is missing?

[thaJeztah]

When using the long syntax, it should print an error and fail;

version: "3.7"
services:
  web:
    image: nginx:alpine
    volumes:
      - type: bind
        source: ./some/path/on/host
        target: /some/path/in/container

docker-compose up
Creating network "repro-2781_default" with the default driver
Creating repro-2781_web_1 ... error

ERROR: for repro-2781_web_1  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/repro/repro-2781/some/path/on/host

ERROR: for web  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/repro/repro-2781/some/path/on/host
ERROR: Encountered errors while bringing up the project.

[volkflo]

@thaJeztah Thanks for the example. My problem was that I tried this with an existing setup.
But it is only working after running docker-compose down -v.
I think this is the correct behavior that is only working for ne created mounts?

[rulatir]

Nooo. This was great feature. How can one then make sure paths exist if they do not yet? You have to wrap docker-compose then into a script?

You have to anyway. Without auto-creation docker will error out. With auto-creation it will create the missing directories as root, which is equally undesirable. There is simply no way to avoid having to supply the missing functionality by writing some code.

[mitar]

With auto-creation it will create the missing directories as root, which is equally undesirable.

There could be a config to configure user under which to create the directory.

[dnck]

With auto-creation it will create the missing directories as root, which is equally undesirable.

There could be a config to configure user under which to create the directory.

That would be great! I'm currently having the issue that docker-compose up creates the directories I need (great), but when I want to tear them down and build them back up again using Jenkins, my build fails because Jenkins can't tear down root owned volumes created by Docker.

[rulatir]

There could be a config to configure user under which to create the directory.

There could be? Optimist!

[qiangli]

There could be a config to configure user under which to create the directory.

or by default under: $USER:$USER

[nicolabeghin]

Any news on this? @thaJeztah tried your suggestion but still not failing if folder missing (maybe due to the compose 3.3 version?) Thanks a lot!

version: "3.3"
services:
  web:
    image: nginx:alpine
    volumes:
      - type: bind
        source: ./some/path/on/host
        target: /some/path/in/container

[thaJeztah]

@nicolabeghin I tried reproducing, but I'm getting the error (as expected);

docker-compose up

Creating network "composetest_default" with the default driver
Creating composetest_web_1 ... error

ERROR: for composetest_web_1  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/test/composetest/some/path/on/host

ERROR: for web  Cannot create container for service web: invalid mount config for type "bind": bind source path does not exist: /Users/sebastiaan/Projects/test/composetest/some/path/on/host
ERROR: Encountered errors while bringing up the project.

@nicolabeghin after you ran docker-compose up, was a some/path/on/host directory created in your current directory? Could you also check if a DOCKER_API_VERSION environment variable is set in your environment?

[nicolabeghin]

@thaJeztah thanks a lot for your followup! below the whole thing.

I confirm there's no DOCKER_API_VERSION environment variable set.

I must outline: running from scratch it did led to error as you suggested, but it didn't anymore after trying to change the version from 3.3 to 3.7 in order to check the minimum version required for this. After reverting back to 3.7, as shown above, it didn't fail anymore in any case and with any version.

[kirillt]

Has syntax changed since April?

Compose file:

version: "3.7"

services:
  db:
    image: hello-world
    volumes:
    - type: bind
      source: ./storage
      target: /storage

Running:

[user@localhost test]# docker-compose up
Creating network "test_default" with the default driver
Pulling db (hello-world:)...
latest: Pulling from library/hello-world
0e03bdcc26d7: Downloading [========>                                          ] 0e03bdcc26d7: Pull complete
Digest: sha256:1a523af650137b8accdaed439c17d684df61ee4d74feac151b5b337bd29e7eec
Status: Downloaded newer image for hello-world:latest
Creating test_db_1 ... error

ERROR: for test_db_1  Cannot create container for service db: invalid mount config for type "bind": bind source path does not exist: /tmp/test/storage

ERROR: for db  Cannot create container for service db: invalid mount config for type "bind": bind source path does not exist: /tmp/test/storage
ERROR: Encountered errors while bringing up the project.

[thaJeztah]

@kirillt does the storage directory exist on your system? When using the advanced volumes syntax, the paths you bind-mount from the host must pre-exist

[kirillt]

@thaJeztah I see that in last post (@nicolabeghin) folder notexisting was created.
Apparently, I interpreted it wrong as a desired behaviour and, in fact, it was a demonstration of the bug.

But is there any way of initializing non-existing folders before mounting them?
Except wrapping docker-compose with a Makefile?

[jjkoehorst]

@kirillt I am very curious to this as well. Did you manage to find a solution?

[kirillt]

@jjkoehorst if I remember correctly, no
I decided that Makefile is fine in that case

[Fire-Dragon-DoL]

Would a decent course of action be to document this behavior in the compose syntax spec as well as in the docker command? Due to the difference between macOS where the directory is created as user ID 1000 and linux where the directory is created as root, there are some undesirable issues showing up, where something developed on macOS has a bug that's never discovered until executed on a linux machine or a very rare occurrence where the macOS developer creates a second user (id 1001) and run docker compose up and the directory created would be for the wrong user.

[sparrowt]

Yes it could do with being made clearer at https://docs.docker.com/reference/compose-file/services/#volumes though, now that I know, it is kind of there, buried in under "Long syntax" > bind > create_host_path where it says:

Creates a directory at the source path on host if there is nothing present. Compose does nothing if there is something present at the path. This is automatically implied by short syntax for backward compatibility with docker-compose legacy.

So the short syntax does have the "create host path" behaviour:

servies:
  foo:
    volumes:
      - ./path/on/host:/path/in/container

whereas the long syntax does not:

services:
  foo:
    volumes:
      - type: bind
        source: ./path/on/host
        target: /path/in/container

unless you explicitly add the following:

        ...
        bind:
          create_host_path: true

It's a shame it is not possible (AFAIK) to opt for the "fail if source is missing" behaviour while using the short syntax. It's also quite a subtle difference and can cause quite some confusion especially as docker creates the host directory as root.

Related docs for the CLI https://docs.docker.com/engine/storage/bind-mounts/#syntax

If you use --volume to bind-mount a file or directory that does not yet exist on the Docker host, Docker automatically creates the directory on the host for you. It's always created as a directory.

--mount does not automatically create a directory if the specified mount path does not exist on the host. Instead, it produces an error:

[Fire-Dragon-DoL]

Yes it could do with being made clearer at https://docs.docker.com/reference/compose-file/services/#volumes though, now that I know, it is kind of there, buried in under "Long syntax" > bind > create_host_path where it says:

Creates a directory at the source path on host if there is nothing present. Compose does nothing if there is something present at the path. This is automatically implied by short syntax for backward compatibility with docker-compose legacy.

So the short syntax does have the "create host path" behaviour:

servies:
  foo:
    volumes:
      - ./path/on/host:/path/in/container

whereas the long syntax does not:

services:
  foo:
    volumes:
      - type: bind
        source: ./path/on/host
        target: /path/in/container

unless you explicitly add the following:

        ...
        bind:
          create_host_path: true

(Related docs for the CLI https://docs.docker.com/engine/storage/bind-mounts/#differences-between--v-and---mount-behavior)

This is almost perfect, it should include the permissions/ownership the directory is created with and it could be part of the docs from my perspective!

[katettudelft]

it should include the permissions/ownership the directory is created with

Was there ever a fix, or at least another open issue regarding this? It's a bit of a stumbling block for me right now.