NTFS junctions don't work with Fuse filesharing

[stephen-turner]

We're having a php composer project where we're symlinking local git repositories into the vendor dir of composer, e.g.:
src/vendor/codename/abc -> src/modules/codename/abc

(changed for human readability and to get the concept of mixing static composer deps with local dev projects using autoloading)

The directories are volume-bound with the whole project dir, so the "symlink" is a local NTFS junction automatically created by composer. We tested volume bindings with no option (default is :cached for the major project files), :cached, :delegated and :consistent - while we're still using :delegated and for local dev DBs, profiler data and logs.

While this still works on my Docker CE 2.1.0.5 on Windows, we just found out manually opening the same PHP file via nano in the container renders different results on later versions of Docker on Windows - though, in theory, it is the same file:

• src/modules/codename/abc/backend/class/app.php
• src/vendor/codename/abc/backend/class/app.php

The latter one does not reflect changes performed on the host, while the first one does.
This just came to my mind: the new inotify-capability might not reflect NTFS junctions, local symlinks or something inbetween - which might be even a Windows issue?

Originally posted by @KevinVonJocoon in #5530 (comment)

[stephen-turner]

Thanks for the report @KevinVonJocoon. I've done some experiments. I'm not exactly sure I've got the same symptoms as you, but maybe I haven't completely understood your symptoms.

I created both junctions and symlinks on the host file system and mounted the parent folder into the container:

mkdir mnt
cd mnt
mkdir target
mklink /j link target  [or /d]
docker run -v (path to mnt):/mnt -it ubuntu bash

I updated files within both link and target, in both container and host file systems, in both 2.1.0.5 and 2.2.0.0, and the file sync seemed to work correctly in both directions, and files got updated in both directories at once. With one big exception: the links are not followable within the container in 2.2.0.0, they just appear as broken files.

Does this match what you are seeing, or is there a different problem? If you have something else, can you construct a small step-by-step repro case? Thank you.

[stephen-turner]

I haven't tried cases where only one side of the link is shared, that might have some different cases.

[KevinVonJocoon]

Actually, I just noticed I forgot about our underlying project structure. Sorry for that - reproduction/base situation is as follows:

We have a main git repository containing submodules, so the real structure looks more like:

• /.git/modules/volumes/ --- all the submodules ---
• /volumes/[...]
• /docker-compose.yml

and they're linked (NTFS-junctioned) like this:

/volumes/src/modules/my-submodule-1 -> /.git/modules/volumes/src/modules/my-submodule-1

Which, in addition, is symlinked/junctioned by composer like:

/volumes/src/vendor/my-submodule-1 -> /volumes/src/modules/my-submodule-1

Lets assume /volumes/src is the directory mounted into the container.
So, we're actually talking about a chain of junctions, one even outside the volume bind directory (the .git one):
/volumes/src/vendor/my-submodule-1 -> /volumes/src/modules/my-submodule-1 -> /.git/modules/volumes/src/modules/my-submodule-1

(Please don't be confused by the fact we're also using "modules" as directory name inside our mounts (e.g. src/modules), which is, of course, not the .git/modules directory)
My colleague just helped me reproducing the case on his machine (2.2, even on edge channel) and it took some time we remembered the submodules are also junctions automatically created by git.

I think you'll get the concept, though those links are making me crazy in my head, having a cold and I can't be sure those example links are even plausible. Hope it helps.

[stephen-turner]

OK, and just to be sure, the path on the left of the arrow in your example is the junction point, the one on the right is the target (where the files actually are)?

What does ls -l vendor from inside the container show?

[KevinVonJocoon]

Yes, it's like that.
ls -l shows up normal directories, no links (though they are local NTFS junctions on the windows host - which is ok and worked up to 2.1.0.5 regarding file change propagation).

[stephen-turner]

We have a fix for this, it will be released soon.

[stephen-turner]

We have a new private build that (we believe) fixes this bug. https://download-stage.docker.com/win/stable/42579/Docker%20Desktop%20Installer.exe. Please try it and let us know how you get on.

[KevinVonJocoon]

Sorry to say this, the Build almost broke my colleague's system. File changes were almost randomly propagated. It seems most of the time an initial file change propagated correctly and anything afterwards broke files (seen as empty files in the container). Luckily, the changes did not propagate back to the host.
As we're using productive machines, we cannot evaluate those changes, if they're breaking our systems at the moment. We backed up our volumes and downgraded to 2.1.0.5.

Were you able to construct a reproduction case/test and evaluate it using your private build?

[stephen-turner]

Yes, we've reproduced it and we're pretty confident we have fixed it, we just wanted to give you the chance to check it too.

I think the issue your colleague is seeing is #5530, which is not yet fixed.

[Terrijoo]

I am not sure if it's really the #5530 issue I am having. (I am the colleague he mentioned)
Basically this happens:

• starting up docker-compose
• loading the site:
  • first time results in various 500ishs (some errors our software throws because of not finding a class for example, which exists though and worked on 2.1.0.5 and 2.2.0.0)
  • reloading the site results in some other 500ish requests now (different errors, some requests [JSON files] load now though)
  • a few more reloads then result in a 200ish but with "Failed to load response"

Sadly I can't produce any diagnostics, since I had to downgrade, because I wasn't able to work at all anymore locally (login form wouldn't even load)

The reason why I think that #5530 isn't the issue here, is, that it worked on 2.2.0.0 (except for when you've changed a php file for example), so it seems it's an issue solely related to this 2.2.0.1 build.

[stephen-turner]

Update: This issue is not fully fixed. It works if the target of the junction is also mounted in the container, but not if it points outside.

[KingGoujian]

Any news on this? Estimate when this might be fixed?

[stephen-turner]

Sorry, no date yet. We're still looking into possible solutions. It's not a trivial bug fix, it will need some redesign.

[KingGoujian]

Version 2.2.0.3 release notes say it is fixing this issue:

Fixed an issue where some users were unable to start a container if there is a symlink in a shared volume. Fixes #5582.

Can someone confirm?

[mat007]

@KingGoujian sorry it’s a typo, I believe the release notes should link to #5611 which is about symlinks.
The junctions are a bit trickier and will require more work on our end therefore taking a little more time.

[mat007]

@KingGoujian or actually for junctions it’s partially fixed as @stephen-turner explained a few days ago:

It works if the target of the junction is also mounted in the container, but not if it points outside.

If the junction targets a folder outside the mounts, then it doesn’t work at the moment.

[Borlander5]

I was having the same problem and after upgrading to 2.2.0.3 have confirmed that new containers work, but my existing containers return the same error when I try to start them. Is there a way to fix those containers, or do they need to be rebuilt? A couple contain files that I would rather not lose.

[stephen-turner]

I have discovered a workaround if the target of the junction point is outside the volume mount but inside the same disk. In that case, if you share the junction point itself with -v, instead of the folder containing the junction point, everything will work correctly. Hopefully this will be useful to some people while we continue to work on it.

[djs55]

Sorry for the delay getting back to you. I've got a development build suitable for testing: https://download-stage.docker.com/win/edge/43083/Docker%20Desktop%20Installer.exe It reports as version 2.2.2.0 (43083). It has prototype fixes for

• missing inotify MODIFY event
• missing hidden files
• filesystem events on long paths
• case-insensitivity fix for create
• fix a panic when removing a watch that doesn't exist
• ability to see filesystem events across directory junctions
• represent directory junctions as directories rather than symlinks

If you have a chance to test the build, let me know how it goes! If something goes wrong, please describe what happened in as much detail as you can and please include diagnostics.

[stephen-turner]

Hi everyone, I'm just wondering if anyone who has junction points in their setup has had a chance to try out @djs55's test build. We would love to hear your feedback. Thanks.

[KingGoujian]

Works on my home setup. Created an environment to test it: https://github.com/KingGoujian/docker-volume-issue-with-junctioned-directory

[NullEntity]

I was afflicted by this bug as well. The dev build resolves the junction issue for me.

[KingGoujian]

Am I correct to assume this issue is not resolved in the newest build 2.2.0.4 (43472)? It didn't say anything in the release notes about this.

[stephen-turner]

Correct. We decided the change was too large to include in a patch release. It will be in the next Edge release and the next full Stable release.

[stephen-turner]

This is now released in Edge 2.2.3.0, so I'm going to close this ticket.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked