WCOW: Build context with nested directories fails with "unclean path" #4741

snorrk · 2024-03-07T03:51:51Z

Using nested directories in build context fails with error: failed to solve: unclean path test2/hello.txt: invalid argument.

Repro

Set-Content Dockerfile @"
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator
COPY . C:/
"@

md test1\test2
"hello" >> test1\test2\hello.txt

buildctl build `
--frontend=dockerfile.v0 `
--local context=test1 `
--local dockerfile=. `
--output=type=image,name=hello-buildkit

[+] Building 0.3s (5/6)
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 128B                                                                                                                                                                   0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                                                                                                         0.1s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => ERROR [internal] load build context                                                                                                                                                                0.1s
 => => transferring context: 59B                                                                                                                                                                       0.0s
 => CACHED [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7                                                            0.1s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7                                                                   0.1s
------
 > [internal] load build context:
------
error: failed to solve: unclean path test2/hello.txt: invalid argument

Version

buildctl --version
buildctl github.com/moby/buildkit v0.13.0-rc2 596ef8f

Windows 11 23H2

The text was updated successfully, but these errors were encountered:

profnandaa · 2024-03-07T04:20:36Z

Thanks for opening the issue! I suspect this must be related to #4696 -- can you try "hello" >> test1/test2/hello.txt instead and lmk?

profnandaa · 2024-03-07T06:57:25Z

Cross posting here from our side-channel discussions. Now, sorry I had mistaken your initial post, the commands there:

md test1\test2
"hello" >> test1\test2\hello.txt

are not inside the dockerfile.

I can't repro the issue, my other suspicion is the containerd version.

profnandaa · 2024-03-07T07:39:25Z

I can repro the issue now on WS2022:

PS C:\Users\Administrator\repro-4741> buildctl build `
>> --frontend=dockerfile.v0 `
>> --local context=test1 `
>> --local dockerfile=. `
>> --output=type=image,name=hello-buildkit
[+] Building 0.5s (5/6)
 => [internal] load build definition from Dockerfile                                                                                    0.1s
 => => transferring dockerfile: 128B                                                                                                    0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                                          0.2s
 => [internal] load .dockerignore                                                                                                       0.0s
 => => transferring context: 2B                                                                                                         0.0s
 => ERROR [internal] load build context                                                                                                 0.0s
 => => transferring context: 59B                                                                                                        0.0s
 => CANCELED [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c7192  0.0s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7    0.0s
------
 > [internal] load build context:
------
error: failed to solve: unclean path test2/hello.txt: invalid argument

Will take a look.

doctorpangloss · 2024-03-16T02:08:55Z

Is there a situation where this doesn't occur but nested directories work?

I'm here because dockerd building is also full of bugs

profnandaa · 2024-03-20T18:41:47Z

I'm still investigating this but I suspect this is around path.Clean() nuances on Windows - https://github.com/tonistiigi/fsutil/blob/7525a1af2bb545e89dc9bced785bff7a3b7f08c2/validator.go#L31

crazy-max · 2024-03-20T19:39:42Z

I'm still investigating this but I suspect this is around path.Clean() nuances on Windows - https://github.com/tonistiigi/fsutil/blob/7525a1af2bb545e89dc9bced785bff7a3b7f08c2/validator.go#L31

Looks similar to #4723 (comment)

profnandaa · 2024-03-27T08:22:44Z

@crazy-max -- that's true. So, turns out filepath.Clean() will turn / to \\ for Windows, that's causing the first failure.
When I fix that, I end up with another path issue downstream:

error: failed to solve: invalid file request test2\hello.txt

ref: vendor/github.com/tonistiigi/fsutil/receive.go:240:

~~I suspect this is because of the path not having /~~.

Yes, turns out, the path is being stored with / in the map, hence can't be found.

(dlv) p p
"test2\\hello.txt"
(dlv) p r.files
map[string]uint32 [
        "test2/hello.txt": 1,
]
(dlv) bt
0  0x00000000011b1676 in github.com/tonistiigi/fsutil.(*receiver).asyncDataFunc
   at C:/dev/container-core/buildkit/vendor/github.com/tonistiigi/fsutil/receive.go:240
1  0x00000000011bcd4c in github.com/tonistiigi/fsutil.(*receiver).asyncDataFunc-fm
   at <autogenerated>:1
2  0x00000000011a277d in github.com/tonistiigi/fsutil.(*DiskWriter).processChange
   at C:/dev/container-core/buildkit/vendor/github.com/tonistiigi/fsutil/diskwriter.go:255
3  0x00000000011a2485 in github.com/tonistiigi/fsutil.(*DiskWriter).requestAsyncFileData.func1
   at C:/dev/container-core/buildkit/vendor/github.com/tonistiigi/fsutil/diskwriter.go:231
4  0x0000000000b84269 in golang.org/x/sync/errgroup.(*Group).Go.func1
   at C:/dev/container-core/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:75
5  0x00000000003525a1 in runtime.goexit
   at C:/Program Files/Go/src/runtime/asm_amd64.s:1695

I'm still on it.

/cc. @gabriel-samfira

profnandaa · 2024-03-27T08:46:10Z

Now fixed it in a hacky way, cleaning it up and will do the PRs first to fsutil (and later vendor it); and then here.

 buildctl build `
>>  --output type=image,name=docker.nandaa.dev/test,push=false `
>>  --progress plain "-frontend=dockerfile.v0" `
>>  --local context=C:/dev/play/dockerfiles/repro-4741/test1 `
>>  --local dockerfile=C:\dev\play\dockerfiles\repro-4741
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 29B
#1 transferring dockerfile: 124B 19.9s
#1 transferring dockerfile: 128B 19.9s done
#1 DONE 19.9s

#2 [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022
#2 DONE 2.6s

#3 [internal] load .dockerignore
#3 transferring context:
#3 transferring context: 2B done
#3 DONE 0.4s

#4 [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720
#4 resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720 0.1s done
#4 DONE 0.2s

#5 [internal] load build context
#5 ...

#4 [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720
#4 ...

#5 [internal] load build context
#5 transferring context: 87B 18.0s done
#5 DONE 18.3s

#4 [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 0B / 116.94MB 0.2s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 6.29MB / 116.94MB 3.9s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 12.58MB / 116.94MB 7.8s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 18.87MB / 116.94MB 11.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 25.17MB / 116.94MB 12.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 31.46MB / 116.94MB 16.5s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 37.75MB / 116.94MB 20.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 44.04MB / 116.94MB 24.0s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 50.33MB / 116.94MB 25.2s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 56.62MB / 116.94MB 29.0s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 62.91MB / 116.94MB 32.9s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 69.21MB / 116.94MB 36.5s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 75.50MB / 116.94MB 37.5s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 81.79MB / 116.94MB 41.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 88.08MB / 116.94MB 44.9s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 94.37MB / 116.94MB 48.8s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 100.66MB / 116.94MB 49.8s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 106.95MB / 116.94MB 53.6s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 113.25MB / 116.94MB 57.0s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 116.94MB / 116.94MB 57.5s done
#4 extracting sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee
#4 extracting sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 24.1s done
#4 DONE 81.8s

#6 [2/2] COPY . C:/
#6 DONE 0.8s

#7 exporting to image
#7 exporting layers
#7 exporting layers 0.9s done
#7 exporting manifest sha256:d1b63c3def742a9f1adaceea08cc5ccf3fb25106d82462f3f1ee08b2b7640b93 0.1s done
#7 exporting config sha256:3c1f55a4d88478ad39dd9239a2f4cb7e513587b11d59acb2731057420e01f768 0.1s done
#7 naming to docker.nandaa.dev/test
#7 naming to docker.nandaa.dev/test 0.0s done
#7 DONE 1.1s

Will also add some more path tests cases in our integration tests suite.

profnandaa · 2024-04-04T06:48:34Z

Fix in tonistiigi/fsutil#196 ; will be vendored here once merged.

doctorpangloss · 2024-04-11T19:59:24Z

what does the schedule look like for this getting released? it's kind of a showstopper for windows, so you're not seeing other bugs related to buildkit windows until this gets released.

Full diff: https://github.com/tonistiigi/fsutil/compare/7525a1af2bb5..497d33b Summary changes: - tonistiigi/fsutil#195 receive: ensure callback errors are propagated - tonistiigi/fsutil#196 Fix file transfers from windows to linux fixes moby#4741 - tonistiigi/fsutil#197 recv: translate linkname to wire format Signed-off-by: Anthony Nandaa <profnandaa@gmail.com>

Full diff: https://github.com/tonistiigi/fsutil/compare/7525a1af2bb5..497d33b Summary changes: - tonistiigi/fsutil#195 receive: ensure callback errors are propagated - tonistiigi/fsutil#196 Fix file transfers from windows to linux fixes moby#4741 - tonistiigi/fsutil#197 recv: translate linkname to wire format Signed-off-by: Anthony Nandaa <profnandaa@gmail.com> (cherry picked from commit 64ea9da)

MaksimAmpl · 2024-05-06T07:50:32Z

Hi @tonistiigi
Can you confirm if this issue has been resolved in the latest reslease?

snorrk · 2024-05-06T08:18:08Z

Hi @tonistiigi
Can you confirm if this issue has been resolved in the latest reslease?

I was able to build a Windows container from a build context with nested directories using the latest release - it looks like the issue has been resolved.

lippertmarkus · 2024-05-06T15:56:18Z

I also verified that it works now

profnandaa self-assigned this Mar 9, 2024

profnandaa added the area/windows label Mar 9, 2024

tonistiigi added this to the v0.13.1 milestone Mar 11, 2024

profnandaa added the area/windows-wcow label Mar 26, 2024

profnandaa mentioned this issue Mar 28, 2024

RFC: fix for #4741 #4806

Closed

profnandaa mentioned this issue Apr 4, 2024

Fix file transfers from windows to linux tonistiigi/fsutil#196

Merged

tonistiigi modified the milestones: v0.13.1, v0.13.2 Apr 19, 2024

profnandaa mentioned this issue Apr 22, 2024

vendor: github.com/tonistiigi/fsutil @ 497d33b #4863

Merged

tonistiigi closed this as completed in #4863 Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WCOW: Build context with nested directories fails with "unclean path" #4741

WCOW: Build context with nested directories fails with "unclean path" #4741

snorrk commented Mar 7, 2024

profnandaa commented Mar 7, 2024

profnandaa commented Mar 7, 2024

profnandaa commented Mar 7, 2024 •

edited

Loading

doctorpangloss commented Mar 16, 2024 •

edited

Loading

profnandaa commented Mar 20, 2024

crazy-max commented Mar 20, 2024

profnandaa commented Mar 27, 2024 •

edited

Loading

profnandaa commented Mar 27, 2024

profnandaa commented Apr 4, 2024

doctorpangloss commented Apr 11, 2024 •

edited

Loading

MaksimAmpl commented May 6, 2024

snorrk commented May 6, 2024

lippertmarkus commented May 6, 2024

WCOW: Build context with nested directories fails with "unclean path" #4741

WCOW: Build context with nested directories fails with "unclean path" #4741

Comments

snorrk commented Mar 7, 2024

Repro

Version

profnandaa commented Mar 7, 2024

profnandaa commented Mar 7, 2024

profnandaa commented Mar 7, 2024 • edited Loading

doctorpangloss commented Mar 16, 2024 • edited Loading

profnandaa commented Mar 20, 2024

crazy-max commented Mar 20, 2024

profnandaa commented Mar 27, 2024 • edited Loading

profnandaa commented Mar 27, 2024

profnandaa commented Apr 4, 2024

doctorpangloss commented Apr 11, 2024 • edited Loading

MaksimAmpl commented May 6, 2024

snorrk commented May 6, 2024

lippertmarkus commented May 6, 2024

profnandaa commented Mar 7, 2024 •

edited

Loading

doctorpangloss commented Mar 16, 2024 •

edited

Loading

profnandaa commented Mar 27, 2024 •

edited

Loading

doctorpangloss commented Apr 11, 2024 •

edited

Loading