[Windows Nodes] "There is no enough space on the disk"

[Stygy]

Is this a request for help?:

YES

Is this an ISSUE or FEATURE REQUEST? (choose one):

ISSUE

What version of acs-engine?:

v0.11.0

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.7.9

What happened:
Cannot pull images because of a "There is not enough space on the disk" error:

K8s shows that there is no disk space issue:

What you expected to happen:
Pod created with no issue

How to reproduce it (as minimally and precisely as possible):

1. Create a Hybrid Cluster with this json: https://goo.gl/9zA9wF
2. Create 6 deployments with microsoft/iis (version 1709) as base image. These will work
3. Create an additional deployment with the same image, the error will start to appear.

Anything else we need to know:
I've upgraded the master machine to D12_V2 but that didn't fix the error. https://goo.gl/g7FjuQ

[andyzhangx]

you need to assign a bigger OS disk size, see example:
https://github.com/Azure/acs-engine/blob/master/examples/windows/kubernetes-manageddisks.json#L20

[Stygy]

I've setup a 200GB disk (https://goo.gl/9zA9wF). What would be a recommended size for the OS disk? Can I resize the disk without rebuilding the cluster?

[andyzhangx]

One possible solution is use the resource disk which has 100GB to download image, need to find a way to change docker settings?

[robinsondotnet]

I've got the same error. Have anyone solved?
I am using a image based on microsoft/dotnet-framework:4.7 . This image is around 12GB. I think that it shouldn't be a problem.

[andyzhangx]

@JiangtianLi do you know anyway to let docker use resource disk in Windows, I only know how to do it in Linux. Resource disk has 100GB size, it would be good place for temp dir.

[Stygy]

I've seen that by default the windows VM's are being created with a 30GB C drive where Docker is. I followed your advice and setup docker to point to another disk with the daemon.json file (adding this line: {"graph": "H:\Images"}). For that I had to Initialize/Partition/Format the attached disks because by default VM only had C and D drives. OSDiskSizeGB is not working as expected.

After that, I tried to start a couple of containers and got constantly the "Failed create pod sandbox" error.

Plus, I see that I'm missing the kubletwin/Pause images after I changed the docker configuration. Is there anyway I can get that? Couldn't pull that from docker hub.

[Stygy]

I should also include that I setup a custom VNET for my cluster, that's the only way I found to get into the windows nodes and check what was happening. And I must mention that I went through a big issue with the Dashboard and DNS deployments because the acs-engine deployment was not setting up the Route Table.

[JiangtianLi]

/cc @PatrickLang on configuring windows docker to use a different disk

[JiangtianLi]

@Stygy How did you set up custom VNET. Did you deploy using https://github.com/Azure/acs-engine/blob/master/docs/custom-vnet.md? Custom VNET in windows cluster doesn't work yet #1767

[Stygy]

Ok, I've solved the disk space issue. I connected to the machine and expanded the C Volume with Diskpart app.
list disk
list volume
select volume 1
extend

Now I only have the Pod sandbox issue. That might be because the VNET setup

[PatrickLang]

Another solution is to move the container storage to D:, but that's not resilient. That shouldn't be a problem because you can just pull the image again if they're missing.

This requires:

1. Change data-root docker daemon config to a path on d:
2. Change DOCKER_TMPDIR to a path on d: (needs a fix that's not in Docker EE 17.06 yet Windows Daemon should respect DOCKER_TMPDIR moby/moby#35077)
3. Re-pull any images you need

[andyzhangx]

I set Windows OSDisk size as 300GB, while after creating some windows containers like aspnet, it has only 3GB disk space left even I have removed all containers. What I could workaround is clean Temp dir to release 13GB space.

rm -Recurse -Force C:\Windows\Temp\*

Is there any way to release the cache data in Windows?