Pods marked as Not Ready and with Insufficient Storage after scaling

[koalalorenzo]

After scaling up a AKS cluster (version 1.9.6, created on the 11th of April 2018) we found the following problems:

• The new nodes are marked as "Not Ready", as it seems something related to the CIDR
• The pods living in the existing nodes gets restarted every ~1h
• The existing nodes gets marked as Insufficient Storage.

Our VM size is Standard_D8s_V3, this cluster was initially created with terraform, but the scaling was instructed via az aks scale cmd.

This cluster, after a day, got stuck and all the deployments were not running anymore. We kept the cluster just in case we need to debug more deeply what happened. The nodes now are not reachable, but the master is still working.

[koalalorenzo]

Probably related to #274 and #102

[sauryadas]

@JunSun17 any idea?

[JunSun17]

@koalalorenzo @sauryadas I think this "Node not ready" is a known issue, can you create a ticket so oncall can take a look?

[jnoller]

closing as stale