figure out why create is broken after https://github.com/kubernetes/kubernetes/pull/68890

[BenTheElder]

/priority critical-urgent
/assign

I've pinned down CI failures to after kubernetes/kubernetes#68890 merged and I can replicate this locally, but I'm not sure why this broke kind just yet. This reliably breaks cluster up.

This changed how some of the static pods are handling DNS, before this they would have used host DNS instead of cluster DNS. I'm not actually sure how this is supposed to bootstrap properly.

FYI @spiffxp :(

cc @MrHohn who I suspect is familiar with DNS policy 😉 and may be able to lend a hand

[BenTheElder]

As far as I can tell this change means if there is a cluster DNS configured on the kubelet, critical static system pods will now use it.

• kubeadm does set a cluster dns
• it's obviously not actually running initially since everything is just starting to boot ..?
• there won't be any fallback servers as far as I can tell

https://github.com/kubernetes/kubernetes/blob/e9fe3f77e9b1810dea6a0918feb1f503ae02fcd7/pkg/kubelet/network/dns/dns.go#L346-L360

Not sure how this is supposed to work correctly...

[BenTheElder]

Specifically the api server talks to itself via localhost rather than an IP, per the logs. My suspicion is that it's not using /etc/hosts and is doing a lookup, which fails since it's pointed at a DNS service that doesn't exist yet...

[BenTheElder]

update:

$ git log
commit 1e4edac01237471910b1b9579cdaed242335178b (HEAD)
Merge: 08ed1631ac b9942796db
Author: k8s-ci-robot <k8s-ci-robot@users.noreply.github.com>
Date:   Wed Sep 26 20:37:47 2018 -0700

    Merge pull request #68886 from mgdevstack/master-emptydir-wrapper
    
    Promote emptyDir wrapper volumes tests to Conformanc

works fine.

moving one commit forward to:

commit e9fe3f77e9b1810dea6a0918feb1f503ae02fcd7 (HEAD -> master, upstream/master, origin/master, origin/HEAD)
Merge: 1e4edac012 8f6ec989e0
Author: k8s-ci-robot <k8s-ci-robot@users.noreply.github.com>
Date:   Wed Sep 26 20:37:57 2018 -0700

    Merge pull request #68890 from andrewrynhard/dnspolicy
    
    kubeadm: Create control plane with ClusterFirstWithHostNet dns policy

breaks kind create.

This seems to confirm that it's related to the DNS policy change.

[BenTheElder]

found it here: kubernetes/kubernetes#69195

work around is here: #45

[BenTheElder]

well we know why, and have a work around. upstreaming improvements now...

[BenTheElder]

Filed kubernetes/kubernetes#69238