testing kubectl logs

[minrk]

(associated incident report incoming)

With recent versions of GKE, it has become possible for kubectl logs to fail with:

$ kubectl logs -f `pod hub`
No SSH tunnels currently open. Were the targets able to accept an ssh-key for user "gke-df315ed616f010764e03"?

we should perhaps have a test for this and notify when it's not working.

Additionally, it appears that this is caused by a node with wonky network status:

The following command:

kubectl get node -o json | jq '.items[] | [.status.addresses[]]'

reports addresses of nodes, which normally looks like:

[
  {
    "address": "10.x.y.z",
    "type": "InternalIP"
  },
  {
    "address": "35.x.y.z",
    "type": "ExternalIP"
  },
  {
    "address": "gke-prod-a-core-pool-fad3fe98-5t3b",
    "type": "Hostname"
  }
]
[
  {
    "address": "10.x.y.z",
    "type": "InternalIP"
  },
  {
    "address": "35.x.y.z",
    "type": "ExternalIP"
  },
  {
    "address": "gke-prod-a-user-be5bcf07-8j7j",
    "type": "Hostname"
  }
]
[
  {
    "address": "10.x.y.z",
    "type": "InternalIP"
  },
  {
    "address": "35.x.y.z",
    "type": "ExternalIP"
  },
  {
    "address": "gke-prod-a-user-be5bcf07-r3wl",
    "type": "Hostname"
  }
]
[
  {
    "address": "10.x.y.z",
    "type": "InternalIP"
  },
  {
    "address": "35.x.y.z",
    "type": "ExternalIP"
  },
  {
    "address": "gke-prod-a-user-be5bcf07-vhh6",
    "type": "Hostname"
  }
]
[
  {
    "address": "10.x.y.z",
    "type": "InternalIP"
  },
  {
    "address": "35.x.y.z",
    "type": "ExternalIP"
  },
  {
    "address": "gke-prod-a-user-be5bcf07-xhxd",
    "type": "Hostname"
  }
]
[
  {
    "address": "10.x.y.z",
    "type": "InternalIP"
  },
  {
    "address": "35.x.y.z",
    "type": "ExternalIP"
  },
  {
    "address": "gke-prod-a-user-be5bcf07-xl58",
    "type": "Hostname"
  }
]

for a recent bad node, this suspiciously reported no internal or external ip for the bad node. We should add the command to our diagnostics when this issue comes up so that we can isolate and remove the problematic node more easily than the past.