Updating pod scheduling

[jhamman]

I've been working today on getting the final pieces of the k8s scheduling implemented in our binder deployment. This has resulted in a few merged PRs upstream (jupyterhub/binderhub#853, jupyterhub/binderhub#856). I've deployed these changes manually and I'm opening this PR to ask for some help debugging a few things.

Problem 1: DinD DaemonSet is not scheduling

I'm trying to get the DinD DaemonSet to tolerate the hub.jupyter.org_dedicated=user:NO_SCHEDULE taint that is set on the user-pool (jupyterhub/binderhub#856). However, new DinD pods are not coming up when a new user-pool node comes live. Then build tasks fail because they can't mount the docker socket.

==> v1beta1/DaemonSet
NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR                      AGE
staging-dind           0        0        0      0           0          hub.jupyter.org/node-purpose=user  3m30s
staging-image-cleaner  0        0        0      0           0          hub.jupyter.org/node-purpose=user  3m30s

Asking for help from @yuvipanda @consideRatio @betatim

[betatim]

From the output you pasted I think k8s is happy because there are zero instances desired and zero instances running. This means "all is good". So the question is: why does it think that zero instances are desired.

Do the nodes you want the daemonset to schedule on have a hub.jupyter.org/node-purpose=user label? If not you either need to add it or remove the node selector from the daemonset. At least that is what I'd check next.

[consideRatio]

Hmmm if the daemonset does not desire being scheduled, it seems like they dont tolerate to schedule on any node it considers viable at all. Viable nodes are those that it has nodeSelector and affinity.nodeAffinity.requiredDuring... for

[jhamman]

Unfortunately, I'm not going to have much time to work on this today but here's a summary of the user-pool:

autoscaling:
  enabled: true
  maxNodeCount: 10
config:
  diskSizeGb: 100
  diskType: pd-ssd
  imageType: COS
  labels:
    hub.jupyter.org/node-purpose: user
  machineType: n1-highmem-16
  metadata:
    disable-legacy-endpoints: 'true'
  oauthScopes:
  - https://www.googleapis.com/auth/devstorage.read_only
  - https://www.googleapis.com/auth/logging.write
  - https://www.googleapis.com/auth/monitoring
  - https://www.googleapis.com/auth/service.management.readonly
  - https://www.googleapis.com/auth/servicecontrol
  - https://www.googleapis.com/auth/trace.append
  serviceAccount: default
  taints:
  - effect: NO_SCHEDULE
    key: hub.jupyter.org_dedicated
    value: user
instanceGroupUrls:
- https://www.googleapis.com/compute/v1/projects/pangeo-181919/zones/us-central1-b/instanceGroupManagers/gke-binder-user-pool-45fded5e-grp
management:
  autoRepair: true
name: user-pool
podIpv4CidrSize: 24
selfLink: https://container.googleapis.com/v1/projects/pangeo-181919/zones/us-central1-b/clusters/binder/nodePools/user-pool
status: RUNNING
version: 1.12.7-gke.17

AFAICT, the labels and taints look correct.

Next, is an example build pod spec:

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              component: binderhub-build
          topologyKey: kubernetes.io/hostname
        weight: 100
  ...
  tolerations:
  - effect: NoSchedule
    key: hub.jupyter.org/dedicated
    operator: Equal
    value: user
  - effect: NoSchedule
    key: hub.jupyter.org_dedicated
    operator: Equal
    value: user
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300

Here's the summary of the DinD Daemonset:

$ kubectl describe daemonsets.apps prod-dind --namespace prod 
Name:           prod-dind
Selector:       name=prod-dind
Node-Selector:  hub.jupyter.org/node-purpose=user
Labels:         app=binder
                component=dind
                heritage=Tiller
                name=prod-dind
                release=prod
Annotations:    <none>
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Scheduled with Up-to-date Pods: 0
Number of Nodes Scheduled with Available Pods: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=binder
           component=dind
           heritage=Tiller
           name=prod-dind
           release=prod
  Containers:
   dind:
    Image:      docker:18.09.2-dind
    Port:       <none>
    Host Port:  <none>
    Args:
      dockerd
      --storage-driver=overlay2
      -H unix:///var/run/dind/prod/docker.sock
    Environment:  <none>
    Mounts:
      /var/lib/docker from dockerlib-dind (rw)
      /var/run/dind/prod from rundind (rw)
  Volumes:
   dockerlib-dind:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/dind/prod
    HostPathType:  DirectoryOrCreate
   rundind:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/dind/prod
    HostPathType:  DirectoryOrCreate
Events:            <none>

Finally, I turned DinD off in our production cluster and things are mostly working for now but this isn't a long term solution.

[jhamman]

In looking at the daemonset config, I realized the tolerations are not list, this has me thinking jupyterhub/binderhub#856 may not have been successful. I have an idea how to fix this and will open up a PR shortly.

[jhamman]

jupyterhub/binderhub#857 was the key. All seems well here. Thanks @betatim and @consideRatio for chiming in!