Enable PodTolerationRestriction admission in AKS

[slenky]

I'd like to deploy Kubeflow resources to AKS spot instances but Azure adds spot nodeTaints by default.

I need to either remove that taint from all spot nodes or use PodTolerationRestriction in the kubeflow namespaces.

[ghost]

@Azure/aks-pm issue needs labels

[palma21]

That taint is added to avoid workloads to inadvertently land on those pools that can cause disruptions so we require an explicit toleration to the workloads that can go there.

Adding your ask for PodTolerationRestriction to the backlog

[brudnyhenry]

Hi,
We have the same problem- for some environments we don't want to have automatic taint added.
For now we have to manually remove the taint with kubectl, but whenever new node is provisioned it has the taint by default.
Is there a way to disable it from the spot nodepool ?

[slenky]

Hi,
We have the same problem- for some environments we don't want to have automatic taint added.
For now we have to manually remove the taint with kubectl, but whenever new node is provisioned it has the taint by default.
Is there a way to disable it from the spot nodepool ?

I am currently deploying the DaemonSet with nodeSelector and kubectl inside container which does untaints for us. So when the new spot instance appears it also gets immediately untained.

I'd rather do some checkbox for disabling that :)

[brudnyhenry]

@slenky could you share how you untaint the node ? We are trying the same, but for an AKS cluster with just spot instances (default nodepool scaled to 0, spot nodepool scales to 0 at night and 1 in the morning)
We tried your approach, but the thing is kubectl from "untainter" pod can not connect to kubectl API to do the untainting, because coredns pod can not be scheduled, and coredns pod can not be scheduled because of the taint... chicken and egg problem :)

[slenky]

@slenky could you share how you untaint the node ? We are trying the same, but for an AKS cluster with just spot instances (default nodepool scaled to 0, spot nodepool scales to 0 at night and 1 in the morning)
We tried your approach, but the thing is kubectl from "untainter" pod can not connect to kubectl API to do the untainting, because coredns pod can not be scheduled, and coredns pod can not be scheduled because of the taint... chicken and egg problem :)

We are having a small system nodes always running so no problems with that. Not sure, but will that work if you set the Kubernetes API host to 10.0.0.1 inside that pod?

daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spot-node-untaint
  namespace: kube-system
  labels:
    app: spot-node-untaint
spec:
  selector:
    matchLabels:
      app: spot-node-untaint
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: spot-node-untaint
    spec:
      serviceAccountName: node-untaint
      containers:
      - name: node-untaint
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        image: bitnami/kubectl:latest
        imagePullPolicy: Always
        command: ["/bin/bash"]
        args: ["/scripts/node-untaint-script.sh"]
        volumeMounts:
          - name: node-untaint-script
            mountPath: /scripts/
            readOnly: true
      volumes:
        - name: node-untaint-script
          configMap:
            name: node-untaint-script
      nodeSelector:
        kubernetes.azure.com/scalesetpriority: "spot"
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: node-untaint-script
  namespace: kube-system
data:
  node-untaint-script.sh: |
    #!/bin/bash

    kubectl taint node ${NODE_NAME} kubernetes.azure.com/scalesetpriority-
    tail -f /dev/null

rbac.yaml

kind: ServiceAccount
apiVersion: v1
metadata:
  name: node-untaint
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-untaint
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["*"]
---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-untaint
roleRef:
  kind: ClusterRole
  name: node-untaint
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: node-untaint
    namespace: kube-system

[matthewi]

This would be very useful outside of spot pools. (and/or PodNodeSelector)

[ghost]

Thank you for the feature request. I'm closing this issue as this feature has shipped and it hasn't had activity for 7 days.