Extend workflow controller to handle creating pods in namespaces with a resource quota and limit range

[gaigepr]

This PR aims to fix an existing issue with argo: creating workflow pods in namespaces with kubernetes resource quotas.

Here is an example resource quota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: q1
  namespace: sg1
spec:
  hard:
    limits.cpu: 1000m
    limits.memory: 1000Mi

As of now, creating a workflow in a namespace with a resource quota (and no limit range) would give an error like:

time="2018-11-15T23:51:35Z" level=info msg="Failed to create pod pods \"hello-world-l69s7\" is forbidden: failed quota: q1: must specify limits.cpu,limits.memory,requests.cpu,requests.memory" namespace=sg1 workflow=hello-world-l69s7

The message says, "failed quota" because the container(s) argo injects into a workflow do not spec those resources and if the user doesn't add resources to their container specs in the flow. If we specify a limit range like:

apiVersion: v1
kind: LimitRange
metadata:
  name: q1
  namespace: sg1
spec:
  limits:
  - type: Container
    default:
      cpu: 200m
      memory: 200Mi
    defaultRequest:
      cpu: 50m
      memory: 200Mi

we solve a part of the problem. The workflow will now fail when there are not enough resources to create a pod. The error looks like:

time="2018-11-16T00:14:50Z" level=info msg="Failed to create pod hello-world-2sqg5 (hello-world-2sqg5): pods \"hello-world-2sqg5\" is forbidden: exceeded quota: q1, requested: limits.cpu=250m,limits.memory=210Mi, used: limits.cpu=1,limits.memory=840Mi, limited: limits.cpu=1,limits.memory=1000Mi" namespace=sg1 workflow=hello-world-2sqg5

To get (infinite) retries to create a workflow pod, some code needed to be added. With this PR, workflows can be dispatched to a namespace with a resource quota and a limit range. They will eventually succeed.

A downside to the current implementation is there is no rate limiting for trying to create a pod that previously failed because of resource constraints. This may overload the k8s master and generally just cost a lot of cpu.

[gaigepr]

Needs better comments.

[gaigepr]

Perhaps throttle retries? if skipNodeInitialization { sleep ... }

[gaigepr]

Should woc implement a MarkForUpdate method in some other pr?

[gaigepr]

Move the strings.Contains call into a function called ExceededQuota or something?

[gaigepr]

Move the strings.Contains call into a function called ExceededQuota or something?

[gaigepr]

this branch is only reached when a node exists but the pod failed to create because of an exceededQuota error. Should a sleep be added here to avoid slamming the master with Pod create requests?

[gaigepr]

this branch is only reached when a node exists but the pod failed to create because of an exceededQuota error. Should a sleep be added here to avoid slamming the master with Pod create requests?

[gaigepr]

this branch is only reached when a node exists but the pod failed to create because of an exceededQuota error. Should a sleep be added here to avoid slamming the master with Pod create requests?

[alexec]

This PR has had no activity in 1 year. Closing.