Add docs for perma-failed Deployments

docs/user-guide/deployments.md

@@ -454,6 +454,163 @@ nginx-deployment-3066724191   0         0         1h
 Note: You cannot rollback a paused Deployment until you resume it.
 
 
+## Deployment status
+
+A Deployment enters various states during its lifecycle. It can be [progressing](#progressing-deployment) while rolling out a new ReplicaSet,
+it can be [complete](#complete-deployment), or it can [fail to progress](#failed-deployment).
+
+### Progressing Deployment
+
+Kubernetes marks a Deployment as _progressing_ when one of the following tasks is performed:
+
+* The Deployment is in the process of creating a new ReplicaSet.
+* The Deployment is scaling up an existing ReplicaSet.
+* The Deployment is scaling down an existing ReplicaSet.
+
+You can monitor the progress for a Deployment by using `kubectl rollout status`.
+
+### Complete Deployment
+
+Kubernetes marks a Deployment as _complete_ when it has the following characteristics: 
+
+* The Deployment has minimum availability. Minimum availability means that the Deployment's number of available replicas
+equals or exceeds the number required by the Deployment strategy.
+* All of the replicas associated with the Deployment have been updated to the latest version you've specified, meaning any
+updates you've requested have been completed.
+
+You can check if a Deployment has completed by using `kubectl rollout status`. If the rollout completed successfully, `kubectl rollout status` returns a zero exit code.
+
+```
+$ kubectl rollout status deploy/nginx
+Waiting for rollout to finish: 2 of 3 updated replicas are available...
+deployment "nginx" successfully rolled out
+$ echo $?
+0
+```
+
+### Failed Deployment
+
+Your Deployment may get stuck trying to deploy its newest ReplicaSet without ever completing. This can occur due to some of the following factors:
+
+* Insufficient quota
+* Readiness probe failures
+* Image pull errors
+* Insufficient permissions
+* Limit ranges
+* Application runtime misconfiguration
+
+One way you can detect this condition is to specify specify a deadline parameter in your Deployment spec: ([`spec.progressDeadlineSeconds`](#progress-deadline-seconds)). `spec.progressDeadlineSeconds` denotes the number of seconds the Deployment controller waits before indicating (via the Deployment status) that the Deployment progress has stalled.
+
+The following `kubectl` command sets the spec with `progressDeadlineSeconds` to make the controller report lack of progress for a Deployment after 10 minutes:
+
+```shell
+$ kubectl patch deployment/nginx-deployment -p '{"spec":{"progressDeadlineSeconds":600}}'
+"nginx-deployment" patched
+```
+Once the deadline has been exceeded, the Deployment controller adds a DeploymentCondition with the following attributes to
+the Deployment's `status.conditions`:
+
+* Type=Progressing
+* Status=False
+* Reason=ProgressDeadlineExceeded
+
+See the [Kubernetes API conventions](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties) for more information on status conditions.
+
+Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with
+`Reason=ProgressDeadlineExceeded`.
+
+**Note:** If you pause a Deployment, Kubernetes does not check progress against your specified deadline. You can safely pause a Deployment in the middle of a rollout and resume without triggering a the condition for exceeding the deadline.
+
+You may experience transient errors with your Deployments, either due to a low timeout that you have set or due to any other kind
+of error that can be treated as transient. For example, let's suppose you have insufficient quota. If you describe the Deployment
+you will notice the following section:
+
+```
+$ kubectl describe deployment nginx-deployment
+<...>
+Conditions:
+  Type            Status  Reason
+  ----            ------  ------
+  Available       True    MinimumReplicasAvailable
+  Progressing     True    ReplicaSetUpdated
+  ReplicaFailure  True    FailedCreate
+<...>
+```
+
+If you run `kubectl get deployment nginx-deployment -o yaml`, the Deployement status might look like this:
+
+```
+status:
+  availableReplicas: 2
+  conditions:
+  - lastTransitionTime: 2016-10-04T12:25:39Z
+    lastUpdateTime: 2016-10-04T12:25:39Z
+    message: Replica set "nginx-deployment-4262182780" is progressing.
+    reason: ReplicaSetUpdated
+    status: "True"
+    type: Progressing
+  - lastTransitionTime: 2016-10-04T12:25:42Z
+    lastUpdateTime: 2016-10-04T12:25:42Z
+    message: Deployment has minimum availability.
+    reason: MinimumReplicasAvailable
+    status: "True"
+    type: Available
+  - lastTransitionTime: 2016-10-04T12:25:39Z
+    lastUpdateTime: 2016-10-04T12:25:39Z
+    message: 'Error creating: pods "nginx-deployment-4262182780-" is forbidden: exceeded quota:
+      object-counts, requested: pods=1, used: pods=3, limited: pods=2'
+    reason: FailedCreate
+    status: "True"
+    type: ReplicaFailure
+  observedGeneration: 3
+  replicas: 2
+  unavailableReplicas: 2
+```
+
+Eventually, once the Deployment progress deadline is exceeded, Kubernetes updates the status and the reason for the Progressing condition:
+
+```
+Conditions:
+  Type            Status  Reason
+  ----            ------  ------
+  Available       True    MinimumReplicasAvailable
+  Progressing     False   ProgressDeadlineExceeded
+  ReplicaFailure  True    FailedCreate
+```
+
+You can address an issue of insufficient quota by scaling down your Deployment, by scaling down other controllers you may be running,
+or by increasing quota in your namespace. If you satisfy the quota conditions and the Deployment controller then completes the Deployment
+rollout, you'll see the Deployment's status update with a successful condition (`Status=True` and `Reason=NewReplicaSetAvailable`).
+
+```
+Conditions:
+  Type          Status  Reason
+  ----          ------  ------
+  Available     True    MinimumReplicasAvailable
+  Progressing   True    NewReplicaSetAvailable
+```
+
+`Type=Available` with `Status=True` means that your Deployment has minimum availability. Minimum availability is dictated
+by the parameters specified in the deployment strategy. `Type=Progressing` with `Status=True` means that your Deployment
+is either in the middle of a rollout and it is progressing or that it has successfully completed its progress and the minimum
+required new replicas are available (see the Reason of the condition for the particulars - in our case
+`Reason=NewReplicaSetAvailable` means that the Deployment is complete).
+
+You can check if a Deployment has failed to progress by using `kubectl rollout status`. `kubectl rollout status` returns a non-zero exit code if the Deployment has exceeded the progression deadline.
+
+```
+$ kubectl rollout status deploy/nginx
+Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
+error: deployment "nginx" exceeded its progress deadline
+$ echo $?
+1
+```
+
+### Operating on a failed deployment
+
+All actions that apply to a complete Deployment also apply to a failed Deployment. You can scale it up/down, roll back
+to a previous revision, or even pause it if you need to apply multiple tweaks in the Deployment pod template.
+
 ## Use Cases 
 
 ### Canary Deployment
@@ -556,6 +713,17 @@ the rolling update starts, such that the total number of old and new Pods do not
 the new Replica Set can be scaled up further, ensuring that the total number of Pods running
 at any time during the update is at most 130% of desired Pods.
 
+### Progress Deadline Seconds
+
+`.spec.progressDeadlineSeconds` is an optional field that specifies the number of seconds you want
+to wait for your Deployment to progress before the system reports back that the Deployment has
+[failed progressing](#failed-deployment) - surfaced as a condition with `Type=Progressing`, `Status=False`.
+and `Reason=ProgressDeadlineExceeded` in the status of the resource. The deployment controller will keep
+retrying the Deployment. In the future, once automatic rollback will be implemented, the deployment
+controller will roll back a Deployment as soon as it observes such a condition.
+
+If specified, this field needs to be greater than `.spec.minReadySeconds`.
+
 ### Min Ready Seconds
 
 `.spec.minReadySeconds` is an optional field that specifies the