Add docs for perma-failed Deployments #1699

0xmichalis · 2016-11-16T15:41:30Z

cc: @janetkuo @kubernetes/docs

This change is

janetkuo · 2016-11-16T19:10:30Z

docs/user-guide/deployments.md

+soon as a failure condition is noticed.
+
+Another thing to note is transient errors eg. your rollout times out and a `ProgressDeadlineExceeded` condition is set to
+your Deployment because don't have enough quota. All you need is to scale down your Deployment, or scale down other Pods


You can't scale down pods. You can scale down other controllers or delete pods.

because don't have enough quota --> because of insufficient quota

janetkuo · 2016-11-16T19:14:01Z

docs/user-guide/deployments.md

+application runtime misconfiguration. In those cases you can specify a timeout parameter in the spec
+(`spec.progressDeadlineSeconds`) that denotes the seconds you want to wait for your Deployment to report any progress.
+Once the deadline is exceeded, meaning your Deployment will not report any progress for more than progressDeadlineSeconds,
+the deployment controller will add a Progressing condition with Status=False and Reason=ProgressDeadlineExceeded in the


Suggest quoting things like "Status=False" with `

janetkuo · 2016-11-16T19:14:52Z

docs/user-guide/deployments.md

+the status of the condition into a succeeded one (Status=True, Reason=NewReplicaSetAvailable).
+
+A complete Deployment for the controller is one that: 
+* has minimum availability, meaning its available replicas are more than those required by the Deployment strategy (`status.


The bullet point isn't rendered well. I guess adding a new line will fix it

Remove the whitespace after status.

janetkuo · 2016-11-16T19:17:39Z

docs/user-guide/deployments.md

+* has minimum availability, meaning its available replicas are more than those required by the Deployment strategy (`status.
+availableReplicas >= spec.replicas - maxUnavailable` where `maxUnavailable` is the maximum unavailable replicas that a
+Deployment allows at any point in time. It depends on the deployment strategy: Recreate deployments don't use it whereas
+Rolling deployments specify it in `spec.strategy.rollingUpdate.maxUnavailable`).


Rolling Update deployments

Not sure I like the distinction (why not RecreateUpdate also then instead of Recreate or Rolling instead of RollingUpdate?) but I wenr ahead and updated it.

janetkuo · 2016-11-16T19:21:19Z

docs/user-guide/deployments.md

+availableReplicas >= spec.replicas - maxUnavailable` where `maxUnavailable` is the maximum unavailable replicas that a
+Deployment allows at any point in time. It depends on the deployment strategy: Recreate deployments don't use it whereas
+Rolling deployments specify it in `spec.strategy.rollingUpdate.maxUnavailable`).
+* has all of its replicas updated to the latest revision (`spec.replicas == status.updatedReplicas`).


Can we make it more clear on what a complete Deployment means: for rolling update deployments, both points should be true; for recreate ones, only the second one needs to be true (the first one doesn't apply)

The first point definitely applies for recreate deployments - in order to have a complete deployment you have to have all pods available, right?

janetkuo · 2016-11-16T19:22:52Z

docs/user-guide/deployments.md

+in the status of the Deployment.
+
+Progress for a Deployment is:
+* the creation of the new replica set.


same here for the bullet point render issue

janetkuo · 2016-11-16T19:24:14Z

docs/user-guide/deployments.md

+```
+
+If you describe the Deployment, you will notice a Conditions section:
+```


This part isn't rendered well either. Maybe a newline before ``` would fix it

janetkuo · 2016-11-16T19:25:16Z

docs/user-guide/deployments.md

+
+This is how the status of the Deployment looks like if you would run `kubectl get deployment nginx-deployment -o yaml`
+(spec of the object is omitted for brevity):
+```


render issue again

janetkuo · 2016-11-16T19:26:22Z

docs/user-guide/deployments.md

+There will be times your Deployment will get stuck trying to deploy its newest ReplicaSet without ever completing,
+due to insufficient quota, readiness probe failures, image pull error, insufficient permissions, limit ranges, or any
+application runtime misconfiguration. In those cases you can specify a timeout parameter in the spec
+(`spec.progressDeadlineSeconds`) that denotes the seconds you want to wait for your Deployment to report any progress.


link to the section below for spec.progressDeadlineSeconds

janetkuo · 2016-11-16T19:27:11Z

docs/user-guide/deployments.md

@@ -556,6 +675,15 @@ the rolling update starts, such that the total number of old and new Pods do not
 the new Replica Set can be scaled up further, ensuring that the total number of Pods running
 at any time during the update is at most 130% of desired Pods.

+### Progress Deadline Seconds
+
+`.spec.progressDeadlineSeconds` is an optional field that specifies the number of seconds you want


Suggest linking back to the "failed deployment" section somewhere

janetkuo · 2016-11-16T19:29:23Z

We can add sub-sections in "failed deployment"

jaredbhatti · 2016-11-22T01:16:12Z

docs/user-guide/deployments.md

+`Reason=ProgressDeadlineExceeded` in the status of the Deployment. For more information on status conditions, see
+[here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties).
+
+Note that at the moment, Kubernetes will do nothing more than just report with a condition with `Reason=ProgressDeadlineExceeded`.


We do not talk about "future features" in our technical docs. It makes tracking changes really hard. Please change this paragraph, or at least remove the sentence starting with "In the future..."

jaredbhatti · 2016-11-22T01:17:30Z

docs/user-guide/deployments.md

@@ -454,6 +454,133 @@ nginx-deployment-3066724191   0         0         1h
 Note: You cannot rollback a paused Deployment until you resume it.


+## Failed Deployment
+
+There will be times your Deployment will get stuck trying to deploy its newest ReplicaSet without ever completing,


Reword to:

"Your Deployment may get stuck trying to deploy its newest ReplicaSet without ever completing, due to:

Insufficient quota

readiness probe failures

image pull error

insufficient permissions

limit ranges

application runtime misconfigurations

jaredbhatti · 2016-11-22T01:21:49Z

docs/user-guide/deployments.md

+is set to your Deployment because of insufficient quota. All you need is to scale down your Deployment, or scale down other
+controllers you may be running, or increase quota in your namespace, either by asking your administrator or by doing it yourself
+assuming you have the right permissions. Once the deployment controller notices that your Deployment is complete, it will
+transition the status of the Progressing condition into a succeeded one (`Status=True` and `Reason=NewReplicaSetAvailable`).


The sentence "All you need..." is very difficult to read. Suggest breaking it into two: "To fix this issue, scale down your Deployment, scale down other controllers you may be running, or increase quota in your namespace. You can do this by either by asking your administrator or doing it yourself assuming you have the right permissions."

jaredbhatti · 2016-11-22T01:24:52Z

Please fix the open docs issues.

0xmichalis · 2016-11-28T10:23:36Z

@jaredbhatti thanks for the review, addressed all of your comments.

devin-donnelly · 2016-12-01T00:41:49Z

docs/user-guide/deployments.md

+Your Deployment may get stuck trying to deploy its newest ReplicaSet without ever completing, due to:
+
+* insufficient quota
+


Please remove the whitespace from in between each bullet, and capitalize the first letter of each.

devin-donnelly · 2016-12-01T00:42:39Z

docs/user-guide/deployments.md

+
+* application runtime misconfiguration
+
+In those cases you can specify a timeout parameter in the spec ([spec.progressDeadlineSeconds](#progress-deadline-seconds))


Comma after "cases"

devin-donnelly · 2016-12-01T00:43:50Z

docs/user-guide/deployments.md

+* application runtime misconfiguration
+
+In those cases you can specify a timeout parameter in the spec ([spec.progressDeadlineSeconds](#progress-deadline-seconds))
+that denotes the seconds you want to wait for your Deployment to report any progress. Once the deadline is exceeded, meaning


"the seconds you want to wait" -> "some number of seconds to wait for your Deployment to report and progress."

Also, put in a paragraph break after "progress"; you need to insert the text from line 509 afterward.

devin-donnelly · 2016-12-01T00:46:28Z

docs/user-guide/deployments.md

+
+In those cases you can specify a timeout parameter in the spec ([spec.progressDeadlineSeconds](#progress-deadline-seconds))
+that denotes the seconds you want to wait for your Deployment to report any progress. Once the deadline is exceeded, meaning
+your Deployment will not report any progress for more than progressDeadlineSeconds, the deployment controller will add a


I'm not sure what this sentence is trying to say. Did you get stuck between drafts/edits? I suggest this rewording:

"Once the deadline has been exceeded, the Deployment controller adds the following attributes to the Deployment's status:

Type=Progressing

Status=False

Reason=ProgressDeadlineExceeded

See the Kubernetes API conventions for more information on status conditions.

devin-donnelly · 2016-12-01T00:47:03Z

docs/user-guide/deployments.md

+your Deployment will not report any progress for more than progressDeadlineSeconds, the deployment controller will add a
+condition with `Type=Progressing`, `Status=False`, and `Reason=ProgressDeadlineExceeded` in the status of the Deployment.
+For more information on status conditions, see [here](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties). 
+Note that at the moment, Kubernetes will do nothing more than just report with a condition with `Reason=ProgressDeadlineExceeded`.


"Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with Reason=ProgressDeadlineExceeded."

devin-donnelly · 2016-12-01T01:31:48Z

docs/user-guide/deployments.md

+
+* scaling down old pods.
+
+To make the controller report lack of progress for a Deployment after 10 minutes:


These lines from 509 to 513 are the most important part of the entire section. They need to be moved up to the top of the section, immediately after the word "progress" on line 474.

devin-donnelly · 2016-12-01T01:33:43Z

docs/user-guide/deployments.md

+"nginx-deployment" patched
+```
+
+If you describe the Deployment, you will notice a Conditions section:


This section is about how to detect and deal with a stalled Deployment. It isn't necessary to explain what a successful deployment looks like first. Remove everything from between lines 516 and 530.

devin-donnelly · 2016-12-01T01:34:36Z

docs/user-guide/deployments.md

+required new replicas are available (see the Reason of the condition for the particulars - in our case
+`Reason=NewReplicaSetAvailable` means that the deployment is complete).
+
+For example, if you have insufficient pod quota and your Deployment tries to create more pods than allowed then


Move everything from line 532 to line 541 up, above line 477.

devin-donnelly · 2016-12-01T01:36:55Z

docs/user-guide/deployments.md

+Note that at the moment, Kubernetes will do nothing more than just report with a condition with `Reason=ProgressDeadlineExceeded`.
+
+Another thing to note is transient errors eg. your rollout times out and a condition with `Reason=ProgressDeadlineExceeded`
+is set to your Deployment because of insufficient quota. To fix this issue, scale down your Deployment, or scale down other


Paragraph break after "quota." Insert the following:

"You can check to see if your Deployment has encountered quota problems by inspecting the Deployment with kubectl get Deployment."

Then move everything from line 544 to 572 underneath that sentence.

devin-donnelly · 2016-12-01T01:41:11Z

docs/user-guide/deployments.md

+  unavailableReplicas: 2
+```
+
+Eventually, once the Deployment progress deadline is exceeded, the status and reason of the Progressing condition


Everything from line 575 to line 591 needs to be moved up to line 475.

0xmichalis · 2016-12-02T14:25:29Z

@devin-donnelly thanks for the review, I think I have addressed most if not all of your comments, ptal

devin-donnelly · 2016-12-02T18:50:12Z

docs/user-guide/deployments.md

+required new replicas are available (see the Reason of the condition for the particulars - in our case
+`Reason=NewReplicaSetAvailable` means that the deployment is complete).
+
+A complete Deployment is one that: 


I still don't think this section belongs here -- Everything between lines 567-585. Why are we describing a complete deployment, and deployment progress, in a section on failed deployments? I think this is either described elsewhere, or should be.

All you need is the last paragraph (line 587 to line 591) under a subheading called "Operating on a failed deployment".

It seems that this section is the only one that mentions complete deployments. My reasoning for having it here was that you need to describe both concepts together in order to make them understood (what's a complete deployment, what's a failed deployment). That being said I am going to create a separate section for complete deployments prior to failed deployments.

devin-donnelly · 2016-12-02T18:50:39Z

@Kargakis, One more structural change. It's much better now, thanks.

devin-donnelly · 2016-12-02T23:07:01Z

@janetkuo , are you doing the tech review on this one?

devin-donnelly · 2016-12-05T19:22:48Z

docs/user-guide/deployments.md

+
+### Progressing Deployment
+
+Progress for a Deployment is:


"If a Deployment is progressing, that means it is performing one of the following tasks:"

devin-donnelly · 2016-12-05T19:22:59Z

docs/user-guide/deployments.md

+
+Progress for a Deployment is:
+
+* the creation of the new replica set.


"Creating a new replica set"

Should use ReplicaSet instead of replica set, based on the latest style guide http://kubernetes.io/docs/contribute/style-guide/#use-camel-case-for-api-objects

devin-donnelly · 2016-12-05T19:23:08Z

docs/user-guide/deployments.md

+Progress for a Deployment is:
+
+* the creation of the new replica set.
+* scaling up new pods.


"Scaling up new pods"

Only ReplicaSets can be scaled up/down

devin-donnelly · 2016-12-05T19:23:18Z

docs/user-guide/deployments.md

+
+* the creation of the new replica set.
+* scaling up new pods.
+* scaling down old pods.


"Scaling down old pods"

devin-donnelly · 2016-12-05T19:23:39Z

docs/user-guide/deployments.md

+
+### Complete Deployment
+
+A complete Deployment is one that: 


"If a Deployment is complete, it has the following characteristics:"

devin-donnelly · 2016-12-05T19:27:10Z

docs/user-guide/deployments.md

+
+A complete Deployment is one that: 
+
+* has minimum availability, meaning its available replicas are more than those required by the Deployment strategy 


"* The Deployment has minimum availability. Minimum availability means that the Deployment's number of available replicas exceeds the number required by the Deployment strategy."

The rest is way too long for a bullet point, and I think it's not really necessary.

"equals or exceeds" (>=, not >)

devin-donnelly · 2016-12-05T19:29:35Z

docs/user-guide/deployments.md

+(`status.availableReplicas >= spec.replicas - maxUnavailable` where `maxUnavailable` is the maximum unavailable replicas
+that a Deployment allows at any point in time. It depends on the deployment strategy: Recreate deployments don't use it whereas
+Rolling Update deployments specify it in [spec.strategy.rollingUpdate.maxUnavailable](#max-unavailable)).
+* has all of its replicas updated to the latest revision (`spec.replicas == status.updatedReplicas`).


"* All of the replicas associated with the Deployment have been updated to the latest version you've specified, meaning any updates you've requested have been completed."

devin-donnelly · 2016-12-05T20:10:38Z

docs/user-guide/deployments.md

+Rolling Update deployments specify it in [spec.strategy.rollingUpdate.maxUnavailable](#max-unavailable)).
+* has all of its replicas updated to the latest revision (`spec.replicas == status.updatedReplicas`).
+
+Note that clients should always take into account generation/observedGeneration for ensuring that the latest state of a


I'm not sure what exactly this paragraph is trying to say. It needs some major revisions.

Who are "clients" in this case? Users? If so, address the user directly as "you."

What is generation/observedGeneration? Who provides the observedGeneration? Looks like a status field, so I guess Kubernetes provides that; what is generation?

By "latest state of a Deployment has been observed by the controller", do you mean "What the user sees in kubectl actually reflects reality?" I'm having trouble parsing what this sentence means.

Who are "clients" in this case? Users? If so, address the user directly as "you."

Client is anything that talks to the api server; a user that uses {kubectl, external client, directly via the API} or a controller (either internal, or external set up by an admin).

What is generation/observedGeneration? Who provides the observedGeneration? Looks like a status field, so I guess Kubernetes provides that; what is generation?

generation can be found in the metadata of an object, and some objects also implement observedGeneration in the status. Generation is updated every time the spec of the object is updated (ie. user changes something) and observedGeneration is updated by the respective controller of the object (controller observes the change that was done by the user).

By "latest state of a Deployment has been observed by the controller", do you mean "What the user sees in kubectl actually reflects reality?" I'm having trouble parsing what this sentence means.

Once a client (again, either user or a controller) observes that metadata.generation == status.observedGeneration, then they can be sure that the current spec of the object has been observed by its controller).

I removed this section in favor of a TODO

devin-donnelly · 2016-12-05T20:11:11Z

I like the new section organization; I have some revisions/comments to the text in the new sections.

devin-donnelly · 2016-12-07T00:31:50Z

Thanks, @Kargakis . I'm just going to do some minor cleanup and get it assigned for tech review, and we'll be ok to merge.

janetkuo · 2016-12-07T01:02:12Z

docs/user-guide/deployments.md

+* Limit ranges
+* Application runtime misconfiguration
+
+In those cases, you can specify a timeout parameter in the spec ([spec.progressDeadlineSeconds](#progress-deadline-seconds))


use `` to quote spec.progressDeadlineSeconds

This sounds like when a Deployment failed, you need to specify a timeout for it. But we actually want the users to specify it beforehand.

Rephrase it so that it's clear that users can use this timeout parameter to specify how the system should determine whether a Deployment is considered "failed to progress".

Done in my own commit.

janetkuo · 2016-12-07T01:24:55Z

docs/user-guide/deployments.md

+equals or exceeds the number required by the Deployment strategy.
+* All of the replicas associated with the Deployment have been updated to the latest version you've specified, meaning any
+updates you've requested have been completed.
+


Add a line here for how to easily determine a Deployment is completed (such as from the output of kubectl rollout status)

I have already added such a sentense in the previous section.

Added an additional line here that also points out the exit code of rollout status in case the Deployment completes.

Added grammar fix.

janetkuo · 2016-12-07T01:31:36Z

docs/user-guide/deployments.md

+In those cases, you can specify a timeout parameter in the spec ([spec.progressDeadlineSeconds](#progress-deadline-seconds))
+that denotes the number of seconds to wait for your Deployment to report any progress.
+
+The following `kubectl` command sets the spec with `progressDeadlineSeconds` to make the controller report lack of progress for a Deployment after 10 minutes:


link to progressDeadlineSeconds below

We link to there just above?

Link at first mention is fine; doesn't need to be linked at every mention.

janetkuo · 2016-12-07T01:33:25Z

docs/user-guide/deployments.md

+```
+
+Once the deadline has been exceeded, the Deployment controller adds a DeploymentCondition with the following attributes to
+the Deployment's status:


Should we make it clear that it's under .status.conditions?

Added code formatting.

janetkuo · 2016-12-07T01:35:50Z

docs/user-guide/deployments.md

+
+See the [Kubernetes API conventions](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties) for more information on status conditions.
+
+Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with


What does "take no action" mean? Please clarify it.

It will do nothing more than update the status.condition with type=Progressing to Status=False, Reason=ProgressDeadlineExceeded.

Isn't "other than ..." clear enough?

I think so.

janetkuo · 2016-12-07T01:37:52Z

docs/user-guide/deployments.md

+
+### Operating on a failed deployment
+
+All actions that apply to a complete Deployment also apply to a failed Deployment. You can scale it up/down, rollback


janetkuo · 2016-12-07T01:40:57Z

docs/user-guide/deployments.md

+
+All actions that apply to a complete Deployment also apply to a failed Deployment. You can scale it up/down, rollback
+to a previous revision, or even pause it if you need to apply multiple tweaks in the Deployment pod template. Note
+that progress for a Deployment is not estimated while the Deployment is paused so you can safely pause a Deployment


"progress for a Deployment is not estimated" would you clarify this?

The controller wont check if the Deployment failed.

This actually belongs way higher in the document; it's kind of arbitrary here. Fixed in my own commit.

janetkuo · 2016-12-07T01:42:33Z

docs/user-guide/deployments.md

+[failed progressing](#failed-deployment) - surfaced as a condition with `Type=Progressing`, `Status=False`.
+and `Reason=ProgressDeadlineExceeded` in the status of the resource. The deployment controller will keep
+retrying the Deployment. In the future, once automatic rollback will be implemented, the deployment
+controller will rollback a Deployment as soon as it observes such a condition. If specified, this field


janetkuo · 2016-12-07T01:43:53Z

docs/user-guide/deployments.md

+and `Reason=ProgressDeadlineExceeded` in the status of the resource. The deployment controller will keep
+retrying the Deployment. In the future, once automatic rollback will be implemented, the deployment
+controller will rollback a Deployment as soon as it observes such a condition. If specified, this field
+needs to be greater than `.spec.minReadySeconds`.


Is this line related to the previous line about auto rollback? If not, separate them (move auto rollback to the end); if so, make it clear that they're relevant

No, it's not.

devin-donnelly · 2016-12-07T23:33:30Z

Hi @Kargakis , would you mind having a look at Janet's feedback from a technical perspective? We're right up against our deadline for 1.5, and we need to get this merged ASAP. Thanks.

0xmichalis · 2016-12-08T11:04:48Z

Hi @devin-donnelly @janetkuo I believe I have addressed most if not all comments, ptal

0xmichalis · 2016-12-09T18:46:24Z

@janetkuo when will these changes land in the master branch?

janetkuo · 2016-12-09T21:48:24Z

@Kargakis yes we'll merge the change in 1.5 branch into master after 1.5 launch

0xmichalis assigned janetkuo Nov 16, 2016

googlebot added the cla: yes label Nov 16, 2016

0xmichalis mentioned this pull request Nov 16, 2016

Add docs for perma-failed deployments #1337

Closed

janetkuo added this to the 1.5 milestone Nov 16, 2016

janetkuo reviewed Nov 16, 2016

View reviewed changes

janetkuo added Needs Docs Review labels Nov 16, 2016

jaredbhatti reviewed Nov 22, 2016

View reviewed changes

jaredbhatti added Docs Review: Open Issues and removed Needs Docs Review labels Nov 22, 2016

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 28, 2016

0xmichalis mentioned this pull request Nov 28, 2016

Common patterns in workload controllers kubernetes/kubernetes#37563

Closed

devin-donnelly suggested changes Dec 1, 2016

View reviewed changes

devin-donnelly suggested changes Dec 2, 2016

View reviewed changes

devin-donnelly approved these changes Dec 5, 2016

View reviewed changes

devin-donnelly suggested changes Dec 5, 2016

View reviewed changes

devin-donnelly added Docs LGTM and removed Docs Review: Open Issues labels Dec 7, 2016

janetkuo reviewed Dec 7, 2016

View reviewed changes

devin-donnelly added the Tech Review: Open Issues label Dec 7, 2016

devin-donnelly removed the Needs Tech Review label Dec 7, 2016

0xmichalis and others added 3 commits December 8, 2016 11:29

Add docs for perma-failed Deployments

a86732a

Update deployments.md

7427bc7

More updates

5ec4e65

devin-donnelly approved these changes Dec 8, 2016

View reviewed changes

Update deployments.md

5186d55

janetkuo approved these changes Dec 8, 2016

View reviewed changes

janetkuo added Tech Review LGTM and removed Tech Review: Open Issues labels Dec 8, 2016

devin-donnelly merged commit fe46050 into kubernetes:release-1.5 Dec 8, 2016

0xmichalis deleted the progressdeadlineseconds branch December 8, 2016 22:37

This was referenced Dec 9, 2016

Link sections that talk about deployment status #1912

Merged

Allow deployments to correctly indicate they are failing to deploy kubernetes/enhancements#122

Closed

		Your Deployment may get stuck trying to deploy its newest ReplicaSet without ever completing, due to:

		* insufficient quota


		* application runtime misconfiguration

		In those cases you can specify a timeout parameter in the spec ([spec.progressDeadlineSeconds](#progress-deadline-seconds))


		* scaling down old pods.

		To make the controller report lack of progress for a Deployment after 10 minutes:


		Progress for a Deployment is:

		* the creation of the new replica set.


		A complete Deployment is one that:

		* has minimum availability, meaning its available replicas are more than those required by the Deployment strategy


		See the [Kubernetes API conventions](https://github.com/kubernetes/kubernetes/blob/{{page.githubbranch}}/docs/devel/api-conventions.md#typical-status-properties) for more information on status conditions.

		Note that in version 1.5, Kubernetes will take no action on a stalled Deployment other than to report a status condition with


		### Operating on a failed deployment

		All actions that apply to a complete Deployment also apply to a failed Deployment. You can scale it up/down, rollback

Add docs for perma-failed Deployments #1699

Add docs for perma-failed Deployments #1699

Conversation

0xmichalis commented Nov 16, 2016 • edited by thockin Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janetkuo commented Nov 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaredbhatti commented Nov 22, 2016

0xmichalis commented Nov 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xmichalis commented Dec 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devin-donnelly commented Dec 2, 2016 • edited Loading

devin-donnelly commented Dec 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xmichalis Dec 6, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devin-donnelly commented Dec 5, 2016

devin-donnelly commented Dec 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xmichalis commented Nov 16, 2016 •

edited by thockin

Loading

devin-donnelly commented Dec 2, 2016 •

edited

Loading

0xmichalis Dec 6, 2016 •

edited

Loading