Modify code to trigger an alert, sending 500s if too many ToDos

README.md

@@ -55,3 +55,17 @@ API in `todo-app`.
 We will add metrics of each of the [four types](https://prometheus.io/docs/concepts/metric_types/).
 
 See the [`todo-app` README](./todo-app/README.md) for more details.
+
+### S3
+
+In this branch, we modify the code in `todo-app` to trigger an already configured alert.
+Our Prometheus instance has already been [configured to alert](./prometheus/alert_rules.yml) if the
+percentage of 500 responses exceeds 10% over a 5 minute period.
+If this alert is triggered, it will send the alert to Alertmanager, which in turn will send a
+message to Slack using the configured webhook.
+
+For details of how the alert is triggered from the API, see the
+[`todo-app` README](./todo-app/README.md) for more details.
+
+To simulate API usage, you can use the included [`api-client`](./api-client/README.md) which will
+allow the error case to be produced more quickly.

todo-app/README.md

@@ -163,3 +163,31 @@ The metrics will most likely be available at the end of response so you may need
 
 You can also view these metrics in your Prometheus UI within a few minutes.
 On the graph page, open the dropdown of metrics and look for the metric names added to the code (`todo_items_created`, etc.).
+
+## S3: Triggering an alert
+
+We can modify the code to produce more errors which will trigger an alert from Prometheus.
+To produce more errors, this branch introduces a change which causes the API to respond
+with `500` if the user attempts to create a ToDo item when there are 10 or more in the list.
+
+The endpoint to create the ToDo has been modified to count the number of ToDos that already exist.
+If there less than 10, proceed to create the new item. If there are 10 or more, respond with 500.
+All other endpoints will continue to work as expected, existing items can be viewed or removed.
+
+To deploy this code, [build the latest image](#building-the-images).
+The Helm chart contains changes in this branch to use this latest image.
+To update your `helm` deployment use the following command:
+
+```
+helm upgrade demo chart
+```
+
+You can check the status of the deployment using the [previous instructions](#deploying-the-application).
+
+Once the new version has been deployed, you can interact with the API directly, by manually attempting to create
+more than 10 ToDo items, or use the API client.
+You can view the status of the alert in the [Prometheus UI](http://192.168.100.100:9090/alerts).
+Once the error condition is true, the alert will go into `Pending` state for one minute before the Alertmanager
+is notified.
+Once the Alertmanager is notified, you can view the alert in the [Alertmanager UI](http://192.168.100.100:9093/#/alerts).
+Alertmanager will then send the alert to the configured receiver, in this case the Slack webhook.

todo-app/app/api/routes.js

@@ -94,22 +94,35 @@ module.exports = (app) => {
 
     // create todo and send back all todos after creation
     app.post('/api/todos', (req, res) => {
-        // create a todo, information comes from AJAX request from Angular
-        Todo.create({
-            text: req.body.text,
-            complete: false
-        }, (err, todo) => {
+        Todo.count({}, (err, count) => {
             if (err) {
                 total_errors.inc();
                 res.send(err);
                 return;
             }
 
-            total_todos.inc();
-            incomplete_todos.inc();
-            // get and return all the todos after you create another
-            getTodos(res);
+            if (count < 10) {
+                // create a todo, information comes from AJAX request from Angular
+                Todo.create({
+                    text: req.body.text,
+                    complete: false
+                }, (err, todo) => {
+                    if (err) {
+                        total_errors.inc();
+                        res.send(err);
+                        return;
+                    }
+
+                    total_todos.inc();
+                    incomplete_todos.inc();
+                    // get and return all the todos after you create another
+                    getTodos(res);
+                });
+            } else {
+                res.status(500).send('Too many ToDo items');
+            }
         });
+
     });
 
     // update a todo with complete status

todo-app/chart/values.yaml

@@ -4,7 +4,7 @@
 replicaCount: 1
 image:
   repository: todo-demo
-  tag: s2
+  tag: s3
   pullPolicy: IfNotPresent
 service:
   name: todo-app