fix: update labs for one prometheus (#79)

docs/get-started/labs/custom-metrics.md

@@ -5,18 +5,14 @@ sidebar_label: Using custom metrics
 ---
 
 :::info
-For this lab Prometheus and Grafana need to be enabled for the Team.
+For this lab Grafana and Alertmanager need to be enabled for the Team.
 :::
 
-## What are custom metrics
-
-Prometheus will collect all kind of standard container metrics like CPU and memory usage, but no (custom) business or customer metrics like the number of customers who logged into your app each hour.
-
-To be able to collect custom metrics you will need to expose this data in your code. This is called instrumentation of code and can include annotating the code with metadata, or adding in logic to calculate and expose data.
+To be able to collect custom metrics you will first need to configure your app to expose metrics. This is called instrumentation of code and can include annotating the code with metadata, or adding in logic to calculate and expose data.
 
 Instrumenting code means you write code to expose information about the technical, business, and customer context. This information can then be collected and analyzed using Prometheus and Grafana.
 
-In this lab we will use a container that exposes custom metrics and then show how the metrics can be collected and analysed.
+In this lab we'll use a container that exposes custom metrics and then show how the metrics can be collected and analysed.
 
 ## Create a Workload
 
@@ -42,7 +38,7 @@ servicePorts:
     targetPort: 8080
     protocol: TCP
     name: web
-replicaCount: 2
+replicaCount: 1
 serviceMonitor:
   create: true
   endpoints:
@@ -52,52 +48,126 @@ serviceMonitor:
       path: /q/metrics
 ```
 
-5. Click `Submit` and then `Deploy Changes`
+5. Click `Submit` and then `Deploy Changes`.
 
-## Check the status of the ServiceMonitor
+Our metrics will now be scraped by the Platform Prometheus. Before we continue, let's first generate some load:
 
-Check if the ServiveMonitor has been picked up by Prometheus:
+1. Expose the `custom-metrics` service (see lab [Expose services](expose-services.md)).
 
-1. In the left menu go to `Apps`.
+2. Run the following command in your terminal:
 
-2. Click on the `Prometheus` app.
+```bash
+for i in {1..1000}; do curl https://custom-metrics-labs.<your-domain>/hello; sleep 10; done
+```
 
-3. In Prometheus, click on `Status` in the top menu and then click `Targets`.
+3. Wait for approximately 10 minutes...
 
-4. You will now see that the ServiceMonitor has the `State` UP:
+## Create a dashboard in Grafana to use the metrics
 
-![metrics](../../img/custom-metrics.png)
+1. In the left menu, click on `Apps` and open `Grafana`.
 
-Our metrics are now being scraped by the Team's Prometheus. Before we continue, let's first generate some load:
+2. Go to the Grafana dashboard homepage.
 
-1. Go to the [Expose services](expose-services.md) lab and expose the `custom-metrics` service
+3. In the top right click on `New` and then `New Dashboard`.
 
-2. Run the following command in your terminal:
+4. Click `+ Add visualization`.
 
-```bash
-for i in {1..1000}; do curl https://custom-metrics-labs.<your-domain>/hello; sleep 10; done
+5. In the `Query` tab select `Prometheus Platform`.
+
+6. In the `A` collapsible section, select a metric from the `Metric` drop-down list. In our example we use the `application_greetings_total` metric.
+
+7. Click `Run queries`.
+
+8. You should now see a `Time series` graph like this:
+
+![dashboards](../../img/dashboards-1.png)
+
+9. Save the dashboard in Grafana.
+
+## Make the dashboard persistent
+
+Now you know how to create a dashboard in Grafana for your custom metrics. You could now save the dashboard, but if Grafana would get re-started, the dashboard will be gone. To make the dashboard persistent we need to add it to a configmap.
+
+1. Go to `apps` and open `Gitea`.
+
+2. In the list of repositories there is a repository called `otomi/team-<team-name>-argocd`. Go to this repository.
+
+3. Click `Add File` and then `New File`.
+
+4. Name the file `my-custom-dashboard.yaml`.
+
+5. Add the following manifest to the file:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: my-custom-dashboard
+  labels:
+    grafana_dashboard: "1"
+    release: grafana-dashboards-labs # change labs to the name of your team
+data:
+  my-dashboard.json: |-
+    # paste your dashboard json here
 ```
 
-3. Wait for approximately 10 minutes...
+6. Before commiting changes, go back to Grafana.
+
+7. Click on `Dashboard settings` (in the top right).
+
+8. In the left menu click `JSON model`.
 
-## See the custom metrics
+9. Copy the JSON model and paste it into the ConfigMap. Make sure to indent with 4.
 
-To see the custom metrics:
+10. Delete the dashboard created in Grafana.
 
-1. Open the `Prometheus` app.
+11. Commit changes in Gitea.
 
-2. In Prometheus, fill in the following Expression: `application_greetings_total`.
+The dashboard will now automatically be loaded into the Team's Grafana instance.
 
-3. Click on `Graph`.
+## Create custom rules
+
+Now we are exporting metrics, these metrics can also be used to generate alerts. To generate alerts, we first need to create a Prometheus `Rule`:
+
+1. Go to `apps` and open `Gitea`.
+
+2. In the list of repositories there is a repository called `otomi/team-<team-name>-argocd`. Go to this repository.
+
+3. Click `Add File` and then `New File`.
+
+4. Name the file `my-custom-rules.yaml`.
+
+5. Add the following manifest to the file:
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  labels:
+    prometheus: system
+  name: labs-custom-rules
+spec:
+  groups:
+  - name: custom.rules
+    rules:
+    - alert: 50GreetingsReached
+      annotations:
+        description: We reached 50 greetings!
+        summary: The number of greetings has reached more than 50.
+      expr: application_greetings_total > 50
+      for: 1m
+      labels:
+        severity: warning
+```
 
-4. You should now see the following:
+6. Commit changes in Gitea.
 
-![metrics](../../img/custom-metrics-1.png)
+## See alerts based on the rule in Alertmanager
 
-## Next steps
+1. Go to `Apps` and open `Alertmanager`
 
-Prometheus is now scraping our custom metrics. You can now use these metrics to:
+2. You will see Alertmanager has received the alerts from Prometheus:
 
-- Create a dashboard in Grafana in the lab [Create custom dashboards](custom-dashboards.md)
+![rules](../../img/rules-3.png)
 
-- Create rules and send alerts in the lab [Create custom rules](custom-rules.md)
+If a receiver has been configured for the Team, like Slack, then you would also have received a message with the alert.

docs/get-started/labs/labs-overview.md

@@ -56,7 +56,7 @@ Welcome to the Application Platform for LKE labs! If you are going to use the Ap
 
 ### [Scan images for vulnerabilities](scan-images.md)
 
-### [Scan your running containers for vulnerabilities](scan-containers.md)
+### [Scan running containers for vulnerabilities](scan-containers.md)
 
 ---
 
@@ -68,11 +68,7 @@ Welcome to the Application Platform for LKE labs! If you are going to use the Ap
 
 ### [Using custom metrics](custom-metrics.md)
 
-### [Create custom dashboards](custom-dashboards.md)
-
-### [Create custom rules](custom-rules.md)
-
-### [Monitoring availability of Workloads](monitor-workloads.md)
+### [Monitoring availability of Services](monitor-services.md)
 
 ---
 

docs/get-started/labs/labs-prerequisites.md

@@ -8,19 +8,27 @@ sidebar_label: Lab Prerequisites
 
 1. DNS is configured. See [here](get-started/installation/overview.md).
 
-2. The following applications are enabled:
+2. Platform Applications that are required for the following labs are enabled:
 
-- Harbor
-- Prometheus
-- Loki
-- Grafana
-- Trivy Operator
+| Lab  | App   |
+| ---- | ----- |
+| Build images | Harbor |
+| Trigger builds | Harbor |
+| Push images to harbor | Harbor |
+| Scan running containers for vulnerabilities | Prometheus, Grafana, Trivy Operator |
+| View container logs | Loki, Grafana |
+| View container metrics | Prometheus |
+| Using custom metrics | Prometheus |
+| Monitoring availability of Workloads | Prometheus |
+| Tracing with Open Telemetry | Loki, Otel, Tempo |
+| Create a RabbitMQ cluster | RabbitMQ |
 
-For the [Using OpenTelemery](use-otel.md) Lab, the Tempo app needs to be enabled together with tracing configured in the `Istio` and `Nginx Ingress` apps.
 
-3. A team called `labs` is created with `Grafana`, `Prometheus` and `Alertmanager` activated.
+For the [Tracing with Open Telemetry](use-otel.md) Lab, tracing needs to be configured in the `Istio` and `Nginx Ingress` apps.
 
-4. A user account is created and added to the `labs` team group in Keycloak. In the labs we'll be using the user `labs-user`.
+3. A team called `labs` is created with `Alertmanager` activated.
+
+4. A user account is created and added to the `labs` Team. In the labs we'll be using the user `labs-user`.
 
 ## Sign in to the Console
 
@@ -30,4 +38,4 @@ For the [Using OpenTelemery](use-otel.md) Lab, the Tempo app needs to be enabled
 
 After sign in, you will see this page:
 
-![Team apps](../../img/team-dashboard.png)
+![Team dashboard](../../img/team-dashboard.png)