diff --git a/docs/get-started/labs/custom-dashboards.md b/docs/get-started/labs/custom-dashboards.md deleted file mode 100644 index 2fae1f99c..000000000 --- a/docs/get-started/labs/custom-dashboards.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -slug: custom-dashboards -title: Create custom dashboards -sidebar_label: Create Custom Dashboards ---- - -In the lab [Using custom metrics](custom-metrics.md) we created a Workload with a ServiceMonitor so the Team's Prometheus can scrape our custom metrics. In this lab we are going to create a custom Grafana dashboard and add the dashboard to a Configmap so the Team's Grafana will automatically load the dashboard. - -## Create a dashboard in Grafana - -1. In the left menu, click on `Apps` and open `Grafana`. - -2. Go to the Grafana dashboard homepage. - -3. In the top right click on `New` and then `New Dashboard`. - -4. Click `+ Add visualization`. - -5. In the `Query` tab select `Prometheus (default)`. - -6. In the `A` collapsible section, select a metric from the `Metric` drop-down list. In our example we use the `application_greetings_total` metric. - -7. Click `Run queries`. - -8. You should now see a `Time series` graph like this: - -![dashboards](../../img/dashboards-1.png) - -9. Save the dashboard in Grafana. - -## Make the dashboard persistent - -Now you know how to create a dashboard in Grafana for your custom metrics. You could now save the dashboard, but if Grafana would get re-started, the dashboard will be gone. To make the dashboard persistent we need to add it to a configmap. - -1. Go to `apps` and open `Gitea`. - -2. In the list of repositories there is a repository called `otomi/team--argocd`. Go to this repository. - -3. Click `Add File` and then `New File`. - -4. Name the file `my-custom-dashboard.yaml`. - -5. Add the following manifest to the file: - -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: my-custom-dashboard - labels: - grafana_dashboard: "1" - release: grafana-dashboards-labs # change labs to the name of your team -data: - my-dashboard.json: |- - # paste your dashboard json here -``` - -6. Before commiting changes, go back to Grafana. - -7. Click on `Dashboard settings` (in the top right). - -8. In the left menu click `JSON model`. - -9. Copy the JSON model and paste it into the ConfigMap. Make sure to indent with 4. - -10. Delete the dashboard created in Grafana. - -11. Commit changes in Gitea. - -The dashboard will now automatically be loaded into the Team's Grafana instance. - - - diff --git a/docs/get-started/labs/custom-metrics.md b/docs/get-started/labs/custom-metrics.md index 7873a48d5..e6efb2f98 100644 --- a/docs/get-started/labs/custom-metrics.md +++ b/docs/get-started/labs/custom-metrics.md @@ -5,18 +5,14 @@ sidebar_label: Using custom metrics --- :::info -For this lab Prometheus and Grafana need to be enabled for the Team. +For this lab Grafana and Alertmanager need to be enabled for the Team. ::: -## What are custom metrics - -Prometheus will collect all kind of standard container metrics like CPU and memory usage, but no (custom) business or customer metrics like the number of customers who logged into your app each hour. - -To be able to collect custom metrics you will need to expose this data in your code. This is called instrumentation of code and can include annotating the code with metadata, or adding in logic to calculate and expose data. +To be able to collect custom metrics you will first need to configure your app to expose metrics. This is called instrumentation of code and can include annotating the code with metadata, or adding in logic to calculate and expose data. Instrumenting code means you write code to expose information about the technical, business, and customer context. This information can then be collected and analyzed using Prometheus and Grafana. -In this lab we will use a container that exposes custom metrics and then show how the metrics can be collected and analysed. +In this lab we'll use a container that exposes custom metrics and then show how the metrics can be collected and analysed. ## Create a Workload @@ -42,7 +38,7 @@ servicePorts: targetPort: 8080 protocol: TCP name: web -replicaCount: 2 +replicaCount: 1 serviceMonitor: create: true endpoints: @@ -52,52 +48,126 @@ serviceMonitor: path: /q/metrics ``` -5. Click `Submit` and then `Deploy Changes` +5. Click `Submit` and then `Deploy Changes`. -## Check the status of the ServiceMonitor +Our metrics will now be scraped by the Platform Prometheus. Before we continue, let's first generate some load: -Check if the ServiveMonitor has been picked up by Prometheus: +1. Expose the `custom-metrics` service (see lab [Expose services](expose-services.md)). -1. In the left menu go to `Apps`. +2. Run the following command in your terminal: -2. Click on the `Prometheus` app. +```bash +for i in {1..1000}; do curl https://custom-metrics-labs./hello; sleep 10; done +``` -3. In Prometheus, click on `Status` in the top menu and then click `Targets`. +3. Wait for approximately 10 minutes... -4. You will now see that the ServiceMonitor has the `State` UP: +## Create a dashboard in Grafana to use the metrics -![metrics](../../img/custom-metrics.png) +1. In the left menu, click on `Apps` and open `Grafana`. -Our metrics are now being scraped by the Team's Prometheus. Before we continue, let's first generate some load: +2. Go to the Grafana dashboard homepage. -1. Go to the [Expose services](expose-services.md) lab and expose the `custom-metrics` service +3. In the top right click on `New` and then `New Dashboard`. -2. Run the following command in your terminal: +4. Click `+ Add visualization`. -```bash -for i in {1..1000}; do curl https://custom-metrics-labs./hello; sleep 10; done +5. In the `Query` tab select `Prometheus Platform`. + +6. In the `A` collapsible section, select a metric from the `Metric` drop-down list. In our example we use the `application_greetings_total` metric. + +7. Click `Run queries`. + +8. You should now see a `Time series` graph like this: + +![dashboards](../../img/dashboards-1.png) + +9. Save the dashboard in Grafana. + +## Make the dashboard persistent + +Now you know how to create a dashboard in Grafana for your custom metrics. You could now save the dashboard, but if Grafana would get re-started, the dashboard will be gone. To make the dashboard persistent we need to add it to a configmap. + +1. Go to `apps` and open `Gitea`. + +2. In the list of repositories there is a repository called `otomi/team--argocd`. Go to this repository. + +3. Click `Add File` and then `New File`. + +4. Name the file `my-custom-dashboard.yaml`. + +5. Add the following manifest to the file: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: my-custom-dashboard + labels: + grafana_dashboard: "1" + release: grafana-dashboards-labs # change labs to the name of your team +data: + my-dashboard.json: |- + # paste your dashboard json here ``` -3. Wait for approximately 10 minutes... +6. Before commiting changes, go back to Grafana. + +7. Click on `Dashboard settings` (in the top right). + +8. In the left menu click `JSON model`. -## See the custom metrics +9. Copy the JSON model and paste it into the ConfigMap. Make sure to indent with 4. -To see the custom metrics: +10. Delete the dashboard created in Grafana. -1. Open the `Prometheus` app. +11. Commit changes in Gitea. -2. In Prometheus, fill in the following Expression: `application_greetings_total`. +The dashboard will now automatically be loaded into the Team's Grafana instance. -3. Click on `Graph`. +## Create custom rules + +Now we are exporting metrics, these metrics can also be used to generate alerts. To generate alerts, we first need to create a Prometheus `Rule`: + +1. Go to `apps` and open `Gitea`. + +2. In the list of repositories there is a repository called `otomi/team--argocd`. Go to this repository. + +3. Click `Add File` and then `New File`. + +4. Name the file `my-custom-rules.yaml`. + +5. Add the following manifest to the file: + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + labels: + prometheus: system + name: labs-custom-rules +spec: + groups: + - name: custom.rules + rules: + - alert: 50GreetingsReached + annotations: + description: We reached 50 greetings! + summary: The number of greetings has reached more than 50. + expr: application_greetings_total > 50 + for: 1m + labels: + severity: warning +``` -4. You should now see the following: +6. Commit changes in Gitea. -![metrics](../../img/custom-metrics-1.png) +## See alerts based on the rule in Alertmanager -## Next steps +1. Go to `Apps` and open `Alertmanager` -Prometheus is now scraping our custom metrics. You can now use these metrics to: +2. You will see Alertmanager has received the alerts from Prometheus: -- Create a dashboard in Grafana in the lab [Create custom dashboards](custom-dashboards.md) +![rules](../../img/rules-3.png) -- Create rules and send alerts in the lab [Create custom rules](custom-rules.md) \ No newline at end of file +If a receiver has been configured for the Team, like Slack, then you would also have received a message with the alert. \ No newline at end of file diff --git a/docs/get-started/labs/custom-rules.md b/docs/get-started/labs/custom-rules.md deleted file mode 100644 index 4fa51f6e6..000000000 --- a/docs/get-started/labs/custom-rules.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -slug: custom-rules -title: Create custom rules -sidebar_label: Create Custom Rules ---- - -Now we are exporting metrics, these metrics can also be used to generate alerts. To generate alerts, we first need to create a Prometheus `Rule`: - -1. Go to `apps` and open `Gitea`. - -2. In the list of repositories there is a repository called `otomi/team--argocd`. Go to this repository. - -3. Click `Add File` and then `New File`. - -4. Name the file `my-custom-rules.yaml`. - -5. Add the following manifest to the file: - -```yaml -apiVersion: monitoring.coreos.com/v1 -kind: PrometheusRule -metadata: - labels: - prometheus: team-labs # change labs to the name of your team - name: labs-custom-rules -spec: - groups: - - name: custom.rules - rules: - - alert: 50GreetingsReached - annotations: - description: We reached 50 greetings! - summary: The number of greetings has reached more than 50. - expr: application_greetings_total > 50 - for: 1m - labels: - severity: warning -``` - -6. Commit changes in Gitea. - -## See rules in Prometheus - -Now let's check to see if Prometheus has picked-up the rule: - -1. Go to `Apps` and open `Prometheus`. - -2. In the menu click on `Status` and then `Rules`. You should now see the rule: - -![rules](../../img/rules-1.png) - -3. In the menu in Prometheus, click `Alerts` - -4. You should see an alert has been fired: - -![rules](../../img/rules-2.png) - -## See alerts in Alertmanager - -1. Go to `Apps` and open `Alertmanager` - -2. You will see Alertmanager has received the alerts from Prometheus: - -![rules](../../img/rules-3.png) - -If a receiver has been configured for the Team, like Slack or email, then you would also have received a message with the alert. \ No newline at end of file diff --git a/docs/get-started/labs/labs-overview.md b/docs/get-started/labs/labs-overview.md index 88c731a39..8918a966f 100644 --- a/docs/get-started/labs/labs-overview.md +++ b/docs/get-started/labs/labs-overview.md @@ -56,7 +56,7 @@ Welcome to the Application Platform for LKE labs! If you are going to use the Ap ### [Scan images for vulnerabilities](scan-images.md) -### [Scan your running containers for vulnerabilities](scan-containers.md) +### [Scan running containers for vulnerabilities](scan-containers.md) --- @@ -68,11 +68,7 @@ Welcome to the Application Platform for LKE labs! If you are going to use the Ap ### [Using custom metrics](custom-metrics.md) -### [Create custom dashboards](custom-dashboards.md) - -### [Create custom rules](custom-rules.md) - -### [Monitoring availability of Workloads](monitor-workloads.md) +### [Monitoring availability of Services](monitor-services.md) --- diff --git a/docs/get-started/labs/labs-prerequisites.md b/docs/get-started/labs/labs-prerequisites.md index c3f66c73d..26db6bad2 100644 --- a/docs/get-started/labs/labs-prerequisites.md +++ b/docs/get-started/labs/labs-prerequisites.md @@ -8,19 +8,27 @@ sidebar_label: Lab Prerequisites 1. DNS is configured. See [here](get-started/installation/overview.md). -2. The following applications are enabled: +2. Platform Applications that are required for the following labs are enabled: -- Harbor -- Prometheus -- Loki -- Grafana -- Trivy Operator +| Lab | App | +| ---- | ----- | +| Build images | Harbor | +| Trigger builds | Harbor | +| Push images to harbor | Harbor | +| Scan running containers for vulnerabilities | Prometheus, Grafana, Trivy Operator | +| View container logs | Loki, Grafana | +| View container metrics | Prometheus | +| Using custom metrics | Prometheus | +| Monitoring availability of Workloads | Prometheus | +| Tracing with Open Telemetry | Loki, Otel, Tempo | +| Create a RabbitMQ cluster | RabbitMQ | -For the [Using OpenTelemery](use-otel.md) Lab, the Tempo app needs to be enabled together with tracing configured in the `Istio` and `Nginx Ingress` apps. -3. A team called `labs` is created with `Grafana`, `Prometheus` and `Alertmanager` activated. +For the [Tracing with Open Telemetry](use-otel.md) Lab, tracing needs to be configured in the `Istio` and `Nginx Ingress` apps. -4. A user account is created and added to the `labs` team group in Keycloak. In the labs we'll be using the user `labs-user`. +3. A team called `labs` is created with `Alertmanager` activated. + +4. A user account is created and added to the `labs` Team. In the labs we'll be using the user `labs-user`. ## Sign in to the Console @@ -30,4 +38,4 @@ For the [Using OpenTelemery](use-otel.md) Lab, the Tempo app needs to be enabled After sign in, you will see this page: -![Team apps](../../img/team-dashboard.png) \ No newline at end of file +![Team dashboard](../../img/team-dashboard.png) \ No newline at end of file diff --git a/docs/get-started/labs/monitor-services.md b/docs/get-started/labs/monitor-services.md new file mode 100644 index 000000000..2a5d6231f --- /dev/null +++ b/docs/get-started/labs/monitor-services.md @@ -0,0 +1,49 @@ +--- +slug: monitor-services +title: Monitoring service availability +sidebar_label: Monitor Services +--- + +When your application is deployed and exposed, you probably would like to get an alert when your application (service) is not available anymore. To automatically monitor you applications for availability, a probe to monitor your service is automatically configured. + +In this lab we'll scale down the replicas of a workload to see how this works. + +## Check alerts in Alertmanager + +1. In the left menu, click on `Apps` and open `Alertmanager`. + +![alertmanager](../../img/team-alertmanager.png) + +2. As you can see, there are currently no alerts: + +![alertmanager](../../img/alertmanager-no-alerts.png) + +## Scale down the replicas of a Workload + +In the lab [Expose Services](expose-services) we exposed the `blue` service. Let's scale down the `blue` Workload and see what happens: + +1. Go to `Workloads` in the left menu and click on the `blue` Workload. + +2. Click on the `Values` tab. + +3. In the workload `values`, set the `replicaCount` to `0`: + +```yaml +replicaCount: 0 +``` + +4. Click `Submit`. + +5. Click `Deploy Changes`. + +## Check the alerts again + +Go back to Alertmanager and see if there are new alerts. You should see the following alert: + +![alertmanager](../../img/alertmanager-with-alert.png) + +Click on `+Info`: + +![alertmanager](../../img/alertmanager-with-alert-2.png) + +The Target `https://blue-labs.labs.try-apl.net/` is down! \ No newline at end of file diff --git a/docs/get-started/labs/monitor-workloads.md b/docs/get-started/labs/monitor-workloads.md deleted file mode 100644 index ddbe12747..000000000 --- a/docs/get-started/labs/monitor-workloads.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -slug: monitor-workloads -title: Monitoring availability -sidebar_label: Monitor Workloads ---- - -When your application is deployed, you would of course like to get an alert when you application (service) is not available anymore. To automatically monitor you applications for availability, a prope to monitor your service is automatically configured. - -## Monitor your application for availability - -1. [Create a Service](expose-services.md) for your app. The service can have an Exposure ingress of type `Cluster` or `Ingress`. - -2. Open Prometheus: - -![kubecfg](../../img/prometheus-teams.png) - -3. In Prometheus, Go to `Status` and click on `Targets`: - -![kubecfg](../../img/targets-up.png) - -In the list of targets you will see: - -- The `PodMonitor` endpoints of the `istio sidecars` os the Team Workloads - -- The `Probes` of all the Team services that are exposed - -4. In Prometheus, Go to `Alerts`: - -![kubecfg](../../img/prometheus-alerts.png) - -In the alerts you will see an (inactive) alert for `ProbeFailing`. If the `State` of a Servive Probe is `Down` the Prometheus `Rule` for this alert will fire. When alertmanager is enabled, and an alert notification receiver is configured, you will automatically receive an alert when your exposed Service is down. \ No newline at end of file diff --git a/docs/img/alertmanager-no-alerts.png b/docs/img/alertmanager-no-alerts.png new file mode 100644 index 000000000..9aa94b3ac Binary files /dev/null and b/docs/img/alertmanager-no-alerts.png differ diff --git a/docs/img/alertmanager-with-alert-2.png b/docs/img/alertmanager-with-alert-2.png new file mode 100644 index 000000000..279f73ca0 Binary files /dev/null and b/docs/img/alertmanager-with-alert-2.png differ diff --git a/docs/img/alertmanager-with-alert.png b/docs/img/alertmanager-with-alert.png new file mode 100644 index 000000000..236c86b4a Binary files /dev/null and b/docs/img/alertmanager-with-alert.png differ diff --git a/docs/img/dashboards-1.png b/docs/img/dashboards-1.png index 55473e766..466e5112f 100644 Binary files a/docs/img/dashboards-1.png and b/docs/img/dashboards-1.png differ diff --git a/docs/img/grafana-teams.png b/docs/img/grafana-teams.png index 583eb9b30..12aea8d2d 100644 Binary files a/docs/img/grafana-teams.png and b/docs/img/grafana-teams.png differ diff --git a/docs/img/loki-teams.png b/docs/img/loki-teams.png index 132dee3ed..b14ce99b9 100644 Binary files a/docs/img/loki-teams.png and b/docs/img/loki-teams.png differ diff --git a/docs/img/rules-3.png b/docs/img/rules-3.png index f588e8986..ac49c0ccb 100644 Binary files a/docs/img/rules-3.png and b/docs/img/rules-3.png differ diff --git a/docs/img/team-alertmanager.png b/docs/img/team-alertmanager.png new file mode 100644 index 000000000..add87aecf Binary files /dev/null and b/docs/img/team-alertmanager.png differ diff --git a/docs/img/team-app-argo.png b/docs/img/team-app-argo.png index 867639fb7..7c7013ed5 100644 Binary files a/docs/img/team-app-argo.png and b/docs/img/team-app-argo.png differ diff --git a/docs/img/team-app-gitea.png b/docs/img/team-app-gitea.png index 5af0a372a..f57f8cd63 100644 Binary files a/docs/img/team-app-gitea.png and b/docs/img/team-app-gitea.png differ diff --git a/sidebar-docs.js b/sidebar-docs.js index 0c18b884f..f742bf9e0 100644 --- a/sidebar-docs.js +++ b/sidebar-docs.js @@ -40,10 +40,8 @@ module.exports = { "get-started/labs/scan-containers", "get-started/labs/view-logs", "get-started/labs/view-metrics", - "get-started/labs/monitor-workloads", + "get-started/labs/monitor-services", "get-started/labs/custom-metrics", - "get-started/labs/custom-dashboards", - "get-started/labs/custom-rules", "get-started/labs/create-netpols", "get-started/labs/use-otel", "get-started/labs/canary-deployment",