SquareOps Technologies Your DevOps Partner for Accelerating cloud journey.
This PGL module is for monitoring and analyzing logs and metrics from various sources. It includes these features Grafana, Prometheus, Loki, Mimir and Loki-scalable.
Grafana is an open-source platform for monitoring and observability, offering customizable dashboards, alerts, and data visualization for a wide range of data sources.
Prometheus is an open-source systems monitoring and alerting toolkit designed for reliability and scalability, providing powerful queries, storage, and visualization of time series data.
Loki is a log aggregation system that allows you to store, search, and analyze large volumes of logs from different sources. With Loki, you can quickly find the relevant logs and troubleshoot issues in your system. It uses a unique indexing method that stores metadata separately from the log data, making it very efficient and scalable.
Mimir is a metric aggregation system that allows you to collect, store, and analyze metrics from various sources. It supports various data sources such as Prometheus, Graphite, and InfluxDB. With Mimir, you can visualize metrics using a variety of charts, graphs, and dashboards.
This PGL module includes multiple dashboards that provide a comprehensive view of your system's health and performance. These dashboards include system performance, error tracking, network performance, and more.
Loki-scalable is a horizontally scalable, highly available distributed logging system designed for storing and querying logs from all your applications and infrastructure.
This module also includes alerting features that allow you to set up custom alerts for specific events or conditions. You can configure alerts to notify you via email, Slack, or other channels, and set up automated responses to resolve issues quickly.
| Resources | Helm Chart Version | K8s supported version |
|---|---|---|
| Kube-Prometheus-Stack | 61.1.0 | 1.23,1.24,1.25,1.26,1.27,1.28,1.29,1.30 |
| Prometheus-Blackbox-Exporter | 8.17.0 | 1.23,1.24,1.25,1.26,1.27,1.28,1.29,1.30 |
| Mimir | 5.4.0 | 1.23,1.24,1.25,1.26,1.27,1.28,1.29,1.30 |
| Loki-Stack | 2.10.2 | 1.23,1.24,1.25,1.26,1.27,1.28,1.29,1.30 |
| Loki-Scalable | 6.7.1 | 1.23,1.24,1.25,1.26,1.27,1.28,1.29 |
| Tempo | 1.6.2 | 1.23,1.24,1.25,1.26,1.27 |
| OTEL | 0.37.0 | 1.23,1.24,1.25,1.26,1.27 |
module "pgl" {
source = "squareops/grafana-stack/kubernetes"
version = "3.0.3"
cluster_name = ""
kube_prometheus_stack_enabled = true
loki_enabled = false
loki_scalable_enabled = true
grafana_mimir_enabled = true
cloudwatch_enabled = true
tempo_enabled = false
deployment_config = {
hostname = "grafana.squareops.com"
storage_class_name = "infra-service-sc"
prometheus_values_yaml = file("./helm/prometheus.yaml")
loki_values_yaml = file("./helm/loki.yaml")
blackbox_values_yaml = file("./helm/blackbox.yaml")
grafana_mimir_values_yaml = file("./helm/mimir.yaml")
tempo_values_yaml = file("./helm/tempo.yaml")
dashboard_refresh_interval = ""
grafana_enabled = true
prometheus_hostname = "prometheus.com"
prometheus_internal_ingress_enabled = false
grafana_ingress_load_balancer = "nlb" ##Choose your load balancer type (e.g., NLB or ALB). If using ALB, ensure you provide the ACM certificate ARN for SSL.
alb_acm_certificate_arn = "" #"arn:aws:acm:${local.region}:444455556666:certificate/certificate_ID"
loki_internal_ingress_enabled = false
loki_hostname = "loki.com"
mimir_s3_bucket_config = {
s3_bucket_name = "${local.environment}-${local.name}-mimir-bucket"
versioning_enabled = "false"
s3_bucket_region = "${local.region}"
s3_object_expiration = 90
}
loki_scalable_config = {
loki_scalable_version = "6.7.1"
loki_scalable_values = file("./helm/loki-scalable.yaml")
s3_bucket_name = "${local.environment}-${local.name}-loki-scalable-bucket"
versioning_enabled = "false"
s3_bucket_region = "${local.region}"
}
promtail_config = {
promtail_version = "6.16.3"
promtail_values = file("./helm/promtail.yaml")
}
tempo_config = {
s3_bucket_name = "${local.environment}-${local.name}-tempo-skaf"
versioning_enabled = false
s3_bucket_region = "${local.region}"
s3_object_expiration = "90"
}
otel_config = {
otel_operator_enabled = false
otel_collector_enabled = false
}
}
exporter_config = {
json = false
nats = false
nifi = false
snmp = false
druid = false
istio = false
kafka = false
mysql = false
redis = false
argocd = false
consul = false
statsd = false
couchdb = false
jenkins = false
mongodb = false
pingdom = false
rabbitmq = false
blackbox = false
postgres = false
conntrack = false
stackdriver = false
push_gateway = false
elasticsearch = false
prometheustosd = false
ethtool_exporter = false
}
}Refer examples for more details.
The required IAM permissions to create resources from this module can be found here
- In order to enable the exporter, it is required to deploy Prometheus/Grafana first.
- The exporter is a tool that extracts metrics data from an application or system and makes it available to be scraped by Prometheus.
- Prometheus is a monitoring system that collects metrics data from various sources, including exporters, and stores it in a time-series database.
- Grafana is a data visualization and dashboard tool that works with Prometheus and other data sources to display the collected metrics in a user-friendly way.
- To deploy Prometheus/Grafana, please follow the installation instructions for each tool in their respective documentation.
- Once Prometheus and Grafana are deployed, the exporter can be configured to scrape metrics data from your application or system and send it to Prometheus.
- Finally, you can use Grafana to create custom dashboards and visualize the metrics data collected by Prometheus.
- If we enable internal ingress for prometheus and loki then we will be able to access it on private endpoint via vpn.
- This module is compatible with EKS version 1.23,1.24,1.25,1.26,1.27,1.28,1.29 which is great news for users deploying the module on an EKS cluster running that version. Review the module's documentation, meet specific configuration requirements, and test thoroughly after deployment to ensure everything works as expected.
No requirements.
| Name | Version |
|---|---|
| aws | n/a |
| helm | n/a |
| kubernetes | n/a |
| null | n/a |
| random | n/a |
| time | n/a |
| Name | Source | Version |
|---|---|---|
| loki_scalable_s3_bucket | terraform-aws-modules/s3-bucket/aws | 4.1.2 |
| s3_bucket_mimir | terraform-aws-modules/s3-bucket/aws | 4.1.2 |
| s3_bucket_temp | terraform-aws-modules/s3-bucket/aws | 4.1.2 |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| blackbox_exporter_version | Version of the Blackbox exporter to deploy. | string |
"8.17.0" |
no |
| cloudwatch_enabled | Whether or not to add CloudWatch as datasource and add some default dashboards for AWS in Grafana. | bool |
false |
no |
| cluster_name | Specifies the name of the EKS cluster. | string |
n/a | yes |
| deployment_config | Configuration options for the Prometheus, Alertmanager, Loki, and Grafana deployments, including the hostname, storage class name, dashboard refresh interval, and S3 bucket configuration for Mimir. | any |
{ |
no |
| exporter_config | allows enabling/disabling various exporters for scraping metrics, including Consul, MongoDB, Redis, and StatsD. | map(any) |
{ |
no |
| grafana_mimir_enabled | Specify whether or not to deploy the Grafana Mimir plugin. | bool |
false |
no |
| grafana_mimir_version | Version of the Grafana Mimir plugin to deploy. | string |
"5.4.0" |
no |
| kube_prometheus_stack_enabled | Specify whether or not to deploy Grafana as part of the Prometheus and Alertmanager stack. | bool |
false |
no |
| loki_enabled | Whether or not to deploy Loki for log aggregation and querying. | bool |
false |
no |
| loki_scalable_enabled | Specify whether or not to deploy the loki scalable | bool |
false |
no |
| loki_stack_version | Version of the Loki stack to deploy. | string |
"2.10.2" |
no |
| pgl_namespace | Name of the Kubernetes namespace where the Grafana deployment will be deployed. | string |
"monitoring" |
no |
| prometheus_chart_version | Version of the Prometheus chart to deploy. | string |
"61.1.0" |
no |
| tempo_enabled | Enable Grafana Tempo | bool |
false |
no |
| Name | Description |
|---|---|
| grafana | Grafana_Info |
To report an issue with a project:
- Check the repository's issue tracker on GitHub
- Search to see if the issue has already been reported
- If you can't find an answer to your question in the documentation or issue tracker, you can ask a question by creating a new issue. Be sure to provide enough context and details so others can understand your problem.
Apache License, Version 2.0, January 2004 (http://www.apache.org/licenses/).
To support a GitHub project by liking it, you can follow these steps:
-
Visit the repository: Navigate to the GitHub repository.
-
Click the "Star" button: On the repository page, you'll see a "Star" button in the upper right corner. Clicking on it will star the repository, indicating your support for the project.
-
Optionally, you can also leave a comment on the repository or open an issue to give feedback or suggest changes.
Starring a repository on GitHub is a simple way to show your support and appreciation for the project. It also helps to increase the visibility of the project and make it more discoverable to others.
We believe that the key to success in the digital age is the ability to deliver value quickly and reliably. That’s why we offer a comprehensive range of DevOps & Cloud services designed to help your organization optimize its systems & Processes for speed and agility.
- We are an AWS Advanced consulting partner which reflects our deep expertise in AWS Cloud and helping 100+ clients over the last 5 years.
- Expertise in Kubernetes and overall container solution helps companies expedite their journey by 10X.
- Infrastructure Automation is a key component to the success of our Clients and our Expertise helps deliver the same in the shortest time.
- DevSecOps as a service to implement security within the overall DevOps process and helping companies deploy securely and at speed.
- Platform engineering which supports scalable,Cost efficient infrastructure that supports rapid development, testing, and deployment.
- 24*7 SRE service to help you Monitor the state of your infrastructure and eradicate any issue within the SLA.
We provide support on all of our projects, no matter how small or large they may be.
To find more information about our company, visit squareops.com, follow us on Linkedin, or fill out a job application. If you have any questions or would like assistance with your cloud strategy and implementation, please don't hesitate to contact us.