Consul metricbeat module

CHANGELOG.next.asciidoc

@@ -324,6 +324,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Added 'server' Metricset to Zookeeper Metricbeat module {issue}8938[8938] {pull}10341[10341]
 - Release AWS module as GA. {pull}10345[10345]
 - Add overview dashboard to Zookeeper Metricbeat module {pull}10379[10379]
+- Add Consul Metricbeat module with Agent Metricset {pull}8631[8631]
 
 *Packetbeat*
 

metricbeat/docker-compose.yml

@@ -14,6 +14,7 @@ services:
       - ./module/aerospike/_meta/env
       - ./module/apache/_meta/env
       - ./module/ceph/_meta/env
+      - ./module/consul/_meta/env
       - ./module/couchbase/_meta/env
       - ./module/couchdb/_meta/env
       - ./module/dropwizard/_meta/env
@@ -58,6 +59,9 @@ services:
   ceph:
     build: ./module/ceph/_meta
 
+  consul:
+    build: ./module/consul/_meta
+
   couchbase:
     build: ./module/couchbase/_meta
 

metricbeat/docs/fields.asciidoc

@@ -19,6 +19,7 @@ grouped in the following categories:
 * <<exported-fields-ceph>>
 * <<exported-fields-cloud>>
 * <<exported-fields-common>>
+* <<exported-fields-consul>>
 * <<exported-fields-couchbase>>
 * <<exported-fields-couchdb>>
 * <<exported-fields-docker-processor>>
@@ -2103,6 +2104,125 @@ required: True
 The document type. Always set to "doc".
 
 
+--
+
+[[exported-fields-consul]]
+== consul fields
+
+Consul module
+
+
+
+
+[float]
+== agent fields
+
+Agent Metricset fetches metrics information from a Consul instance running as Agent
+
+
+
+
+*`consul.agent.autopilot.healthy`*::
++
+--
+type: boolean
+
+Overall health of the local server cluster
+
+--
+
+[float]
+== runtime fields
+
+Runtime related metrics
+
+
+
+*`consul.agent.runtime.sys.bytes`*::
++
+--
+type: long
+
+Number of bytes of memory obtained from the OS.
+
+--
+
+*`consul.agent.runtime.malloc_count`*::
++
+--
+type: long
+
+Heap objects allocated
+
+--
+
+*`consul.agent.runtime.heap_objects`*::
++
+--
+type: long
+
+Objects allocated on the heap and is a general memory pressure indicator. This may burst from time to time but should return to a steady state value.
+
+--
+
+*`consul.agent.runtime.goroutines`*::
++
+--
+type: long
+
+Running goroutines and is a general load pressure indicator. This may burst from time to time but should return to a steady state value.
+
+--
+
+
+*`consul.agent.runtime.alloc.bytes`*::
++
+--
+type: long
+
+Bytes allocated by the Consul process.
+
+--
+
+[float]
+== garbage_collector fields
+
+Garbage collector metrics
+
+
+*`consul.agent.runtime.garbage_collector.runs`*::
++
+--
+type: long
+
+Garbage collector total executions
+
+--
+
+[float]
+== pause fields
+
+Time that the garbage collector has paused the app
+
+
+
+*`consul.agent.runtime.garbage_collector.pause.current.ns`*::
++
+--
+type: long
+
+Garbage collector pause time in nanoseconds
+
+--
+
+
+*`consul.agent.runtime.garbage_collector.pause.total.ns`*::
++
+--
+type: long
+
+Nanoseconds consumed by stop-the-world garbage collection pauses since Consul started.
+
 --
 
 [[exported-fields-couchbase]]

metricbeat/docs/modules/consul.asciidoc

@@ -0,0 +1,53 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-module-consul]]
+== consul module
+
+beta[]
+
+This is the https://www.consul.io[Hashicorp's Consul] Metricbeat module. It is still in beta and under active development to add new Metricsets and introduce enhancements.
+
+[float]
+== Compatibility
+
+The module is being tested with https://github.com/hashicorp/docker-consul/blob/9bd2aa7ecf2414b8712e055f2374699148e8941c/0.X/Dockerfile[1.4.2] version
+
+[float]
+== Metricsets
+
+The following Metricset is included:
+
+=== `agent`
+
+`agent` Metricset fetches information from a Consul Agent in 'Client' mode. It fetches information about the health of the autopilot, runtime metrics and raft data.
+
+
+[float]
+=== Example configuration
+
+The consul module supports the standard configuration options that are described
+in <<configuration-metricbeat>>. Here is an example configuration:
+
+[source,yaml]
+----
+metricbeat.modules:
+- module: consul
+  metricsets:
+  - agent
+  enabled: true
+  period: 10s
+  hosts: ["localhost:8500"]
+
+----
+
+[float]
+=== Metricsets
+
+The following metricsets are available:
+
+* <<metricbeat-metricset-consul-agent,agent>>
+
+include::consul/agent.asciidoc[]
+

metricbeat/docs/modules/consul/agent.asciidoc

@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[metricbeat-metricset-consul-agent]]
+=== consul agent metricset
+
+beta[]
+
+include::../../../module/consul/agent/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-consul,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../module/consul/agent/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -19,6 +19,8 @@ This file is generated! See scripts/docs_collector.py
 |<<metricbeat-metricset-ceph-osd_df,osd_df>>   
 |<<metricbeat-metricset-ceph-osd_tree,osd_tree>>   
 |<<metricbeat-metricset-ceph-pool_disk,pool_disk>>   
+|<<metricbeat-module-consul,consul>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
+.1+| .1+|  |<<metricbeat-metricset-consul-agent,agent>> beta[]  
 |<<metricbeat-module-couchbase,Couchbase>>     |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .3+| .3+|  |<<metricbeat-metricset-couchbase-bucket,bucket>>   
 |<<metricbeat-metricset-couchbase-cluster,cluster>>   
@@ -171,6 +173,7 @@ include::modules/aerospike.asciidoc[]
 include::modules/apache.asciidoc[]
 include::modules/aws.asciidoc[]
 include::modules/ceph.asciidoc[]
+include::modules/consul.asciidoc[]
 include::modules/couchbase.asciidoc[]
 include::modules/couchdb.asciidoc[]
 include::modules/docker.asciidoc[]

metricbeat/metricbeat.reference.yml

@@ -154,6 +154,15 @@ metricbeat.modules:
   hosts: ["localhost:5000"]
   enabled: true
 
+#------------------------------- consul Module -------------------------------
+- module: consul
+  metricsets:
+  - agent
+  enabled: true
+  period: 10s
+  hosts: ["localhost:8500"]
+
+
 #------------------------------ Couchbase Module -----------------------------
 - module: couchbase
   metricsets: ["bucket", "cluster", "node"]

metricbeat/module/consul/_meta/Dockerfile

@@ -0,0 +1,9 @@
+FROM consul:1.4.2
+
+ENV CONSUL_BIND_INTERFACE='eth0'
+
+EXPOSE 8500
+
+# Use the same healthcheck as the Windows version of the image.
+# https://github.com/Microsoft/mssql-docker/blob/a3020afeec9be1eb2d67645ac739438eb8f2c545/windows/mssql-server-windows/dockerfile#L31
+HEALTHCHECK --interval=1s --retries=90 CMD curl http://0.0.0.0:8500/v1/agent/metrics

metricbeat/module/consul/_meta/config.yml

@@ -0,0 +1,7 @@
+- module: consul
+  metricsets:
+  - agent
+  enabled: true
+  period: 10s
+  hosts: ["localhost:8500"]
+

metricbeat/module/consul/_meta/docs.asciidoc

@@ -0,0 +1,15 @@
+This is the https://www.consul.io[Hashicorp's Consul] Metricbeat module. It is still in beta and under active development to add new Metricsets and introduce enhancements.
+
+[float]
+== Compatibility
+
+The module is being tested with https://github.com/hashicorp/docker-consul/blob/9bd2aa7ecf2414b8712e055f2374699148e8941c/0.X/Dockerfile[1.4.2] version
+
+[float]
+== Metricsets
+
+The following Metricset is included:
+
+=== `agent`
+
+`agent` Metricset fetches information from a Consul Agent in 'Client' mode. It fetches information about the health of the autopilot, runtime metrics and raft data.

metricbeat/module/consul/_meta/env

@@ -0,0 +1,2 @@
+CONSUL_HOST=consul
+CONSUL_PORT=8500

metricbeat/module/consul/_meta/fields.yml

@@ -0,0 +1,9 @@
+- key: consul
+  title: "consul"
+  description: >
+    Consul module
+  release: beta
+  fields:
+    - name: consul
+      type: group
+      fields:

metricbeat/module/consul/agent/_meta/data.json

@@ -0,0 +1,50 @@
+{
+    "@timestamp": "2017-10-12T08:05:34.853Z",
+    "agent": {
+        "hostname": "host.example.com",
+        "name": "host.example.com"
+    },
+    "consul": {
+        "agent": {
+            "autopilot": {
+                "healthy": true
+            },
+            "raft": {
+                "commit_time": {
+                    "ms": 0.04560700058937073
+                }
+            },
+            "runtime": {
+                "alloc": {
+                    "bytes": 6900784
+                },
+                "garbage_collector": {
+                    "pause": {
+                        "total": {
+                            "ns": 99490250
+                        }
+                    },
+                    "runs": 287
+                },
+                "goroutines": 77,
+                "heap_objects": 42590,
+                "malloc_count": 7087169,
+                "sys": {
+                    "bytes": 74119416
+                }
+            }
+        }
+    },
+    "event": {
+        "dataset": "consul.agent",
+        "duration": 115000,
+        "module": "consul"
+    },
+    "metricset": {
+        "name": "agent"
+    },
+    "service": {
+        "address": "localhost:8500",
+        "type": "consul"
+    }
+}

metricbeat/module/consul/agent/_meta/docs.asciidoc

@@ -0,0 +1,13 @@
+This is the 'agent' Metricset of the Hashicorp's Consul Metricbeat module.
+
+* *agent.autopilot.healthy*: Tracks the overall health of the local server cluster. If all servers are considered healthy by Autopilot, this will be set to 1. If any are unhealthy, this will be 0.
+* *agent.raft.apply*: This metric gives the number of logs committed since the last interval.
+* *agent.raft.commit_time.ms*: This tracks the average time in milliseconds it takes to commit a new entry to the transaction log of the leader
+* *agent.runtime.alloc.bytes*: This measures the number of bytes allocated by the Consul process.
+* *agent.runtime.garbage_collector.pause.current.ns*: Garbage collector pause time in nanoseconds
+* *agent.runtime.garbage_collector.pause.total.ns*: Number of nanoseconds consumed by stop-the-world garbage collection pauses since Consul started.
+* *agent.runtime.garbage_collector.runs*: Garbage collector total executions
+* *agent.runtime.goroutines*: Number of running goroutines and is a general load pressure indicator. This may burst from time to time but should return to a steady state value.
+* *agent.runtime.heap_objects*: This measures the number of objects allocated on the heap and is a general memory pressure indicator. This may burst from time to time but should return to a steady state value.
+* *agent.runtime.malloc_count*: Heap objects allocated
+* *agent.runtime.sys.bytes*: Total number of bytes of memory obtained from the OS.