Cherry-pick #15990 to 7.x: Add citadel metricset for Istio Metricbeat module

CHANGELOG.next.asciidoc

@@ -201,6 +201,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add `key/value` mode for SQL module. {issue}15770[15770] {pull]15845[15845]
 - Add support for Unix socket in Memcached metricbeat module. {issue}13685[13685] {pull}15822[15822]
 - Make the `system/cpu` metricset collect normalized CPU metrics by default. {issue}15618[15618] {pull}15729[15729]
+- Add citadel metricset for Istio Metricbeat module {pull}15990[15990]
 - Add support for processors in light modules. {issue}14740[14740] {pull}15923[15923]
 
 *Packetbeat*

metricbeat/docs/fields.asciidoc

@@ -16602,6 +16602,135 @@ istio Module
 
 
 
+[float]
+=== citadel
+
+Contains statistics related to the Istio Citadel service
+
+
+
+*`istio.citadel.grpc.method`*::
++
+--
+The grpc method
+
+
+type: keyword
+
+--
+
+*`istio.citadel.grpc.service`*::
++
+--
+The grpc service
+
+
+type: keyword
+
+--
+
+*`istio.citadel.grpc.type`*::
++
+--
+The type of the respective grpc service
+
+
+type: keyword
+
+--
+
+*`istio.citadel.secret_controller_svc_acc_created_cert.count`*::
++
+--
+The number of certificates created due to service account creation.
+
+
+type: long
+
+--
+
+*`istio.citadel.server_root_cert_expiry_seconds`*::
++
+--
+The unix timestamp, in seconds, when Citadel root cert will expire. We set it to negative in case of internal error.
+
+
+type: float
+
+--
+
+*`istio.citadel.grpc.server.handled`*::
++
+--
+Total number of RPCs completed on the server, regardless of success or failure.
+
+
+type: long
+
+--
+
+*`istio.citadel.grpc.server.msg.received`*::
++
+--
+Total number of RPC stream messages received on the server.
+
+
+type: long
+
+--
+
+*`istio.citadel.grpc.server.msg.sent`*::
++
+--
+Total number of gRPC stream messages sent by the server.
+
+
+type: long
+
+--
+
+*`istio.citadel.grpc.server.started`*::
++
+--
+Total number of RPCs started on the server.
+
+
+type: long
+
+--
+
+*`istio.citadel.grpc.server.handling.latency.ms.bucket.*`*::
++
+--
+The response latency (milliseconds) of gRPC that had been application-level handled by the server.
+
+
+type: object
+
+--
+
+*`istio.citadel.grpc.server.handling.latency.ms.sum`*::
++
+--
+The response latency of gRPC, sum of latencies in milliseconds
+
+
+type: long
+
+format: duration
+
+--
+
+*`istio.citadel.grpc.server.handling.latency.ms.count`*::
++
+--
+The response latency of gRPC, number of metrics
+
+
+type: long
+
+--
+
 [float]
 === galley
 

metricbeat/docs/modules/istio.asciidoc

@@ -11,7 +11,7 @@ beta[]
 This is the Istio module. The Istio module collects metrics from the
 Istio https://istio.io/docs/tasks/observability/metrics/querying-metrics/#about-the-prometheus-add-on[prometheus exporters endpoints].
 
-The default metricsets are `mesh`, `mixer`, `pilot`, `galley`.
+The default metricsets are `mesh`, `mixer`, `pilot`, `galley`, `citadel`.
 
 [float]
 === Compatibility
@@ -55,13 +55,22 @@ metricbeat.modules:
   period: 10s
   # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
+
+# Istio citadel. To collect all Citadel-generated metrics.
+- module: istio
+  metricsets: ["citadel"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]
 ----
 
 [float]
 === Metricsets
 
 The following metricsets are available:
 
+* <<metricbeat-metricset-istio-citadel,citadel>>
+
 * <<metricbeat-metricset-istio-galley,galley>>
 
 * <<metricbeat-metricset-istio-mesh,mesh>>
@@ -70,6 +79,8 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-istio-pilot,pilot>>
 
+include::istio/citadel.asciidoc[]
+
 include::istio/galley.asciidoc[]
 
 include::istio/mesh.asciidoc[]

metricbeat/docs/modules/istio/citadel.asciidoc

@@ -0,0 +1,23 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-istio-citadel]]
+=== istio citadel metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/istio/citadel/_meta/docs.asciidoc[]
+
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-istio,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/istio/citadel/_meta/data.json[]
+----

metricbeat/docs/modules_list.asciidoc

@@ -104,7 +104,8 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-module-ibmmq,IBM MQ>>  beta[]   |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
 .1+| .1+|  |<<metricbeat-metricset-ibmmq-qmgr,qmgr>> beta[]  
 |<<metricbeat-module-istio,istio>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
-.4+| .4+|  |<<metricbeat-metricset-istio-galley,galley>> beta[]  
+.5+| .5+|  |<<metricbeat-metricset-istio-citadel,citadel>> beta[]  
+|<<metricbeat-metricset-istio-galley,galley>> beta[]  
 |<<metricbeat-metricset-istio-mesh,mesh>> beta[]  
 |<<metricbeat-metricset-istio-mixer,mixer>> beta[]  
 |<<metricbeat-metricset-istio-pilot,pilot>> beta[]  

x-pack/metricbeat/metricbeat.reference.yml

@@ -516,6 +516,13 @@ metricbeat.modules:
   # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
 
+# Istio citadel. To collect all Citadel-generated metrics.
+- module: istio
+  metricsets: ["citadel"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]
+
 #------------------------------- Jolokia Module -------------------------------
 - module: jolokia
   #metricsets: ["jmx"]

x-pack/metricbeat/module/istio/_meta/config.reference.yml

@@ -25,3 +25,10 @@
   period: 10s
   # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
+
+# Istio citadel. To collect all Citadel-generated metrics.
+- module: istio
+  metricsets: ["citadel"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]

x-pack/metricbeat/module/istio/_meta/config.yml

@@ -25,3 +25,10 @@
   period: 10s
   # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
   hosts: ["localhost:15014"]
+
+# Istio citadel. To collect all Citadel-generated metrics.
+- module: istio
+  metricsets: ["citadel"]
+  period: 10s
+  # use istio-pilot.istio-system:15014, when deploying Metricbeat in a kubernetes cluster as Pod or Daemonset
+  hosts: ["localhost:15014"]

x-pack/metricbeat/module/istio/_meta/docs.asciidoc

@@ -1,7 +1,7 @@
 This is the Istio module. The Istio module collects metrics from the
 Istio https://istio.io/docs/tasks/observability/metrics/querying-metrics/#about-the-prometheus-add-on[prometheus exporters endpoints].
 
-The default metricsets are `mesh`, `mixer`, `pilot`, `galley`.
+The default metricsets are `mesh`, `mixer`, `pilot`, `galley`, `citadel`.
 
 [float]
 === Compatibility

x-pack/metricbeat/module/istio/citadel/_meta/data.json

@@ -0,0 +1,24 @@
+{
+    "@timestamp": "2019-03-01T08:05:34.853Z",
+    "event": {
+        "dataset": "istio.citadel",
+        "duration": 115000,
+        "module": "istio"
+    },
+    "istio": {
+        "citadel": {
+            "secret_controller_svc_acc_created_cert": {
+                "count": 58
+            },
+            "server_root_cert_expiry_seconds": 1894287345
+        }
+    },
+    "metricset": {
+        "name": "citadel",
+        "period": 10000
+    },
+    "service": {
+        "address": "127.0.0.1:55555",
+        "type": "istio"
+    }
+}

x-pack/metricbeat/module/istio/citadel/_meta/docs.asciidoc

@@ -0,0 +1 @@
+This is the citadel metricset of the module istio. This metricset collects Citadel-specific metrics.

x-pack/metricbeat/module/istio/citadel/_meta/fields.yml

@@ -0,0 +1,59 @@
+- name: citadel
+  type: group
+  description: >
+    Contains statistics related to the Istio Citadel service
+  release: beta
+  fields:
+    - name: grpc.method
+      type: keyword
+      description: >
+        The grpc method
+    - name: grpc.service
+      type: keyword
+      description: >
+        The grpc service
+    - name: grpc.type
+      type: keyword
+      description: >
+        The type of the respective grpc service
+
+    - name: secret_controller_svc_acc_created_cert.count
+      type: long
+      description: >
+        The number of certificates created due to service account creation.
+    - name: server_root_cert_expiry_seconds
+      type: float
+      description: >
+        The unix timestamp, in seconds, when Citadel root cert will expire. We set it to negative in case of internal error.
+    - name: grpc.server.handled
+      type: long
+      description: >
+        Total number of RPCs completed on the server, regardless of success or failure.
+    - name: grpc.server.msg.received
+      type: long
+      description: >
+        Total number of RPC stream messages received on the server.
+    - name: grpc.server.msg.sent
+      type: long
+      description: >
+        Total number of gRPC stream messages sent by the server.
+    - name: grpc.server.started
+      type: long
+      description: >
+        Total number of RPCs started on the server.
+
+    - name: grpc.server.handling.latency.ms.bucket.*
+      type: object
+      object_type: long
+      description: >
+        The response latency (milliseconds) of gRPC that had been application-level handled by the server.
+    - name: grpc.server.handling.latency.ms.sum
+      type: long
+      format: duration
+      description: >
+        The response latency of gRPC, sum of latencies in milliseconds
+    - name: grpc.server.handling.latency.ms.count
+      type: long
+      description: >
+        The response latency of gRPC, number of metrics
+

x-pack/metricbeat/module/istio/citadel/_meta/testdata/config.yml

@@ -0,0 +1,3 @@
+type: http
+url: "/metrics"
+suffix: plain