Merge pull request #4 from YANG-DB/otel-opensearch-data-prep

Otel Opensearch data prep

.github/architecture.md

@@ -29,6 +29,7 @@ They are combined of:
  - [Shipping](https://opentelemetry.io/docs/demo/services/shipping/)
  - [Fluent-Bit](../src/fluent-bit/README.md) *(nginx's otel log exported)* 
  - [Integrations](../src/integrations/README.md) *(pre-canned OpenSearch assets)* 
+ - [DataPrepper](../src/dataprepper/README.md) *(OpenSearch's ingestion pipeline)
 
 Backend supportive services
  - [Load Generator](http://load-generator:8089)
@@ -45,16 +46,21 @@ Backend supportive services
    - See [description](../src/featureflagservice/README.md)
  - [Grafana](http://grafana:3000)
    - See [description](https://github.com/YANG-DB/opentelemetry-demo/blob/12d52cbb23bbf4226f6de2dfec840482a0a7d054/docker-compose.yml#L637)
-
 ---
 
 ## Purpose
 The purpose of this demo is to demonstrate the different capabilities of OpenSearch Observability to investigate and reflect your system.
 
- - Integrations - the integration service is a list of pre-canned assets that are loaded in a combined manner to allow users the ability
-for simple and automatic way to discover and review their services topology.
+### Ingestion 
+The ingestion capabilities for OpenSearch is to be able to support multiple pipelines:
+  - [Data-Prepper](https://github.com/opensearch-project/data-prepper/) is an OpenSearch ingestion project that allows ingestion of OTEL standard signals using Otel-Collector
+  - [Jaeger](https://opensearch.org/docs/latest/observing-your-data/trace/trace-analytics-jaeger/) is an ingestion framework which has a build in capability for pushing OTEL signals into OpenSearch
+  - [Fluent-Bit](https://docs.fluentbit.io/manual/pipeline/outputs/opensearch) is an ingestion framework which has a build in capability for pushing OTEL signals into OpenSearch
+
+### Integrations -
+The integration service is a list of pre-canned assets that are loaded in a combined manner to allow users the ability for simple and automatic way to discover and review their services topology.
 
-These integrations contain the following assets:
+These (demo-sample) integrations contain the following assets:
  - components & index template mapping
  - datasources 
  - data-stream & indices
@@ -78,7 +84,7 @@ Once these assets are loaded - the user can start reviewing its Observability da
 
 ![Traces](img/traces.png)
 
-
+![ServiceGraph](img/service-graph.png)
 ---
 
 ### **Scenarios**

docker-compose.yml

@@ -583,6 +583,19 @@ services:
   # ********************
   # Telemetry Components
   # ********************
+  # data-prepper
+  data-prepper:
+    image: opensearchproject/data-prepper:latest
+    container_name: data-prepper
+    restart: unless-stopped
+    volumes:
+      - ./src/dataprepper/trace_analytics_no_ssl_2x.yml:/usr/share/data-prepper/pipelines/pipelines.yaml
+      - ./src/dataprepper/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
+    ports:
+      - "21890:21890"
+    depends_on:
+      - opensearch-node1
+
   # Jaeger
   jaeger:
     image: jaegertracing/jaeger-collector:latest
@@ -670,6 +683,7 @@ services:
       - "8888"          # metrics endpoint
     depends_on:
       - jaeger-agent
+      - data-prepper
     logging: *logging
 
   # Prometheus

src/dataprepper/README.md

@@ -0,0 +1,52 @@
+![data-prepper](dataPrepper.svg)
+
+# What is Data Prepper
+
+[Data Prepper](https://github.com/opensearch-project/data-prepper/blob/main/docs/overview.md) is an open source utility service. Data Prepper is a server side data collector with abilities to filter, enrich, transform, normalize and aggregate data for downstream analytics and visualization. The broader vision for Data Prepper is to enable an end-to-end data analysis life cycle from gathering raw logs to facilitating sophisticated and actionable interactive ad-hoc analyses on the data.
+
+# What is Data Prepper Integration
+
+Data Prepper integration is concerned with the following aspects
+
+- Allow simple and automatic generation of all schematic structured
+    - traces ( including specific fields mapping to map to SS4O schema)
+    - services ( adding support for specific service mapping category)
+    - metrics (using the standard SS4O schema)
+
+- Add Dashboard Assets for correlation between traces-services-metrics
+
+- Add correlation queries to investigate traces based metrics
+
+# Data - Prepper Trace Fields
+Data Prepper uses the following [Traces](https://github.com/opensearch-project/data-prepper/blob/main/docs/schemas/trace-analytics/otel-v1-apm-span-index-template.md) mapping file
+The next fields are used:
+```text
+
+- traceId - A unique identifier for a trace. All spans from the same trace share the same traceId.
+- spanId - A unique identifier for a span within a trace, assigned when the span is created.
+- traceState - Conveys information about request position in multiple distributed tracing graphs.
+- parentSpanId - The spanId of this span's parent span. If this is a root span, then this field must be empty.
+- name - A description of the span's operation.
+- kind - The type of span. See OpenTelemetry - SpanKind.
+- startTime - The start time of the span.
+- endTime - The end time of the span.
+- durationInNanos - Difference in nanoseconds between startTime and endTime.
+- serviceName - Currently derived from the opentelemetry.proto.resource.v1.Resource associated with the span, the resource from the span originates.
+- events - A list of events. See OpenTelemetry - Events.
+- links - A list of linked spans. See OpenTelemetry - Links.
+- droppedAttributesCount - The number of attributes that were discarded.
+- droppedEventsCount - The number of events that were discarded.
+- droppedLinksCount - The number of links that were dropped.
+- span.attributes.* - All span attributes are split into a list of keywords.
+- resource.attributes.* - All resource attributes are split into a list of keywords.
+- status.code - The status of the span. See OpenTelemetry - Status.
+
+```
+There are some additional `trace.group` related fields which are not part of the [OTEL spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md) for traces
+```text
+- traceGroup - A derived field, the name of the trace's root span.
+- traceGroupFields.endTime - A derived field, the endTime of the trace's root span.
+- traceGroupFields.statusCode - A derived field, the status.code of the trace's root span.
+- traceGroupFields.durationInNanos - A derived field, the durationInNanos of the trace's root span.
+
+```

src/dataprepper/data-prepper-config.yaml

@@ -0,0 +1 @@
+ssl: false

src/dataprepper/root-ca.pem

@@ -0,0 +1,24 @@
+-----BEGIN CERTIFICATE-----
+MIID/jCCAuagAwIBAgIBATANBgkqhkiG9w0BAQsFADCBjzETMBEGCgmSJomT8ixk
+ARkWA2NvbTEXMBUGCgmSJomT8ixkARkWB2V4YW1wbGUxGTAXBgNVBAoMEEV4YW1w
+bGUgQ29tIEluYy4xITAfBgNVBAsMGEV4YW1wbGUgQ29tIEluYy4gUm9vdCBDQTEh
+MB8GA1UEAwwYRXhhbXBsZSBDb20gSW5jLiBSb290IENBMB4XDTE4MDQyMjAzNDM0
+NloXDTI4MDQxOTAzNDM0NlowgY8xEzARBgoJkiaJk/IsZAEZFgNjb20xFzAVBgoJ
+kiaJk/IsZAEZFgdleGFtcGxlMRkwFwYDVQQKDBBFeGFtcGxlIENvbSBJbmMuMSEw
+HwYDVQQLDBhFeGFtcGxlIENvbSBJbmMuIFJvb3QgQ0ExITAfBgNVBAMMGEV4YW1w
+bGUgQ29tIEluYy4gUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
+ggEBAK/u+GARP5innhpXK0c0q7s1Su1VTEaIgmZr8VWI6S8amf5cU3ktV7WT9SuV
+TsAm2i2A5P+Ctw7iZkfnHWlsC3HhPUcd6mvzGZ4moxnamM7r+a9otRp3owYoGStX
+ylVTQusAjbq9do8CMV4hcBTepCd+0w0v4h6UlXU8xjhj1xeUIz4DKbRgf36q0rv4
+VIX46X72rMJSETKOSxuwLkov1ZOVbfSlPaygXIxqsHVlj1iMkYRbQmaTib6XWHKf
+MibDaqDejOhukkCjzpptGZOPFQ8002UtTTNv1TiaKxkjMQJNwz6jfZ53ws3fh1I0
+RWT6WfM4oeFRFnyFRmc4uYTUgAkCAwEAAaNjMGEwDwYDVR0TAQH/BAUwAwEB/zAf
+BgNVHSMEGDAWgBSSNQzgDx4rRfZNOfN7X6LmEpdAczAdBgNVHQ4EFgQUkjUM4A8e
+K0X2TTnze1+i5hKXQHMwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IB
+AQBoQHvwsR34hGO2m8qVR9nQ5Klo5HYPyd6ySKNcT36OZ4AQfaCGsk+SecTi35QF
+RHL3g2qffED4tKR0RBNGQSgiLavmHGCh3YpDupKq2xhhEeS9oBmQzxanFwWFod4T
+nnsG2cCejyR9WXoRzHisw0KJWeuNlwjUdJY0xnn16srm1zL/M/f0PvCyh9HU1mF1
+ivnOSqbDD2Z7JSGyckgKad1Omsg/rr5XYtCeyJeXUPcmpeX6erWJJNTUh6yWC/hY
+G/dFC4xrJhfXwz6Z0ytUygJO32bJG4Np2iGAwvvgI9EfxzEv/KP+FGrJOvQJAq4/
+BU36ZAa80W/8TBnqZTkNnqZV
+-----END CERTIFICATE-----

src/dataprepper/trace_analytics_no_ssl_2x.yml

@@ -0,0 +1,37 @@
+entry-pipeline:
+  delay: "100"
+  source:
+    otel_trace_source:
+      ssl: false
+  sink:
+    - pipeline:
+        name: "raw-pipeline"
+    - pipeline:
+        name: "service-map-pipeline"
+raw-pipeline:
+  source:
+    pipeline:
+      name: "entry-pipeline"
+  processor:
+    - otel_trace_raw:
+  sink:
+    - opensearch:
+        hosts: [ "https://opensearch-node1:9200" ]
+        insecure: true
+        username: "admin"
+        password: "admin"
+        index_type: trace-analytics-raw
+service-map-pipeline:
+  delay: "100"
+  source:
+    pipeline:
+      name: "entry-pipeline"
+  processor:
+    - service_map_stateful:
+  sink:
+    - opensearch:
+        hosts: ["https://opensearch-node1:9200"]
+        insecure: true
+        username: "admin"
+        password: "admin"
+        index_type: trace-analytics-service-map

src/otelcollector/otelcol-config.yml

@@ -13,6 +13,11 @@ receivers:
             - "https://*"
 
 exporters:
+  otlp/2:
+    endpoint: "data-prepper:21890"
+    tls:
+      insecure: true
+      insecure_skip_verify: true
   otlp:
     endpoint: "jaeger:4317"
     tls:
@@ -41,7 +46,7 @@ service:
     traces:
       receivers: [otlp]
       processors: [batch]
-      exporters: [otlp, logging, spanmetrics]
+      exporters: [otlp, logging, spanmetrics, otlp/2]
     metrics:
       receivers: [otlp, spanmetrics]
       processors: [filter, batch]