Prometheus Remote Write Exporter (6/6)

.flake8

@@ -4,7 +4,7 @@ ignore =
   F401  # unused import, defer to pylint
   W503  # allow line breaks before binary ops
   W504  # allow line breaks after binary ops
-  E203  # allow whitespace before ':' (https://github.com/psf/black#slices) 
+  E203  # allow whitespace before ':' (https://github.com/psf/black#slices)
 exclude =
   .bzr
   .git

.pylintrc

@@ -7,7 +7,7 @@ extension-pkg-whitelist=
 
 # Add list of files or directories to be excluded. They should be base names, not
 # paths.
-ignore=CVS,gen
+ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml
 
 # Add files or directories matching the regex patterns to be excluded. The
 # regex matches against base names, not paths.
@@ -192,7 +192,7 @@ ignored-classes=optparse.Values,thread._local,_thread._local
 # (useful for modules/projects where namespaces are manipulated during runtime
 # and thus existing member attributes cannot be deduced by static analysis. It
 # supports qualified module names, as well as Unix pattern matching.
-ignored-modules=
+ignored-modules=docker-compose,README,Dockerfile,cortex-config,requirements
 
 # Show a hint with possible names when a member name was not found. The aspect
 # of finding the hint is based on edit distance.

exporter/opentelemetry-exporter-prometheus-remote-write/README.md

@@ -0,0 +1,177 @@
+# OpenTelemetry Python SDK Prometheus Remote Write Exporter
+This package contains an exporter to send [OTLP](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md)
+metrics from the Python SDK directly to a Prometheus Remote Write integrated
+backend (such as Cortex or Thanos) without having to run an instance of the
+Prometheus server. The image below shows the two Prometheus exporters in the OpenTelemetry Python SDK.
+
+
+Pipeline 1 illustrates the setup required for a Prometheus "pull" exporter.
+
+
+Pipeline 2 illustrates the setup required for the Prometheus Remote Write exporter.
+
+![Prometheus SDK pipelines](https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png)
+
+
+The Prometheus Remote Write Exporter is a "push" based exporter and only works with the OpenTelemetry [push controller](https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py).
+The controller periodically collects data and passes it to the exporter. This
+exporter then converts the data into [`timeseries`](https://prometheus.io/docs/concepts/data_model/) and sends it to the Remote Write integrated backend through HTTP
+POST requests. The metrics collection datapath is shown below:
+
+![controller_datapath_final](https://user-images.githubusercontent.com/20804975/100486582-79d1f380-30d2-11eb-8d17-d3e58e5c34e9.png)
+
+See the `example` folder for a demo usage of this exporter
+
+# Table of Contents
+   * [Summary](#opentelemetry-python-sdk-prometheus-remote-write-exporter)
+   * [Table of Contents](#table-of-contents)
+      * [Installation](#installation)
+      * [Quickstart](#quickstart)
+      * [Configuring the Exporter](#configuring-the-exporter)
+      * [Securing the Exporter](#securing-the-exporter)
+         * [Authentication](#authentication)
+         * [TLS](#tls)
+      * [Supported Aggregators](#supported-aggregators)
+      * [Error Handling](#error-handling)
+      * [Retry Logic](#retry-logic)
+      * [Contributing](#contributing)
+         * [Design Doc](#design-doc)
+
+## Installation
+
+* To install from the latest PyPi release,
+  run `pip install opentelemetry-exporter-prometheus-remote-write`
+* To install from the local repository, run
+  `pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/` in
+  the project root
+
+## Quickstart
+
+```python
+from opentelemetry import metrics
+from opentelemetry.sdk.metrics import MeterProvider
+from opentelemetry.exporter.prometheus_remote_write import (
+    PrometheusRemoteWriteMetricsExporter
+)
+
+# Sets the global MeterProvider instance
+metrics.set_meter_provider(MeterProvider())
+
+# The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here.
+meter = metrics.get_meter(__name__)
+
+exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed
+
+metrics.get_meter_provider().start_pipeline(meter, exporter, 5)
+```
+
+## Configuring the Exporter
+
+The exporter can be configured through parameters passed to the constructor.
+Here are all the options:
+
+* `endpoint`: url where data will be sent **(Required)**
+* `basic_auth`: username and password for authentication **(Optional)**
+* `headers`: additional headers for remote write request as determined by the remote write backend's API **(Optional)**
+* `timeout`: timeout for requests to the remote write endpoint in seconds **(Optional)**
+* `proxies`: dict mapping request proxy protocols to proxy urls **(Optional)**
+* `tls_config`: configuration for remote write TLS settings **(Optional)**
+
+Example with all the configuration options:
+
+```python
+exporter = PrometheusRemoteWriteMetricsExporter(
+    endpoint="http://localhost:9009/api/prom/push",
+    timeout=30,
+    basic_auth={
+        "username": "user",
+        "password": "pass123",
+    },
+    headers={
+        "X-Scope-Org-ID": "5",
+        "Authorization": "Bearer mytoken123",
+    },
+    proxies={
+        "http": "http://10.10.1.10:3000",
+        "https": "http://10.10.1.10:1080",
+    },
+    tls_config={
+        "cert_file": "path/to/file",
+        "key_file": "path/to/file",
+        "ca_file": "path_to_file",
+		"insecure_skip_verify": true, # for developing purposes
+    }
+)
+
+```
+## Securing the Exporter
+
+### Authentication
+
+The exporter provides two forms of authentication which are shown below. Users
+can add their own custom authentication by setting the appropriate values in the `headers` dictionary
+
+1. Basic Authentication
+Basic authentication sets a HTTP Authorization header containing a base64 encoded username/password pair. See [RFC 7617](https://tools.ietf.org/html/rfc7617) for more information. This
+
+```python
+exporter = PrometheusRemoteWriteMetricsExporter(
+    basic_auth={"username": "base64user",  "password": "base64pass"}
+)
+```
+2. Bearer Token Authentication
+This custom configuration can be achieved by passing in a custom `header` to
+the constructor. See [RFC 6750](https://tools.ietf.org/html/rfc6750) for more information.
+
+
+```python
+header = {
+    "Authorization": "Bearer mytoken123"
+}
+```
+
+### TLS
+Users can add TLS to the exporter's HTTP Client by providing certificate and key files in the `tls_config` parameter.
+
+## Supported Aggregators
+
+* Sum
+* MinMaxSumCount
+* Histogram
+* LastValue
+* ValueObserver
+
+## Error Handling
+In general, errors are raised by the calling function. The exception is for
+failed requests where any error status code is logged as a
+warning instead.
+
+This is because the exporter does not implement any retry logic
+as it sends cumulative metrics data. This means that data will be preserved even if some exports fail.
+
+For example, consider a situation where a user increments a Counter instrument 5 times and an export happens between each increment. If the exports happen like so:
+```
+SUCCESS FAIL FAIL SUCCESS SUCCESS
+1       2    3    4       5
+```
+Then the recieved data will be:
+```
+1 4 5
+```
+The end result is the same since the aggregations are cumulative
+## Contributing
+
+This exporter's datapath is as follows:
+
+![Exporter datapath](https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png)
+*Entites with `*` after their name are not actual classes but rather logical
+groupings of functions within the exporter.*
+
+If you would like to learn more about the exporter's structure and design decisions please view the design document below
+
+### Design Doc
+
+[Design Document](TODO: add link)
+
+This document is stored elsewhere as it contains large images which will
+significantly increase the size of this repo.

exporter/opentelemetry-exporter-prometheus-remote-write/examples/Dockerfile

@@ -0,0 +1,8 @@
+FROM python:3.7
+WORKDIR /code
+
+COPY . .
+RUN apt-get update -y && apt-get install libsnappy-dev -y
+RUN pip install -e .
+RUN pip install -r ./examples/requirements.txt
+CMD ["python", "./examples/sampleapp.py"]

exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md

@@ -0,0 +1,42 @@
+# Prometheus Remote Write Exporter Example
+This example uses [Docker Compose](https://docs.docker.com/compose/) to set up:
+
+1. A Python program that creates 5 instruments with 5 unique
+aggregators and a randomized load generator
+2. An instance of [Cortex](https://cortexmetrics.io/) to recieve the metrics
+data
+3. An instance of [Grafana](https://grafana.com/) to visualizse the exported
+data
+
+## Requirements
+* Have Docker Compose [installed](https://docs.docker.com/compose/install/)
+
+*Users do not need to install Python as the app will be run in the Docker Container*
+
+## Instructions
+1. Run `docker-compose up -d` in the the `examples/` directory
+
+The `-d` flag causes all services to run in detached mode and frees up your
+terminal session. This also causes no logs to show up. Users can attach themselves to the service's logs manually using `docker logs ${CONTAINER_ID} --follow`
+
+2. Log into the Grafana instance at [http://localhost:3000](http://localhost:3000)
+   * login credentials are `username: admin` and `password: admin`
+   * There may be an additional screen on setting a new password. This can be skipped and is optional
+
+3. Navigate to the `Data Sources` page
+   * Look for a gear icon on the left sidebar and select `Data Sources`
+
+4. Add a new Prometheus Data Source
+   * Use `http://cortex:9009/api/prom` as the URL
+   * (OPTIONAl) set the scrape interval to `2s` to make updates appear quickly
+   * click `Save & Test`
+
+5. Go to `Metrics Explore` to query metrics
+   * Look for a compass icon on the left sidebar
+   * click `Metrics` for a dropdown list of all the available metrics
+   * (OPTIONAL) Adjust time range by clicking the `Last 6 hours` button on the upper right side of the graph
+   * (OPTIONAL) Set up auto-refresh by selecting an option under the dropdown next to the refresh button on the upper right side of the graph
+   * Click the refresh button and data should show up on hte graph
+
+6. Shutdown the services when finished
+   * Run `docker-compose down` in the examples directory

exporter/opentelemetry-exporter-prometheus-remote-write/examples/cortex-config.yml

@@ -0,0 +1,100 @@
+# This Cortex Config is copied from the Cortex Project documentation
+# Source: https://github.com/cortexproject/cortex/blob/master/docs/configuration/single-process-config.yaml
+
+# Configuration for running Cortex in single-process mode.
+# This configuration should not be used in production.
+# It is only for getting started and development.
+
+# Disable the requirement that every request to Cortex has a
+# X-Scope-OrgID header. `fake` will be substituted in instead.
+auth_enabled: false
+
+server:
+  http_listen_port: 9009
+
+  # Configure the server to allow messages up to 100MB.
+  grpc_server_max_recv_msg_size: 104857600
+  grpc_server_max_send_msg_size: 104857600
+  grpc_server_max_concurrent_streams: 1000
+
+distributor:
+  shard_by_all_labels: true
+  pool:
+    health_check_ingesters: true
+
+ingester_client:
+  grpc_client_config:
+    # Configure the client to allow messages up to 100MB.
+    max_recv_msg_size: 104857600
+    max_send_msg_size: 104857600
+    use_gzip_compression: true
+
+ingester:
+  # We want our ingesters to flush chunks at the same time to optimise
+  # deduplication opportunities.
+  spread_flushes: true
+  chunk_age_jitter: 0
+
+  walconfig:
+    wal_enabled: true
+    recover_from_wal: true
+    wal_dir: /tmp/cortex/wal
+
+  lifecycler:
+    # The address to advertise for this ingester.  Will be autodiscovered by
+    # looking up address on eth0 or en0; can be specified if this fails.
+    # address: 127.0.0.1
+
+    # We want to start immediately and flush on shutdown.
+    join_after: 0
+    min_ready_duration: 0s
+    final_sleep: 0s
+    num_tokens: 512
+    tokens_file_path: /tmp/cortex/wal/tokens
+
+    # Use an in memory ring store, so we don't need to launch a Consul.
+    ring:
+      kvstore:
+        store: inmemory
+      replication_factor: 1
+
+# Use local storage - BoltDB for the index, and the filesystem
+# for the chunks.
+schema:
+  configs:
+    - from: 2019-07-29
+      store: boltdb
+      object_store: filesystem
+      schema: v10
+      index:
+        prefix: index_
+        period: 1w
+
+storage:
+  boltdb:
+    directory: /tmp/cortex/index
+
+  filesystem:
+    directory: /tmp/cortex/chunks
+
+  delete_store:
+    store: boltdb
+
+purger:
+  object_store_type: filesystem
+
+frontend_worker:
+  # Configure the frontend worker in the querier to match worker count
+  # to max_concurrent on the queriers.
+  match_max_concurrent: true
+
+# Configure the ruler to scan the /tmp/cortex/rules directory for prometheus
+# rules: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules
+ruler:
+  enable_api: true
+  enable_sharding: false
+  storage:
+    type: local
+    local:
+      directory: /tmp/cortex/rules
+

exporter/opentelemetry-exporter-prometheus-remote-write/examples/docker-compose.yml

@@ -0,0 +1,33 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3.8"
+
+services:
+  cortex:
+    image: quay.io/cortexproject/cortex:v1.5.0
+    command:
+      - -config.file=./config/cortex-config.yml
+    volumes:
+      - ./cortex-config.yml:/config/cortex-config.yml:ro
+    ports:
+      - 9009:9009
+  grafana:
+    image: grafana/grafana:latest
+    ports:
+      - 3000:3000
+  sample_app:
+    build:
+      context: ../
+      dockerfile: ./examples/Dockerfile