Skip to content

Commit

Permalink
Humio exporter init (open-telemetry#3022)
Browse files Browse the repository at this point in the history
This is the first PR for the Humio exporter, containing readme, configuration, factory, and relevant tests.
The first series of PRs will focus on implementing exporting of traces, with logs to follow soon.

Apologies for the large PR, but I hope that the changes are sufficiently trivial.

**Link to tracking Issue:**
open-telemetry#3021

**Testing:**
- `config_test.go`: Tests that the configuration options can be correctly loaded, including defaults. Also tests configuration sanitization.
- `factory-test.go`: Tests that a factory only creates a new exporter (currently a stub) if provided with a valid configuration.

**Documentation:**
Added documentation for the purpose of the exporter and all its current configuration options. This includes Humio endpoints, ingest tokens, and storage options for Humio.

Also added examples of configuring the exporter at various degrees of detail.

Lastly, I added references to configuration options inherited from:
- [HTTP Client Configuration](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#client-configuration)
- [TLS Configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings)
- [Queueing, Retry, and Timeout Configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md#configuration)
  • Loading branch information
Xitric authored and pmatyjasek-sumo committed Apr 28, 2021
1 parent b617ef8 commit 577dd14
Show file tree
Hide file tree
Showing 11 changed files with 2,333 additions and 0 deletions.
1 change: 1 addition & 0 deletions exporter/humioexporter/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
60 changes: 60 additions & 0 deletions exporter/humioexporter/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Humio Exporter
Exports data to Humio using JSON over the HTTP [Ingest API](https://docs.humio.com/reference/api/ingest/).

Supported pipeline types: traces (with metrics and logs to follow soon)

> :construction: This exporter is currently intended for evaluation purposes only!
## Getting Started
This exporter requires the following configuration options:

- `ingest_token` (no default): The token that has been issued in relation to the Humio repository to export data into. This token grants write-only access to a single, specific Humio repository. See [Ingest Tokens](https://docs.humio.com/docs/ingesting-data/ingest-tokens/) for more details.
- `endpoint` (no default): The base URL on which the Humio backend can be reached, in the form `host:port`. For testing this locally with the Humio Docker image, the endpoint could be `http://localhost:8080/`. For use with the Humio cloud, the URLs are as follows, both of which use port `80`:
- EU: `https://cloud.humio.com/`
- US: `https://cloud.us.humio.com/`

As defined in the [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings), TLS is enabled by default. This can be disabled by overriding the following configuration options:

- `insecure` (default: `false`): Whether to enable client transport security for the exporter's HTTP connection. Not recommended for production deployments.
- `insecure_skip_verify` (default: `false`): Whether to skip verifying the server's certificate chain or not. Not recommended for production deployments.

In addition, the following global configuration options can be overridden:

- `tags` (no default): A series of key-value pairs used to target specific Data Sources for storage inside a Humio repository. Refer to [Humio Tagging](https://docs.humio.com/docs/parsers/tagging/) for more details.
- `disable_service_tag` (default: `false`): By default, the service name will be used to tag all exported events in addition to user-provided tags. If disabled, only the user-provided tags will be used. However, at least one tag _must_ be specified.

### Traces
For exporting structured data (traces), the following configuration options are available:

- `unix_timestamps` (default: `false`): Whether to use Unix or ISO 8601 formatted timestamps when exporting data to Humio. If this is set to `true`, timestamps will be represented in milliseconds (Unix time) in UTC.
- `timezone` (default: host timezone): When using Unix timestamps, this option can be provided to specify the local timezone of the events. If not specified, the local time zone of the host is used instead. An example is `Europe/Copenhagen`.

## Advaced Configuration
This exporter, like many others, includes shared configuration helpers for the following advanced settings:

- [HTTP Client Configuration](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp#client-configuration)
- [TLS Configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings)
- [Queueing, Retry, and Timeout Configuration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md#configuration)

## Example Configuration
Below are two examples of configurations specific to this exporter. For a more advanced example with all available configuration options, see [This Example](testdata/config.yaml).

```yaml
exporters:
humio:
ingest_token: "00000000-0000-0000-0000-0000000000000"
endpoint: "https://my-humio-host:8080"
humio/advanced:
ingest_token: "00000000-0000-0000-0000-0000000000000"
endpoint: "http://localhost:8080"
timeout: 10s
disable_service_tag: true
tags:
host: "web_server"
environment: "production"
logs:
log_parser: "custom-parser"
traces:
unix_timestamps: true
timezone: "Europe/Copenhagen"
```
146 changes: 146 additions & 0 deletions exporter/humioexporter/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package humioexporter

import (
"errors"
"net/url"
"path"

"go.opentelemetry.io/collector/config"
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/exporter/exporterhelper"
)

const (
basePath = "api/v1/ingest/"
unstructuredPath = basePath + "humio-unstructured"
structuredPath = basePath + "humio-structured"
)

// LogsConfig represents the Humio configuration settings specific to logs
type LogsConfig struct {
// The name of a custom log parser to use, if no parser is associated with the ingest token
LogParser string `mapstructure:"log_parser"`
}

// TracesConfig represents the Humio configuration settings specific to traces
type TracesConfig struct {
// Whether to use Unix timestamps, or to fall back to ISO 8601 formatted strings
UnixTimestamps bool `mapstructure:"unix_timestamps"`

// The time zone to use when representing timestamps in Unix time
TimeZone string `mapstructure:"timezone"`
}

// Config represents the Humio configuration settings
type Config struct {
// Inherited settings
*config.ExporterSettings `mapstructure:"-"`
confighttp.HTTPClientSettings `mapstructure:",squash"`
exporterhelper.QueueSettings `mapstructure:"sending_queue"`
exporterhelper.RetrySettings `mapstructure:"retry_on_failure"`

//Ingest token for identifying and authorizing with a Humio repository
IngestToken string `mapstructure:"ingest_token"`

// Endpoint for the unstructured ingest API, created internally
unstructuredEndpoint *url.URL

// Endpoint for the structured ingest API, created internally
structuredEndpoint *url.URL

// Key-value pairs used to target specific data sources for storage inside Humio
Tags map[string]string `mapstructure:"tags,omitempty"`

// Whether this exporter should automatically add the service name as a tag
DisableServiceTag bool `mapstructure:"disable_service_tag"`

// Configuration options specific to logs
Logs LogsConfig `mapstructure:"logs"`

// Configuration options specific to traces
Traces TracesConfig `mapstructure:"traces"`
}

// Validate ensures that a valid configuration has been provided, such that we can fail early
func (c *Config) Validate() error {
if c.IngestToken == "" {
return errors.New("requires an ingest_token")
}

if c.Endpoint == "" {
return errors.New("requires an endpoint")
}

if c.DisableServiceTag && len(c.Tags) == 0 {
return errors.New("requires at least one custom tag when disabling service tag")
}

if c.Traces.UnixTimestamps && c.Traces.TimeZone == "" {
return errors.New("requires a time zone when using Unix timestamps")
}

// Ensure that it is possible to construct a URL to access the unstructured ingest API
if c.unstructuredEndpoint == nil {
endp, err := c.getEndpoint(unstructuredPath)
if err != nil {
return errors.New("unable to create URL for unstructured ingest API")
}
c.unstructuredEndpoint = endp
}

// Ensure that it is possible to construct a URL to access the structured ingest API
if c.structuredEndpoint == nil {
endp, err := c.getEndpoint(structuredPath)
if err != nil {
return errors.New("unable to create URL for structured ingest API")
}
c.structuredEndpoint = endp
}

if c.Headers == nil {
c.Headers = make(map[string]string)
}

// We require these headers, which should not be overwritten by the user
if contentType, ok := c.Headers["Content-Type"]; ok && contentType != "application/json" {
return errors.New("the Content-Type must be application/json, which is also the default for this header")
}
c.Headers["Content-Type"] = "application/json"

if _, ok := c.Headers["Authorization"]; ok {
return errors.New("the Authorization header must not be overwritten, since it is automatically generated from the ingest token")
}
c.Headers["Authorization"] = "Bearer " + c.IngestToken

// Fallback User-Agent if not overridden by user
if _, ok := c.Headers["User-Agent"]; !ok {
c.Headers["User-Agent"] = "opentelemetry-collector-contrib Humio"
}

return nil
}

// Get a URL for a specific destination path on the Humio endpoint
func (c *Config) getEndpoint(dest string) (*url.URL, error) {
res, err := url.Parse(c.Endpoint)
if err != nil {
return res, err
}

res.Path = path.Join(res.Path, dest)
return res, nil
}
Loading

0 comments on commit 577dd14

Please sign in to comment.