[exporter/mezmoexporter] Add the Mezmo Log exporter (#9743)

* Added implementation of Mezmo Log exporter.

* Additional files to prepare for PR to contrib

* Initial revision

* Changes to support batching of log uploads to not exceed maximum allowed POST size

* Update exporter/mezmoexporter/exporter.go

Slimmed code down to be more compact.

Co-authored-by: David Ashpole <dashpole@google.com>

* Update exporter/mezmoexporter/exporter.go

Co-authored-by: David Ashpole <dashpole@google.com>

* Update exporter/mezmoexporter/exporter.go

Co-authored-by: David Ashpole <dashpole@google.com>

* Update exporter/mezmoexporter/exporter.go

Co-authored-by: David Ashpole <dashpole@google.com>

* Update exporter/mezmoexporter/exporter.go

Co-authored-by: David Ashpole <dashpole@google.com>

* Update exporter/mezmoexporter/exporter.go

Co-authored-by: David Ashpole <dashpole@google.com>

* Added the appropriate code owners the the Mezmo Exporter

* Moved test functions to test file

* Cleaned up README and standardized URLs across examples

* Cleanup to remove commented code and non-exported functions

* Fixed a typo

* Added TODO entry for removing hostname from the HTTP POST when it's no longer required by the backend

* Removed year from the Copyright header

* Fixed a typo in the mezmo exporter definition

* Made the ingest_url an optional configuration with a known default value.

* Changes to enable a default ingest URL as well as allow an override from the config.

* Updates from rm -fr go.sum
go mod tidy -compat=1.17

* Added new Mezmo Exporter component

* Added return value checks for some tests

* Removed some unneeded code from the last merge

* Reverted the release version to unreleased

* Proper version for the new component

* Applied make goporto updates

* Fixed a unit test case for the default config test

* Bumped the version number for the go.opentelemetry.io/collector versions

* Module updates for latest release

* Removed the newrelicexporter as it has been removed

* Updates from the latest make gotidy for the merge to main

* Pulled semconv from go.opentelemetry.io/collector/semconv instead of go.opentelemetry.io/collector/semconv/model

* Moved the mezmo new component under Unreleased and fixed the associated PR number

Co-authored-by: David Ashpole <dashpole@google.com>

.github/CODEOWNERS

@@ -43,6 +43,7 @@ exporter/kafkaexporter/                              @open-telemetry/collector-c
 exporter/loadbalancingexporter/                      @open-telemetry/collector-contrib-approvers @jpkrohling
 exporter/logzioexporter/                             @open-telemetry/collector-contrib-approvers @jkowall @Doron-Bargo @yotamloe
 exporter/lokiexporter/                               @open-telemetry/collector-contrib-approvers @gramidt @jpkrohling
+exporter/mezmoexporter/                              @open-telemetry/collector-contrib-approvers @dashpole @billmeyer @gjanco
 exporter/observiqexporter/                           @open-telemetry/collector-contrib-approvers @binaryfissiongames
 exporter/prometheusexporter/                         @open-telemetry/collector-contrib-approvers @Aneurysm9
 exporter/prometheusremotewriteexporter/              @open-telemetry/collector-contrib-approvers @Aneurysm9

CHANGELOG.md

@@ -10,6 +10,7 @@
 
 - `bigipreceiver`: Add implementation of F5 Big-IP Metric Receiver (#9680)
 - `expvarreceiver`: Initial work for a receiver designed to scrape `memstats` from Golang applications. (#9747)
+- `mezmoexporter`: Add implementation of Mezmo Log exporter (#9743)
 - `nsxtreceiver`: Added implementation of NSX-T Metric Receiver (#9568)
 
 ### 💡 Enhancements 💡

cmd/configschema/go.mod

@@ -257,6 +257,7 @@ require (
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter v0.51.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/logzioexporter v0.51.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter v0.51.0 // indirect
+	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/mezmoexporter v0.0.0-00010101000000-000000000000 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opencensusexporter v0.51.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/parquetexporter v0.51.0 // indirect
 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.51.0 // indirect
@@ -607,6 +608,8 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/logzi
 
 replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter => ../../exporter/lokiexporter
 
+replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/mezmoexporter => ../../exporter/mezmoexporter
+
 replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opencensusexporter => ../../exporter/opencensusexporter
 
 replace github.com/open-telemetry/opentelemetry-collector-contrib/exporter/parquetexporter => ../../exporter/parquetexporter

exporter/mezmoexporter/Makefile

@@ -0,0 +1 @@
+include ../../Makefile.Common

exporter/mezmoexporter/README.md

@@ -0,0 +1,30 @@
+# Mezmo Exporter
+
+This exporter supports sending OpenTelemetry log data to [LogDNA (Mezmo)](https://logdna.com).
+
+# Configuration options:
+
+- `ingest_url` (optional): Specifies the URL to send ingested logs to.  If not specified, will default to `https://logs.logdna.com/log/ingest`.
+- `ingest_key` (required): Ingestion key used to send log data to LogDNA.  See [Ingestion Keys](https://docs.logdna.com/docs/ingestion-key) for more details.
+
+# Example:
+## Simple Log Data
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: ":4317"
+
+exporters:
+  mezmo:
+    ingest_url: "https://logs.logdna.com/log/ingest"
+    ingest_key: "00000000000000000000000000000000"
+
+service:
+  pipelines:
+    logs:
+      receivers: [ otlp ]
+      exporters: [ mezmo ]
+```

exporter/mezmoexporter/config.go

@@ -0,0 +1,79 @@
+// Copyright  The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mezmoexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/mezmoexporter"
+
+import (
+	"fmt"
+	"net/url"
+	"time"
+
+	"go.opentelemetry.io/collector/config"
+	"go.opentelemetry.io/collector/config/confighttp"
+	"go.opentelemetry.io/collector/exporter/exporterhelper"
+)
+
+const (
+	// defaultTimeout
+	defaultTimeout time.Duration = 5 * time.Second
+
+	// defaultIngestURL
+	defaultIngestURL = "https://logs.logdna.com/log/ingest"
+
+	// See https://docs.logdna.com/docs/ingestion#service-limits for details
+
+	// Maximum payload in bytes that can be POST'd to the REST endpoint
+	maxBodySize     = 10 * 1024 * 1024
+	maxMessageSize  = 16 * 1024
+	maxMetaDataSize = 32 * 1024
+	maxAppnameLen   = 512
+	maxLogLevelLen  = 80
+)
+
+// Config defines configuration for Mezmo exporter.
+type Config struct {
+	config.ExporterSettings       `mapstructure:",squash"`
+	confighttp.HTTPClientSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
+	exporterhelper.QueueSettings  `mapstructure:"sending_queue"`
+	exporterhelper.RetrySettings  `mapstructure:"retry_on_failure"`
+
+	// IngestURL is the URL to send telemetry to.
+	IngestURL string `mapstructure:"ingest_url"`
+
+	// Token is the authentication token provided by Mezmo.
+	IngestKey string `mapstructure:"ingest_key"`
+}
+
+// returns default http client settings
+func createDefaultHTTPClientSettings() confighttp.HTTPClientSettings {
+	return confighttp.HTTPClientSettings{
+		Timeout: defaultTimeout,
+	}
+}
+
+func (c *Config) Validate() error {
+	var err error
+	var parsed *url.URL
+
+	parsed, err = url.Parse(c.IngestURL)
+	if c.IngestURL == "" || err != nil {
+		return fmt.Errorf("\"ingest_url\" must be a valid URL")
+	}
+
+	if parsed.Host == "" {
+		return fmt.Errorf("\"ingest_url\" must contain a valid host")
+	}
+
+	return nil
+}

exporter/mezmoexporter/config_test.go

@@ -0,0 +1,87 @@
+// Copyright  The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package mezmoexporter
+
+import (
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"go.opentelemetry.io/collector/component/componenttest"
+	"go.opentelemetry.io/collector/config"
+	"go.opentelemetry.io/collector/config/confighttp"
+	"go.opentelemetry.io/collector/exporter/exporterhelper"
+	"go.opentelemetry.io/collector/service/servicetest"
+)
+
+func TestLoadDefaultConfig(t *testing.T) {
+	factories, err := componenttest.NopFactories()
+	assert.Nil(t, err)
+
+	factory := NewFactory()
+	factories.Exporters[typeStr] = factory
+	cfg, err := servicetest.LoadConfigAndValidate(filepath.Join("testdata", "config.yaml"), factories)
+
+	require.NoError(t, err)
+	require.NotNil(t, cfg)
+
+	// Verify the "default"/required values-only configuration
+	e := cfg.Exporters[config.NewComponentID(typeStr)]
+
+	// Our expected default configuration should use the defaultIngestURL
+	defaultCfg := factory.CreateDefaultConfig().(*Config)
+	defaultCfg.IngestURL = defaultIngestURL
+	defaultCfg.IngestKey = "00000000000000000000000000000000"
+	assert.Equal(t, defaultCfg, e)
+}
+
+func TestLoadAllSettingsConfig(t *testing.T) {
+	factories, err := componenttest.NopFactories()
+	assert.Nil(t, err)
+
+	factory := NewFactory()
+	factories.Exporters[typeStr] = factory
+	cfg, err := servicetest.LoadConfigAndValidate(filepath.Join("testdata", "config.yaml"), factories)
+
+	require.NoError(t, err)
+	require.NotNil(t, cfg)
+
+	// Verify values from the config override the default configuration
+	e := cfg.Exporters[config.NewComponentIDWithName(typeStr, "allsettings")]
+
+	// Our expected default configuration should use the defaultIngestURL
+	expectedCfg := Config{
+		ExporterSettings: config.NewExporterSettings(config.NewComponentIDWithName(typeStr, "allsettings")),
+		HTTPClientSettings: confighttp.HTTPClientSettings{
+			Timeout: 5 * time.Second,
+		},
+		RetrySettings: exporterhelper.RetrySettings{
+			Enabled:         false,
+			InitialInterval: 99 * time.Second,
+			MaxInterval:     199 * time.Second,
+			MaxElapsedTime:  299 * time.Minute,
+		},
+		QueueSettings: exporterhelper.QueueSettings{
+			Enabled:      false,
+			NumConsumers: 7,
+			QueueSize:    17,
+		},
+		IngestURL: "https://alternate.logdna.com/log/ingest",
+		IngestKey: "1234509876",
+	}
+	assert.Equal(t, &expectedCfg, e)
+}

exporter/mezmoexporter/doc.go

@@ -0,0 +1,16 @@
+// Copyright  The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package mezmoexporter implements an exporter that sends data to Mezmo.
+package mezmoexporter // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/mezmoexporter"

exporter/mezmoexporter/example/otel-collector-config.yaml

@@ -0,0 +1,17 @@
+receivers:
+  nop:
+
+processors:
+  nop:
+
+exporters:
+  mezmo:
+    ingest_url: "https://logs.logdna.com/log/ingest"
+    ingest_key: "00000000000000000000000000000000"
+
+service:
+  pipelines:
+    logs:
+      receivers: [ nop ]
+      processors: [ nop ]
+      exporters: [ mezmo ]