Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Profiles Signal #10154

Closed
wants to merge 50 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
7c2fc48
chore: adds basic profiling support
petethepig Jun 14, 2023
da10c9b
uses an empty profile proto
petethepig Jun 14, 2023
5a41d99
fixes
petethepig Jun 14, 2023
7a00f04
wip
petethepig Jun 14, 2023
628905a
wip
petethepig Jun 14, 2023
8fec712
wip
petethepig Jun 14, 2023
9e56312
wip
petethepig Jun 15, 2023
82cbb23
wip
petethepig Jun 15, 2023
5b0febd
wip
petethepig Jun 20, 2023
f62e78b
wip
petethepig Jun 20, 2023
d180b5a
wip
petethepig Jun 20, 2023
897f657
wip
petethepig Jun 20, 2023
5bf5ffc
improvements
petethepig Jul 26, 2023
fb7fc48
adds benchmarking code
petethepig Jul 26, 2023
5eb4b0f
otlp fix
petethepig Jul 27, 2023
2e4260d
conversion + benchmark code
petethepig Jul 27, 2023
9c7cf43
adds a dockerized example
petethepig Jul 27, 2023
233adf3
jfr and ruby benchmarks
petethepig Aug 3, 2023
cf5ddf9
more benchmarks
petethepig Aug 3, 2023
b0977ad
more benchmarking, adds some examples
petethepig Aug 24, 2023
4d5897a
improvements
petethepig Aug 24, 2023
40bc740
pprofextended snapshot
petethepig Oct 5, 2023
ed1960f
improvements
petethepig Oct 5, 2023
cfdffe1
separate array for locations
petethepig Oct 14, 2023
cd9041f
more bechmarking
petethepig Oct 19, 2023
8ba53ee
panic fix
petethepig Oct 20, 2023
978fe04
Merge pull request #1 from petethepig/profiling
petethepig Oct 23, 2023
37c1031
pre-otep benchmarks
petethepig Nov 2, 2023
732fa3b
fixes
petethepig Nov 2, 2023
2b7545b
profiles otep v2
petethepig May 10, 2024
4f99342
Merge branch 'main' into v2
petethepig May 10, 2024
c3e59dc
compiles
petethepig May 10, 2024
f7fa382
test fix
petethepig May 10, 2024
a0df479
support for state in Profiles
petethepig May 10, 2024
0a6ff79
Refactor test setup in profiles_test.go and otlp_test.go
petethepig May 10, 2024
18c7791
updates
petethepig May 13, 2024
b630531
disabling some tests
petethepig May 13, 2024
f01f0de
test fix
petethepig May 13, 2024
11531b3
fixes
petethepig May 13, 2024
37829a9
wip
petethepig May 14, 2024
771a93b
rollbacks
petethepig May 14, 2024
08c8a6c
changes
petethepig May 15, 2024
0ad3c17
fixes
petethepig May 15, 2024
f06f113
fixes
petethepig May 15, 2024
7283c29
improvements
petethepig May 15, 2024
e8c82b0
fixes
petethepig May 15, 2024
5700f5a
removes debug logging
petethepig May 15, 2024
de9ed8d
removes old code
petethepig May 15, 2024
18f418c
rollback mdatagen changes
petethepig May 15, 2024
cd9da6c
chore: Update stability level for profiles processor
petethepig May 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ OPENTELEMETRY_PROTO_SRC_DIR=pdata/internal/opentelemetry-proto
OPENTELEMETRY_PROTO_VERSION=v1.2.0

# Find all .proto files.
OPENTELEMETRY_PROTO_FILES := $(subst $(OPENTELEMETRY_PROTO_SRC_DIR)/,,$(wildcard $(OPENTELEMETRY_PROTO_SRC_DIR)/opentelemetry/proto/*/v1/*.proto $(OPENTELEMETRY_PROTO_SRC_DIR)/opentelemetry/proto/collector/*/v1/*.proto))
OPENTELEMETRY_PROTO_FILES := $(subst $(OPENTELEMETRY_PROTO_SRC_DIR)/,,$(wildcard $(OPENTELEMETRY_PROTO_SRC_DIR)/opentelemetry/proto/*/v1/*.proto $(OPENTELEMETRY_PROTO_SRC_DIR)/opentelemetry/proto/collector/*/v1/*.proto $(OPENTELEMETRY_PROTO_SRC_DIR)/opentelemetry/proto/*/v1experimental/*.proto $(OPENTELEMETRY_PROTO_SRC_DIR)/opentelemetry/proto/collector/*/v1experimental/*.proto))

# Target directory to write generated files to.
PROTO_TARGET_GEN_DIR=pdata/internal/data/protogen
Expand Down Expand Up @@ -206,6 +206,8 @@ genproto_sub:
# HACK: Workaround for istio 1.15 / envoy 1.23.1 mistakenly emitting deprecated field.
# reserved 1000 -> repeated ScopeLogs deprecated_scope_logs = 1000;
sed 's/reserved 1000;/repeated ScopeLogs deprecated_scope_logs = 1000;/g' $(PROTO_INTERMEDIATE_DIR)/opentelemetry/proto/logs/v1/logs.proto 1<> $(PROTO_INTERMEDIATE_DIR)/opentelemetry/proto/logs/v1/logs.proto
# reserved 1000 -> repeated ScopeProfiles deprecated_scope_profiles = 1000;
sed 's/reserved 1000;/repeated ScopeProfiles deprecated_scope_profiles = 1000;/g' $(PROTO_INTERMEDIATE_DIR)/opentelemetry/proto/profiles/v1experimental/profiles.proto 1<> $(PROTO_INTERMEDIATE_DIR)/opentelemetry/proto/profiles/v1experimental/profiles.proto
# reserved 1000 -> repeated ScopeMetrics deprecated_scope_metrics = 1000;
sed 's/reserved 1000;/repeated ScopeMetrics deprecated_scope_metrics = 1000;/g' $(PROTO_INTERMEDIATE_DIR)/opentelemetry/proto/metrics/v1/metrics.proto 1<> $(PROTO_INTERMEDIATE_DIR)/opentelemetry/proto/metrics/v1/metrics.proto
# reserved 1000 -> repeated ScopeSpans deprecated_scope_spans = 1000;
Expand Down
22 changes: 22 additions & 0 deletions component/componenttest/obsreporttest.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ func (tts *TestTelemetry) CheckExporterEnqueueFailedLogs(enqueueFailed int64) er
return tts.prometheusChecker.checkExporterEnqueueFailed(tts.id, "log_records", enqueueFailed)
}

func (tts *TestTelemetry) CheckExporterEnqueueFailedProfiles(enqueueFailed int64) error {
return tts.prometheusChecker.checkExporterEnqueueFailed(tts.id, "profiles", enqueueFailed)
}

// CheckExporterLogs checks that for the current exported values for logs exporter metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckExporterLogs(sentLogRecords, sendFailedLogRecords int64) error {
Expand All @@ -76,6 +80,12 @@ func (tts *TestTelemetry) CheckExporterMetricGauge(metric string, val int64) err
return tts.prometheusChecker.checkExporterMetricGauge(tts.id, metric, val)
}

// CheckExporterProfiles checks that for the current exported values for profiles exporter metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckExporterProfiles(sentProfiles, sendFailedProfiles int64) error {
return tts.prometheusChecker.checkExporterProfiles(tts.id, sentProfiles, sendFailedProfiles)
}

// CheckProcessorTraces checks that for the current exported values for trace exporter metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckProcessorTraces(acceptedSpans, refusedSpans, droppedSpans int64) error {
Expand All @@ -94,6 +104,12 @@ func (tts *TestTelemetry) CheckProcessorLogs(acceptedLogRecords, refusedLogRecor
return tts.prometheusChecker.checkProcessorLogs(tts.id, acceptedLogRecords, refusedLogRecords, droppedLogRecords)
}

// CheckProcessorProfiles checks that for the current exported values for profiles exporter metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckProcessorProfiles(acceptedProfiles, refusedProfiles, droppedProfiles int64) error {
return tts.prometheusChecker.checkProcessorProfiles(tts.id, acceptedProfiles, refusedProfiles, droppedProfiles)
}

// CheckReceiverTraces checks that for the current exported values for trace receiver metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckReceiverTraces(protocol string, acceptedSpans, droppedSpans int64) error {
Expand All @@ -106,6 +122,12 @@ func (tts *TestTelemetry) CheckReceiverLogs(protocol string, acceptedLogRecords,
return tts.prometheusChecker.checkReceiverLogs(tts.id, protocol, acceptedLogRecords, droppedLogRecords)
}

// CheckReceiverProfiles checks that for the current exported values for profiles receiver metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckReceiverProfiles(protocol string, acceptedProfiles, droppedProfiles int64) error {
return tts.prometheusChecker.checkReceiverProfiles(tts.id, protocol, acceptedProfiles, droppedProfiles)
}

// CheckReceiverMetrics checks that for the current exported values for metrics receiver metrics match given values.
// When this function is called it is required to also call SetupTelemetry as first thing.
func (tts *TestTelemetry) CheckReceiverMetrics(protocol string, acceptedMetricPoints, droppedMetricPoints int64) error {
Expand Down
12 changes: 12 additions & 0 deletions component/componenttest/otelprometheuschecker.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ func (pc *prometheusChecker) checkReceiverMetrics(receiver component.ID, protoco
return pc.checkReceiver(receiver, "metric_points", protocol, accepted, dropped)
}

func (pc *prometheusChecker) checkReceiverProfiles(receiver component.ID, protocol string, accepted, dropped int64) error {
return pc.checkReceiver(receiver, "profiles", protocol, accepted, dropped)
}

func (pc *prometheusChecker) checkReceiver(receiver component.ID, datatype, protocol string, acceptedMetricPoints, droppedMetricPoints int64) error {
receiverAttrs := attributesForReceiverMetrics(receiver, protocol)
return multierr.Combine(
Expand All @@ -60,6 +64,10 @@ func (pc *prometheusChecker) checkProcessorLogs(processor component.ID, accepted
return pc.checkProcessor(processor, "log_records", accepted, refused, dropped)
}

func (pc *prometheusChecker) checkProcessorProfiles(processor component.ID, accepted, refused, dropped int64) error {
return pc.checkProcessor(processor, "profiles", accepted, refused, dropped)
}

func (pc *prometheusChecker) checkProcessor(processor component.ID, datatype string, accepted, refused, dropped int64) error {
processorAttrs := attributesForProcessorMetrics(processor)
return multierr.Combine(
Expand All @@ -80,6 +88,10 @@ func (pc *prometheusChecker) checkExporterMetrics(exporter component.ID, sent, s
return pc.checkExporter(exporter, "metric_points", sent, sendFailed)
}

func (pc *prometheusChecker) checkExporterProfiles(exporter component.ID, sent, sendFailed int64) error {
return pc.checkExporter(exporter, "profiles", sent, sendFailed)
}

func (pc *prometheusChecker) checkExporter(exporter component.ID, datatype string, sent, sendFailed int64) error {
exporterAttrs := attributesForExporterMetrics(exporter)
errs := pc.checkCounter(fmt.Sprintf("exporter_sent_%s", datatype), sent, exporterAttrs)
Expand Down
21 changes: 21 additions & 0 deletions component/componenttest/testdata/prometheus_response
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ exporter_send_failed_log_records{exporter="fakeExporter"} 36
# HELP exporter_sent_log_records Number of logs successfully sent to destination.
# TYPE exporter_sent_log_records counter
exporter_sent_log_records{exporter="fakeExporter"} 103
# HELP exporter_send_failed_profiles Number of logs in failed attempts to send to destination.
# TYPE exporter_send_failed_profiles counter
exporter_send_failed_profiles{exporter="fakeExporter"} 32
# HELP exporter_sent_profiles Number of logs successfully sent to destination.
# TYPE exporter_sent_profiles counter
exporter_sent_profiles{exporter="fakeExporter"} 106
# HELP processor_accepted_spans Number of spans successfully pushed into the next component in the pipeline.
# TYPE processor_accepted_spans counter
processor_accepted_spans{processor="fakeProcessor"} 42
Expand Down Expand Up @@ -46,6 +52,18 @@ processor_dropped_log_records{processor="fakeProcessor"} 14
# HELP receiver_accepted_log_records Number of log records successfully pushed into the pipeline.
# TYPE receiver_accepted_log_records counter
receiver_accepted_log_records{receiver="fakeReceiver",transport="fakeTransport"} 102
# HELP processor_accepted_profiles Number of log records successfully pushed into the next component in the pipeline.
# TYPE processor_accepted_profiles counter
processor_accepted_profiles{processor="fakeProcessor"} 102
# HELP processor_refused_profiles Number of log records that were rejected by the next component in the pipeline.
# TYPE processor_refused_profiles counter
processor_refused_profiles{processor="fakeProcessor"} 35
# HELP processor_dropped_profiles Number of log records that were dropped.
# TYPE processor_dropped_profiles counter
processor_dropped_profiles{processor="fakeProcessor"} 14
# HELP receiver_accepted_profiles Number of log records successfully pushed into the pipeline.
# TYPE receiver_accepted_profiles counter
receiver_accepted_profiles{receiver="fakeReceiver",transport="fakeTransport"} 102
# HELP receiver_accepted_metric_points Number of metric points successfully pushed into the pipeline.
# TYPE receiver_accepted_metric_points counter
receiver_accepted_metric_points{receiver="fakeReceiver",transport="fakeTransport"} 7
Expand All @@ -55,6 +73,9 @@ receiver_accepted_spans{receiver="fakeReceiver",transport="fakeTransport"} 42
# HELP receiver_refused_log_records Number of log records that could not be pushed into the pipeline.
# TYPE receiver_refused_log_records counter
receiver_refused_log_records{receiver="fakeReceiver",transport="fakeTransport"} 35
# HELP receiver_refused_profiles Number of log records that could not be pushed into the pipeline.
# TYPE receiver_refused_profiles counter
receiver_refused_profiles{receiver="fakeReceiver",transport="fakeTransport"} 35
# HELP receiver_refused_metric_points Number of metric points that could not be pushed into the pipeline.
# TYPE receiver_refused_metric_points counter
receiver_refused_metric_points{receiver="fakeReceiver",transport="fakeTransport"} 41
Expand Down
3 changes: 3 additions & 0 deletions component/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -176,4 +176,7 @@ var (

// DataTypeLogs is the data type tag for logs.
DataTypeLogs = mustNewDataType("logs")

// DataTypeProfiles is the data type tag for profiles.
DataTypeProfiles = mustNewDataType("profiles")
)
Loading
Loading