Add performance test for filelog based kubernetes container logs receiver #2564

sumo-drosiek · 2021-03-05T15:02:43Z

Add performance test for filelog based kubernetes container logs receiver

This PR contains one performance test (for kubernetes containerd logs with format autodetection)

Link to tracking Issue:
#2266

Testing:
Perf tests

Documentation:
N/A

codecov · 2021-03-05T15:17:28Z

Codecov Report

Merging #2564 (c945b11) into main (a7fbb60) will increase coverage by 22.13%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #2564       +/-   ##
===========================================
+ Coverage   69.18%   91.31%   +22.13%     
===========================================
  Files          34      429      +395     
  Lines        1603    21457    +19854     
===========================================
+ Hits         1109    19594    +18485     
- Misses        420     1394      +974     
- Partials       74      469      +395

Flag	Coverage Δ
integration	`69.18% <ø> (ø)`
unit	`90.22% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
receiver/receivercreator/receiver.go	`66.66% <0.00%> (ø)`
...cedetectionprocessor/internal/resourcedetection.go	`97.05% <0.00%> (ø)`
...nalfxexporter/translation/logdata_to_signalfxv2.go	`82.75% <0.00%> (ø)`
exporter/dynatraceexporter/metrics_exporter.go	`86.27% <0.00%> (ø)`
receiver/memcachedreceiver/scraper.go	`15.15% <0.00%> (ø)`
exporter/awsxrayexporter/factory.go	`100.00% <0.00%> (ø)`
receiver/kubeletstatsreceiver/kubelet/cert.go	`63.63% <0.00%> (ø)`
...ctionprocessor/internal/aws/elasticbeanstalk/fs.go	`0.00% <0.00%> (ø)`
...er/awsemfexporter/mapwithexpiry/map_with_expiry.go	`100.00% <0.00%> (ø)`
...ver/k8sclusterreceiver/collection/metadatastore.go	`0.00% <0.00%> (ø)`
... and 407 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7fbb60...c945b11. Read the comment docs.

testbed/datasenders/k8s.go

tigrannajaryan · 2021-03-05T15:15:12Z

testbed/datasenders/k8s.go

+
+// NewFileLogK8sWriter creates a new data sender that will write kubernetes containerd
+// log entries to a file, to be tailed by FileLogReceiver and sent to the collector.
+func NewFileLogK8sWriter(config string) *FileLogK8sWriter {


Add comment to explain what config must contain.

tigrannajaryan · 2021-03-05T15:18:15Z

testbed/tests_unstable_exe/log_test.go

@@ -31,4 +34,42 @@ func TestMain(m *testing.M) {
 	testbed.DoTestMain(m, contribPerfResultsSummary)
 }

-// This file is left with no tests as a placeholder for future unstable exe tests
+func TestLog10kDPS(t *testing.T) {


Can we move this to the stable log_test.go instead?

sure, I was not sure what exactly unstable means

The tests_unstable_exe/log_test.go is for testing the unstable executable.
Stanza/filelog receiver previously was part of unstable executable. It was moved to stable exe yesterday.

sumo-drosiek · 2021-03-05T16:20:08Z

@tigrannajaryan All fixed, but Ci seems to be unstable Today

tigrannajaryan · 2021-03-05T17:44:48Z

I think there is a linting error (in addition to failing load test, which is probably CircleCI fault).

sumo-drosiek · 2021-03-08T06:37:32Z

@tigrannajaryan I doubt this lint error related to this PR, as all recent PRs have same issue eg. https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib/10631/workflows/5e23abf9-f769-4d99-908e-c046ea55bbda/jobs/89573

make[1]: Entering directory '/home/circleci/project'
golangci-lint run --allow-parallel-runners
ERRO Running error: context loading failed: no go files to analyze 
make[1]: *** [Makefile.Common:102: lint] Error 5

sumo-drosiek · 2021-03-08T14:07:12Z

@tigrannajaryan I added more tests in this PR and example is available in PR #2606

tigrannajaryan · 2021-03-08T16:01:41Z

@tigrannajaryan I doubt this lint error related to this PR, as all recent PRs have same issue eg. https://app.circleci.com/pipelines/github/open-telemetry/opentelemetry-collector-contrib/10631/workflows/5e23abf9-f769-4d99-908e-c046ea55bbda/jobs/89573
make[1]: Entering directory '/home/circleci/project'
golangci-lint run --allow-parallel-runners
ERRO Running error: context loading failed: no go files to analyze 
make[1]: *** [Makefile.Common:102: lint] Error 5

Rebasing from main should help. It was broken.

tigrannajaryan · 2021-03-08T16:02:38Z

testbed/datasenders/k8s.go

+// NewFileLogK8sWriter creates a new data sender that will write kubernetes containerd
+// log entries to a file, to be tailed by FileLogReceiver and sent to the collector.
+//
+// config is an OTC config appended to the receivers section after executing fmt.Sprintf on it.


What is OTC? If it refers to collector just use Collector or Otel Collector or Otelcol. We don't normally use OTC.

I will use Otelcol then

testbed/datasenders/k8s.go

djaglowski · 2021-03-08T16:50:31Z

@sumo-drosiek The latest release of opentelemetry-log-collection will require some minor changes to this code.

This should be a very straightforward change from using the term labels to attributes.

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com> Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

sumo-drosiek · 2021-03-08T21:06:46Z

internal/stanza/register.go

 	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/json"
 	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/regex"
 	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/severity"
 	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/time"
+	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/transformer/metadata"
+	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/transformer/restructure"
+	_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/transformer/router"


I was unable to make linter happy with comment above transformer section. At least one of two linters has been always raising some issues:

Check License finished successfully internal/stanza/register.go: Import groups are not in the proper order: ["Third party" "Third party"]

or

register.go:22: File is not `goimports`-ed with -local github.com/open-telemetry/opentelemetry-collector-contrib (goimports) _ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/time"

or

register.go:23: File is not `gofmt`-ed with `-s` (gofmt)

I tried three different approaches

_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/time" // comment here _ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/transformer/metadata"

_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/time" // comment here _ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/transformer/metadata"

_ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/parser/time" // comment here (preceding line is tab indented) _ "github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/transformer/metadata"

No need to split into two groups - looks good to me like this.

sumo-drosiek · 2021-03-08T21:22:27Z

@tigrannajaryan @djaglowski Everything addressed, CI is green :)

djaglowski

LGTM

tigrannajaryan · 2021-03-09T16:03:10Z

Thank you @sumo-drosiek

Ensure that the receiver finishes all ongoing requests, and does not accept any new request after shutdown is finished. Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

**Description:** This PR adds example configuration to handle kubernetes container logs using filelog receiver This depends on the #2564 **Link to tracking Issue:** #2266 **Testing:** N/A tests which were performed to get results are going to be issued in another PR **Documentation:** README.md

…iver (#2564) Add performance test for filelog based kubernetes container logs receiver This PR contains one performance test (for kubernetes containerd logs with format autodetection) **Link to tracking Issue:** #2266

**Description:** This PR adds example configuration to handle kubernetes container logs using filelog receiver This depends on the #2564 **Link to tracking Issue:** #2266 **Testing:** N/A tests which were performed to get results are going to be issued in another PR **Documentation:** README.md

sumo-drosiek requested a review from djaglowski as a code owner March 5, 2021 15:02

sumo-drosiek requested a review from a team March 5, 2021 15:02

github-actions bot assigned jrcamp Mar 5, 2021

sumo-drosiek mentioned this pull request Mar 5, 2021

Add support for collecting Kubernetes pod log file in nodes #2266

Closed

tigrannajaryan reviewed Mar 5, 2021

View reviewed changes

sumo-drosiek force-pushed the drosiek-stanza-transform-ops branch 4 times, most recently from 02a5e52 to 339dc66 Compare March 5, 2021 16:10

jrcamp assigned tigrannajaryan and unassigned jrcamp Mar 5, 2021

sumo-drosiek mentioned this pull request Mar 8, 2021

example: add kubernetes logs example #2606

Merged

tigrannajaryan reviewed Mar 8, 2021

View reviewed changes

sumo-drosiek force-pushed the drosiek-stanza-transform-ops branch 2 times, most recently from 2662e93 to c75f8ac Compare March 8, 2021 20:52

Dominik Rosiek and others added 5 commits March 8, 2021 22:02

internal: stanza: add transformers required by k8s container parser

f026379

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

testbed: add tests for kubernetes containerd logs

2bbe3f3

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com> Co-authored-by: Tigran Najaryan <4194920+tigrannajaryan@users.noreply.github.com>

testbed: logs: rename k8s containerd test to container

1804d62

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

testbed: logs: add k8s cri containerd test

ef406d6

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

testbed: logs: add cri containerd test (no k8s metadata)

c945b11

Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>

sumo-drosiek force-pushed the drosiek-stanza-transform-ops branch from c75f8ac to c945b11 Compare March 8, 2021 21:02

sumo-drosiek commented Mar 8, 2021

View reviewed changes

djaglowski approved these changes Mar 8, 2021

View reviewed changes

tigrannajaryan merged commit ccc3f95 into open-telemetry:main Mar 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add performance test for filelog based kubernetes container logs receiver #2564

Add performance test for filelog based kubernetes container logs receiver #2564

sumo-drosiek commented Mar 5, 2021

codecov bot commented Mar 5, 2021 •

edited

Loading

tigrannajaryan Mar 5, 2021

tigrannajaryan Mar 5, 2021

sumo-drosiek Mar 5, 2021

tigrannajaryan Mar 5, 2021

sumo-drosiek commented Mar 5, 2021

tigrannajaryan commented Mar 5, 2021

sumo-drosiek commented Mar 8, 2021 •

edited

Loading

sumo-drosiek commented Mar 8, 2021

tigrannajaryan commented Mar 8, 2021

tigrannajaryan Mar 8, 2021

sumo-drosiek Mar 8, 2021

djaglowski commented Mar 8, 2021 •

edited

Loading

sumo-drosiek Mar 8, 2021

djaglowski Mar 8, 2021

sumo-drosiek commented Mar 8, 2021

djaglowski left a comment

tigrannajaryan commented Mar 9, 2021

Add performance test for filelog based kubernetes container logs receiver #2564

Add performance test for filelog based kubernetes container logs receiver #2564

Conversation

sumo-drosiek commented Mar 5, 2021

codecov bot commented Mar 5, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sumo-drosiek commented Mar 5, 2021

tigrannajaryan commented Mar 5, 2021

sumo-drosiek commented Mar 8, 2021 • edited Loading

sumo-drosiek commented Mar 8, 2021

tigrannajaryan commented Mar 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djaglowski commented Mar 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sumo-drosiek commented Mar 8, 2021

djaglowski left a comment

Choose a reason for hiding this comment

tigrannajaryan commented Mar 9, 2021

codecov bot commented Mar 5, 2021 •

edited

Loading

sumo-drosiek commented Mar 8, 2021 •

edited

Loading

djaglowski commented Mar 8, 2021 •

edited

Loading