Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BenchmarkExportConntrackConns panics #2984

Closed
antoninbas opened this issue Nov 4, 2021 · 4 comments · Fixed by #2994 or #3004
Closed

BenchmarkExportConntrackConns panics #2984

antoninbas opened this issue Nov 4, 2021 · 4 comments · Fixed by #2994 or #3004
Assignees
Labels
area/flow-visibility/exporter Issues or PRs related to the Flow Exporter functions in the Agent kind/bug Categorizes issue or PR as related to a bug.

Comments

@antoninbas
Copy link
Contributor

Describe the bug
BenchmarkExportConntrackConns is panicking after merging #2854.

2021-11-04T22:34:25.4378372Z I1104 22:34:25.437109    1669 main.go:108] "Parse result" parseSet=map[BenchmarkPoll-2:[BenchmarkPoll-2 679 16542213.00 ns/op 1371489 B/op 53375 allocs/op]]
2021-11-04T22:34:28.2056529Z I1104 22:34:28.199618    1669 main.go:266] "Exec command output" out="goos: linux\ngoarch: amd64\npkg: antrea.io/antrea/pkg/agent/flowexporter/exporter\ncpu: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz\nBenchmarkExportConntrackConns\npanic: runtime error: invalid memory address or nil pointer dereference [recovered]\n\tpanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x13405ff]\n\ngoroutine 8 [running]:\ngithub.com/golang/mock/gomock.(*Controller).finish(0xc0004be300, 0x0, {0x148af60, 0x242cd00})\n\t/home/runner/go/pkg/mod/github.com/golang/mock@v1.6.0/gomock/controller.go:288 +0x30a\ngithub.com/golang/mock/gomock.(*Controller).Finish(0x0)\n\t/home/runner/go/pkg/mod/github.com/golang/mock@v1.6.0/gomock/controller.go:269 +0x38\npanic({0x148af60, 0x242cd00})\n\t/opt/hostedtoolcache/go/1.17.2/x64/src/runtime/panic.go:1038 +0x215\nantrea.io/antrea/pkg/agent/config.IsIPv4Enabled(...)\n\t/home/runner/work/antrea/antrea/pkg/agent/config/node_config.go:158\nantrea.io/antrea/pkg/agent/flowexporter/exporter.NewFlowExporter({0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/runner/work/antrea/antrea/pkg/agent/flowexporter/exporter/exporter.go:165 +0xff\nantrea.io/antrea/pkg/agent/flowexporter/exporter.setupExporter(0x1)\n\t/home/runner/work/antrea/antrea/pkg/agent/flowexporter/exporter/exporter_perf_test.go:163 +0x157\nantrea.io/antrea/pkg/agent/flowexporter/exporter.BenchmarkExportConntrackConns(0xc000126480)\n\t/home/runner/work/antrea/antrea/pkg/agent/flowexporter/exporter/exporter_perf_test.go:85 +0x99\ntesting.(*B).runN(0xc000126480, 0x1)\n\t/opt/hostedtoolcache/go/1.17.2/x64/src/testing/benchmark.go:192 +0x126\ntesting.(*B).run1.func1()\n\t/opt/hostedtoolcache/go/1.17.2/x64/src/testing/benchmark.go:232 +0x59\ncreated by testing.(*B).run1\n\t/opt/hostedtoolcache/go/1.17.2/x64/src/testing/benchmark.go:225 +0xa5\nexit status 2\nFAIL\tantrea.io/antrea/pkg/agent/flowexporter/exporter\t0.035s\nFAIL\n"
2021-11-04T22:34:28.2065996Z I1104 22:34:28.199644    1669 main.go:267] "Exec command error" err=""
2021-11-04T22:34:28.2067237Z I1104 22:34:28.199683    1669 main.go:108] "Parse result" parseSet=map[]
2021-11-04T22:34:28.3533754Z F1104 22:34:28.352983    1669 main.go:60] failed to run a benchmark: failed to run '/opt/hostedtoolcache/go/1.17.2/x64/bin/go test -run '^$' -bench BenchmarkExportConntrackConns -benchtime 10x -timeout 10m -cpu 2 -v -benchmem ./pkg/agent/flowexporter/exporter' command: exit status 1
2021-11-04T22:34:28.3536726Z goroutine 1 [running]:
2021-11-04T22:34:28.3538014Z k8s.io/klog/v2.stacks(0x1)
2021-11-04T22:34:28.3539456Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:1038 +0x8a
2021-11-04T22:34:28.3540892Z k8s.io/klog/v2.(*loggingT).output(0x85eaa0, 0x3, 0x0, 0xc0000cc070, 0x1, {0x71961e, 0x10}, 0xc00015a400, 0x0)
2021-11-04T22:34:28.3542389Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:987 +0x7bd
2021-11-04T22:34:28.3543815Z k8s.io/klog/v2.(*loggingT).printDepth(0xc0000a4120, 0x8e160, 0x0, {0x0, 0x0}, 0x60, {0xc0007e6020, 0x1, 0x1})
2021-11-04T22:34:28.3545316Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:735 +0x1ae
2021-11-04T22:34:28.3546405Z k8s.io/klog/v2.(*loggingT).print(...)
2021-11-04T22:34:28.3547640Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:717
2021-11-04T22:34:28.3548615Z k8s.io/klog/v2.Fatal(...)
2021-11-04T22:34:28.3549712Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:1512
2021-11-04T22:34:28.3551007Z main.main()
2021-11-04T22:34:28.3552545Z 	/home/runner/work/benchci/benchci/main.go:60 +0xe5
2021-11-04T22:34:28.3553280Z 
2021-11-04T22:34:28.3554062Z goroutine 18 [chan receive]:
2021-11-04T22:34:28.3555088Z k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
2021-11-04T22:34:28.3556360Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:1181 +0x6a
2021-11-04T22:34:28.3557493Z created by k8s.io/klog/v2.init.0
2021-11-04T22:34:28.3558674Z 	/home/runner/go/pkg/mod/k8s.io/klog/v2@v2.20.0/klog.go:420 +0xfb

This is causing the Go Benchmark Test workflow to fail on the main branch.

@antoninbas antoninbas added kind/bug Categorizes issue or PR as related to a bug. area/flow-visibility/exporter Issues or PRs related to the Flow Exporter functions in the Agent labels Nov 4, 2021
@antoninbas
Copy link
Contributor Author

@heanlan could you take a look at this?

@wenqiq we should probably run the benchmarks once for each PR (without bench-ci) to make sure that they are not broken, without consuming too many Github Actions minutes. What do you think?

@heanlan
Copy link
Contributor

heanlan commented Nov 4, 2021

@heanlan could you take a look at this?

I'll look into it.

@wenqiq
Copy link
Contributor

wenqiq commented Nov 5, 2021

@wenqiq we should probably run the benchmarks once for each PR (without bench-ci) to make sure that they are not broken, without consuming too many Github Actions minutes. What do you think?

The Go benchmark test workflow has recorded the benchmark error of #2854 : https://github.com/antrea-io/antrea/actions/runs/1423364727

We can avoid this if we run the Go benchmark workflow for each PR, it will take about 17minutes which run the benchmarks twice. I think we can add a step in go.yaml to run the benchmarks, it will only take about 9-10minutes.

@antoninbas
Copy link
Contributor Author

@wenqiq 10 minutes is still pretty large when run every time the PR is updated.
If something likes go test -run=XXX -bench=. -benchtime=1x performs the same job (without the possibility of flakes) and does it much faster, then we should consider doing that instead.

wenqiq added a commit to wenqiq/antrea that referenced this issue Nov 8, 2021
Run the benchmarks once for each PR.

Fixes antrea-io#2984

Signed-off-by: Wenqi Qiu <wenqiq@vmware.com>
wenqiq added a commit to wenqiq/antrea that referenced this issue Nov 8, 2021
Run the benchmarks once for each PR.

Fixes antrea-io#2984

Signed-off-by: Wenqi Qiu <wenqiq@vmware.com>
wenqiq added a commit to wenqiq/antrea that referenced this issue Nov 8, 2021
Run the benchmarks once for each PR.

Fixes antrea-io#2984

Signed-off-by: Wenqi Qiu <wenqiq@vmware.com>
wenqiq added a commit to wenqiq/antrea that referenced this issue Nov 9, 2021
Run the benchmarks once for each PR.

Fixes antrea-io#2984

Signed-off-by: Wenqi Qiu <wenqiq@vmware.com>
tnqn pushed a commit that referenced this issue Nov 16, 2021
Run the benchmarks once for each PR.

Fixes #2984

Signed-off-by: Wenqi Qiu <wenqiq@vmware.com>
jianjuns pushed a commit to jianjuns/antrea that referenced this issue Dec 22, 2021
Run the benchmarks once for each PR.

Fixes antrea-io#2984

Signed-off-by: Wenqi Qiu <wenqiq@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/flow-visibility/exporter Issues or PRs related to the Flow Exporter functions in the Agent kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants