Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend go test CI from 10 to 15 miniutes #1418

Merged
merged 1 commit into from
Apr 8, 2024
Merged

Conversation

dylandreimerink
Copy link
Member

The default builtin timeout in the go test runner is 10 min and the GH runner is also limited to 10 min. In practice the GH runner is quicker to kill tests that exceed this time limit, in which case we abruptly exit without logs on which test caused the timeout.

This change extends the runner timeout to 15 min to give the go runner a chance to kill the bad test and to tell us which test was taking longer so that we can fix the issue.

The default builtin timeout in the go test runner is 10 min and the GH runner is also limited to 10 min. In practice the GH runner is quicker to kill tests that exceed this time limit, in which case we abruptly exit without logs on which test caused the timeout.

This change extends the runner timeout to 15 min to give the go runner a chance to kill the bad test and to tell us which test was taking longer so that we can fix the issue.

Signed-off-by: Dylan Reimerink <dylan.reimerink@isovalent.com>
@dylandreimerink dylandreimerink requested a review from a team as a code owner April 5, 2024 12:33
@dylandreimerink dylandreimerink requested a review from lmb April 5, 2024 12:33
@dylandreimerink
Copy link
Member Author

Speak of the devil. This was meant for another PR but this CI run also turned up something:

=== Failed
=== FAIL: perf TestPerfReaderWakeupEvents (unknown)
panic: test timed out after 10m0s
running tests:
	TestPerfReaderWakeupEvents (10m0s)

goroutine 29 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:2259 +0x3b9
created by time.goFunc
	/opt/hostedtoolcache/go/1.21.8/x64/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000a2680, {0x71cfe6?, 0x51bbfc?}, 0x72d1a0)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x94c4c0?)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc0000a2680, 0xc0000f9bd8)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000090320?, {0x945f40, 0xd, 0xd}, {0xc0000f9c90?, 0x4105c9?, 0x94bbe0?})
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc000090320)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1925 +0x636
github.com/cilium/ebpf/internal/testutils/fdtrace.TestMain(0xc0000366f0?)
	/home/runner/work/ebpf/ebpf/internal/testutils/fdtrace/fd.go:25 +0x70
github.com/cilium/ebpf/perf.TestMain(...)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:29
main.main()
	_testmain.go:77 +0x1c7

goroutine 27 [chan receive]:
testing.(*T).Parallel(0xc00026a000)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1403 +0x205
github.com/cilium/ebpf/perf.TestPause(0xc00026a000)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:408 +0x39
testing.tRunner(0xc00026a000, 0x72d168)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1648 +0x3ad

goroutine 28 [syscall]:
syscall.Syscall6(0x1?, 0x77?, 0x1?, 0x97a120?, 0x77?, 0x0?, 0xc0000fba48?)
	/opt/hostedtoolcache/go/1.21.8/x64/src/syscall/syscall_linux.go:91 +0x30
golang.org/x/sys/unix.EpollWait(0x0?, {0xc0000d2150?, 0x446411?, 0xc00009a4e0?}, 0xc0000fba18?)
	/home/runner/go/pkg/mod/golang.org/x/sys@v0.15.0/unix/zsyscall_linux_amd64.go:55 +0x4f
github.com/cilium/ebpf/internal/unix.EpollWait(...)
	/home/runner/work/ebpf/ebpf/internal/unix/types_linux.go:129
github.com/cilium/ebpf/internal/epoll.(*Poller).Wait(0xc0000cc5a0, {0xc0000d2150?, 0x2, 0x2}, {0xc0000fbab0?, 0x1?, 0x0?})
	/home/runner/work/ebpf/ebpf/internal/epoll/poller.go:145 +0x2a5
github.com/cilium/ebpf/perf.(*Reader).ReadInto(0xc0000d46c0, 0x1?)
	/home/runner/work/ebpf/ebpf/perf/reader.go:362 +0x2c5
github.com/cilium/ebpf/perf.(*Reader).Read(...)
	/home/runner/work/ebpf/ebpf/perf/reader.go:336
github.com/cilium/ebpf/perf.checkRecord({0x785618, 0xc00026a1a0}, 0xf?)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:167 +0x6e
github.com/cilium/ebpf/perf.TestPerfReaderWakeupEvents(0xc00026a1a0)
	/home/runner/work/ebpf/ebpf/perf/reader_test.go:528 +0x527
testing.tRunner(0xc00026a1a0, 0x72d1a0)
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.8/x64/src/testing/testing.go:1648 +0x3ad

Copy link
Collaborator

@lmb lmb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@lmb lmb merged commit 0ec5f53 into main Apr 8, 2024
14 of 15 checks passed
@lmb lmb deleted the pr/ci-increase-timeout branch April 8, 2024 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants