Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: segfault in timeSleepUntil #35462

Closed
bcmills opened this issue Nov 8, 2019 · 6 comments
Closed

runtime: segfault in timeSleepUntil #35462

bcmills opened this issue Nov 8, 2019 · 6 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone
Go1.14

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 8, 2019

Lots of failures involving variations on the following stack:

        runtime.sigpanic()
        	/workdir/go/src/runtime/signal_unix.go:651 +0x46a
        runtime.timeSleepUntil(0x78f68410ac)
        	/workdir/go/src/runtime/time.go:1258 +0xbe
        runtime.sysmon()
        	/workdir/go/src/runtime/proc.go:4455 +0xa1
        runtime.mstart1()
        	/workdir/go/src/runtime/proc.go:1125 +0xc3
        runtime.mstart()
        	/workdir/go/src/runtime/proc.go:1090 +0x6e

2019-11-08T17:01:18-a762221/linux-amd64-clang
2019-11-06T02:49:10-de70de6/openbsd-386-62
2019-11-05T14:34:56-79a00a3/openbsd-386-62
2019-11-05T00:17:34-04e80fb/linux-arm
2019-11-04T21:38:36-d9ee4b2/openbsd-amd64-62
2019-11-03T05:01:00-4497d7e/darwin-amd64-10_14
2019-11-03T04:57:50-c6621d9/netbsd-amd64-8_0
2019-11-02T22:02:43-3873e54/darwin-amd64-10_14
2019-11-02T21:51:18-62e53b7/dragonfly-amd64
2019-11-02T21:51:09-61fa798/netbsd-amd64-8_0
2019-11-02T19:25:54-afe50c1/openbsd-amd64-62
2019-11-02T19:21:24-5597e3d/linux-386-clang
2019-11-02T03:00:20-1e4a358/openbsd-amd64-62
2019-11-01T21:41:41-9bde9b4/solaris-amd64-oraclerel
2019-11-01T18:49:50-5aea597/darwin-amd64-10_12
2019-11-01T07:59:09-5b31021/linux-386
2019-10-31T17:21:56-48c0cef/freebsd-amd64-race
2019-10-31T17:09:48-a9b37ae/freebsd-amd64-race

CC @ianlancetaylor @cherrymui @aclements @mknyszek
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Nov 8, 2019
@bcmills bcmills added this to the Go1.14 milestone Nov 8, 2019
@ianlancetaylor ianlancetaylor self-assigned this Nov 8, 2019
@ianlancetaylor
Copy link
Member

Any failure before https://golang.org/cl/205558, committed on Wed Nov 6 15:46:26 2019 +0000, is likely fixed by that commit. I think that leaves us with only one failure:

https://build.golang.org/log/9298618f82f5cb0e6d17e745644af793111adc4c

@ianlancetaylor
Copy link
Member

The stack trace implies that there is a nil entry in pp.timers. Which should, of course, be impossible.

I'm not having any luck reproducing this.

@bcmills bcmills mentioned this issue Nov 11, 2019
Closed
@ianlancetaylor
Copy link
Member

Two more cases of this:

2019-11-11T22:40:45-c32aab3/darwin-amd64-10_11
2019-11-12T01:07:15-ec73263/openbsd-386-62

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/206938 mentions this issue: runtime: acquire timersLocks around moveTimers

@ianlancetaylor
Copy link
Member

One way this could happen would be if the number of P's changes, and that causes some calls to moveTimers. That runs with the world stopped, but the sysmon thread can still kick in and look through the timers. If that happens concurrently with moveTimers, we could see garbage. I sent CL 206938 to fix that. But I'm not sure that explains all the failures here, as some of them don't have any obvious calls to runtime.GOMAXPROCS.

gopherbot pushed a commit that referenced this issue Nov 13, 2019
@ianlancetaylor
runtime: acquire timersLocks around moveTimers
e762378 
In the discussion of CL 171828 we decided that it was not necessary to
acquire timersLock around the call to moveTimers, because the world is
stopped. However, that is not correct, as sysmon runs even when the world
is stopped, and it calls timeSleepUntil which looks through the timers.
timeSleepUntil acquires timersLock, but that doesn't help if moveTimers
is running at the same time.

Updates #6239
Updates #27707
Updates #35462

Change-Id: I346c5bde594c4aff9955ae430b37c2b6fc71567f
Reviewed-on: https://go-review.googlesource.com/c/go/+/206938
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@ianlancetaylor
Copy link
Member

Hasn't recurred for a week, calling this fixed.

@golang golang locked and limited conversation to collaborators Nov 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
3 participants
@ianlancetaylor @bcmills @gopherbot