runtime: "fatal error: missed stack barrier" failures on the darwin/arm64 android/arm64 builders

[eliasnaur]

The iOS builders are not the most reliable, but I have seen quite a few GC-related builder failures that look like they could be real:

http://build.golang.org/log/eaa998d553346cee530c589e84f08eb0a2e524c0
http://build.golang.org/log/5bf02c8fea879278f4c0f9dc8d22a8caf8752f07
http://build.golang.org/log/405e3f41f9efe579885e11a4ecd4dc626017be38
http://build.golang.org/log/84cf52a8162f42452627ad54a72a04e48afb6f1d
http://build.golang.org/log/743957631e9b3af9376e63759780329aae4d0543
http://build.golang.org/log/d97088062bd9ffa500fe37c24c0d373a996ee0cb

(all darwin/arm64, I haven't seen a similar failure on darwin/arm)

The iOS builders had falled into disarray for months until around the 25. of March, so I don't think I can come closer to an offending CL, if any.

[eliasnaur]

I found similar failures on the more reliable android/arm64 builders:

http://build.golang.org/log/10e13869523708ea0968eb6fc2d8c0871a55825a
http://build.golang.org/log/102819b526620f5699bb1bbaba795271ada8f462
http://build.golang.org/log/767c37d013529f5d76b1c018f3f4c6710469aaee

I haven't seen anything on the android/386, android/arm nor on linux/arm64, which reminds me of physical page size issues like #9993 (according to #11886 the linux/arm64 builder is using 4k pages).

[eliasnaur]

cc @aclements @randall77

[aclements]

The android/arm64 ones are all in TestStackBarrierProfiling, which uses a funny debug mode that may simply be broken on arm64 (it's definitely broken on ppc64).

The darwin/arm64 ones are more concerning. It looks like the stack barrier list is just wrong (e.g., [@@@ ==> *0xd=0x0] in the last one). That's a bad pointer and a bad value. The "missed stack barrier" panic is almost certainly just a symptom of the earlier nil-pointer dereference in gcRemoveStackBarrier, but I'm not sure where the bad barrier pointers came from in the first place.

Is this builder a multicore ARM64 or single core? If it's multicore, this could be a weak memory order issue.

[eliasnaur]

I don't know the exact hardware on the builder, but the builder name (darwin-arm64-a7ios) suggests it is running on an Apple A7 processor. A7 has 2 cores according to https://en.wikipedia.org/wiki/Apple_A7.

[mwhudson]

I'm not aware of any single core arm64 implementations fwiw. There probably are some, but they're definitely not the common case.

[crawshaw]

TestStackBarrierProfiling just failed on android/arm with missed stack barrier:

https://build.golang.org/log/032ea3f75a48e9760600042046066e24a5e3234e

[eliasnaur]

Here are some recent failures. Twice on android/arm64 (which may be expected)

https://build.golang.org/log/3f674668ee150f41b147bd038d54abc6540ebdf3
https://build.golang.org/log/9cc449cbedd4ae6cdf9d3377ba99201f383db611

But also one on linux/arm64:

https://build.golang.org/log/6c5f8cce8abb0fd0cb095820cedf2e7764d6c191

[gopherbot]

CL https://golang.org/cl/23134 mentions this issue.

[gopherbot]

CL https://golang.org/cl/23291 mentions this issue.

[eliasnaur]

Here's a recent one from an android/arm64 builder:

https://build.golang.org/log/e7ef66722bdccf4446c52fbfbb8783883662c2b4

You mentioned that TestStackBarrierProfiling on android/arm64 might not work reliably for other reasons than this issue's topic. Should I open a new issue, and perhaps skip TestStackBarrierProfiling on android/arm64 (and perhaps other GOOS/GOARCH combinations) in the meantime?