runtime: macOS 10.10 builder is red with TSAN memory alloc failures #26475
Labels
FrozenDueToAge
NeedsFix
The path to resolution is known, but the work has not been done.
OS-Darwin
release-blocker
Milestone
Comments
|
@randall77 debugged this a bit when I blamed the parent commit of 7951d90:
|
|
Since Dec 2017 (in https://go-review.googlesource.com/82355), the Mac builders are always fresh VMs booted per build. They should be identical between build.golang.org and gomote and trybots. The only difference is the build system doesn't use |
|
I was able to reproduce with |
|
I'm able to reproduce at 94076fe (@randall77's commit, 7951d90~), but not at 7951d90~~, so this does appear to be from commit 94076fe. |
I don't think so. |
|
Ah, interesting, my commit to use TSAN-compatible hints in race mode (2d4c1ff) changed the nature of the failure: https://build.golang.org/log/6d2ecc136e9649755ba2a8c0f38ce9e656a854eb This is still really weird, though. Before we've even run user code, it's claiming that it couldn't get a single successful mmap in the entire TSAN heap range. Maybe this is some conflict with CoreFoundation global initializers, given that 94076fe started linking in CoreFoundation? |
|
It is in fact blowing through all of the arena hints. The mmap calls are succeeding, but returning a different address than requested: I made it spin after blowing through all of the hints and used Notably, there's nothing in [0xc000000000, 0xe000000000) at all. Maybe CoreFoundation in macOS 10.10 is intercepting |
|
Now I'm extra confused. I tried the following C program: #include <sys/mman.h>
#include <stdio.h>
#include <errno.h>
int main()
{
void *x = mmap((void*)0xc000000000ull, 0x4000000, PROT_NONE, MAP_PRIVATE|MAP_ANON, -1, 0);
printf("%p %d\n", x, errno);
return 0;
}Even when compiled with just |
|
MAP_FIXED? |
|
It does honor the address with |
|
This keeps getting weirder.
|
|
Change https://golang.org/cl/125304 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jul 23, 2018
For unknown reasons, linking against CoreFoundation on macOS 10.10 sometimes causes mmap to ignore the hint address, which makes the Go allocator incompatible with TSAN. Currently, the effect of this is to run the allocator out of arena hints on the very first allocation, causing a "too many address space collisions for -race mode" panic. This CL skips the cgo tests that link against CoreFoundation in race mode. Updates #26475. Updates #26513. Change-Id: I52ec638c99acf5d4966e68ff0054f7679680dac6 Reviewed-on: https://go-review.googlesource.com/125304 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
|
macOS 10.10 is passing now. I'm going to close this even though we don't completely understand this bug. #26513 tracks reverting this when we drop macOS 10.10 support. |
|
This appears to be popping up again. It has failed all of the builds for macOS 10.10 over the last day. The latest failure: https://build.golang.org/log/cccb90e1ae132b4037e0ffbdcc58fea77aa72b39 |
|
This is still failing for all darwin-amd64-10_10 builds. /cc owners: @aclements @rsc @RLH @randall77 |
|
The fact that it completely ignores the hint is very weird, but the kernel is well within its power to give back whatever address it wants. So given that, and also given that the race detector expects the heap within a certain address range, maybe it actually does make sense to map new memory as We already fail if we can't maintain the invariant that the heap only lives in a certain region, so if we can't map it with Anyway, I'll give |
|
Change https://golang.org/cl/153897 mentions this issue: |
|
Change https://golang.org/cl/155097 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Dec 19, 2018
This change disables the test TestArenaCollision on Darwin in race mode to deal with the fact that Darwin 10.10 must use MAP_FIXED in race mode to ensure we retain our heap in a particular portion of the address space which the race detector needs. The test specifically checks to make sure a manually mapped region's space isn't re-used, which is definitely possible with MAP_FIXED because it replaces whatever mapping already exists at a given address. This change then also makes it so that MAP_FIXED is only used in race mode and on Darwin, not all BSDs, because using MAP_FIXED breaks this test for FreeBSD in addition to Darwin. Updates #26475. Fixes #29340. Change-Id: I1c59349408ccd7eeb30c4bf2593f48316b23ab2f Reviewed-on: https://go-review.googlesource.com/c/155097 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
Change https://golang.org/cl/205757 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Nov 7, 2019
This change removes a hack which was added to deal with Darwin 10.10's weird ignorance of mapping hints which would cause race mode to fail since it requires the heap to live within a certain address range. We no longer support 10.10, and this is potentially causing problems related to the page allocator, so drop this code. Updates #26475. Updates #35112. Change-Id: I0e1c6f8c924afe715a2aceb659a969d7c7b6f749 Reviewed-on: https://go-review.googlesource.com/c/go/+/205757 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
|
Change https://golang.org/cl/214057 mentions this issue: |
|
Change https://golang.org/cl/214058 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Jan 10, 2020
Go 1.14 will drop support for macOS 10.10, see #23011 This reverts CL 125304 Updates #26475 Updates #26513 Change-Id: Ia13eef30f22d67103f7ae45424124fbb116e1261 Reviewed-on: https://go-review.googlesource.com/c/go/+/214057 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot
pushed a commit
that referenced
this issue
Jan 15, 2020
Go 1.14 will drop support for macOS 10.10, see #23011 This reverts CL 155097 Updates #26475 Updates #29340 Change-Id: I64d0275141407313b73068436ee81d13eacc4c76 Reviewed-on: https://go-review.googlesource.com/c/go/+/214058 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The release notes for Go 1.11 say that our minimum macOS supported version is "macOS 10.10 Yosemite or later", but our macOS 10.10 column is currently red.
The first failure I see is: (rev 7951d90, which I happened to approve perhaps mistakenly thinking it was trivially correct and only affected debug mode.. maybe wrong?)
https://build.golang.org/log/17de002c5f0eb3ea6136185d5f943d1f38eb5a63
/cc @randall77 @aclements @dvyukov
The text was updated successfully, but these errors were encountered: