runtime: address space conflict on Linux/Arm64 #11886

jperrin · 2015-07-27T17:56:55Z

I'm attempting to build the golang 1.5 beta2 on the Arm64 port of CentOS 7, however I appear to be running into memory issues. The initial cross-compile completes successfully on an x86_64 CentOS-7 host via the following command

GOOS=linux GOARCH=arm64 GOROOT_BOOTSTRAP=/opt/go1.4.2 ./bootstrap.bash

However when I move the resulting bootstrap tarball to the arm64 box, I immediately get a memory address space conflict no matter what go command I run.

[jperrin@apma3 go-linux-arm64-bootstrap]$ ./bin/go help
runtime: address space conflict: map(0x481fff8000) = 0x481fff0000
fatal error: runtime: address space conflict

runtime stack:
runtime.throw(0x599ec0, 0x1f)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/panic.go:527 +0x80 fp=0x3fff9b7bd60 sp=0x3fff9b7bd40
runtime.sysMap(0x481fff8000, 0x8000, 0x4820000000, 0x7b7fb8)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mem_linux.go:147 +0x134 fp=0x3fff9b7bda0 sp=0x3fff9b7bd60
runtime.mHeap_MapBits(0x7997a0, 0x4820100000)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mbitmap.go:141 +0xec fp=0x3fff9b7bdd0 sp=0x3fff9b7bda0
runtime.mHeap_SysAlloc(0x7997a0, 0x100000, 0x792954)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/malloc.go:424 +0x15c fp=0x3fff9b7be50 sp=0x3fff9b7bdd0
runtime.mHeap_Grow(0x7997a0, 0x8, 0x0)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mheap.go:625 +0x70 fp=0x3fff9b7beb0 sp=0x3fff9b7be50
runtime.mHeap_AllocSpanLocked(0x7997a0, 0x1, 0x0)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mheap.go:529 +0x6bc fp=0x3fff9b7bf10 sp=0x3fff9b7beb0
runtime.mHeap_Alloc_m(0x7997a0, 0x1, 0x15, 0x0)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mheap.go:422 +0x14c fp=0x3fff9b7bf40 sp=0x3fff9b7bf10
runtime.mHeap_Alloc.func1()
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mheap.go:481 +0x50 fp=0x3fff9b7bf70 sp=0x3fff9b7bf40
runtime.systemstack(0x3fff9b7bfa8)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/asm_arm64.s:251 +0xb8 fp=0x3fff9b7bf80 sp=0x3fff9b7bf70
runtime.mHeap_Alloc(0x7997a0, 0x1, 0x10000000015, 0x97880)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mheap.go:482 +0x60 fp=0x3fff9b7bfd0 sp=0x3fff9b7bf80
runtime.mCentral_Grow(0x7a2170, 0x9322c)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mcentral.go:205 +0xc0 fp=0x3fff9b7c040 sp=0x3fff9b7bfd0
runtime.mCentral_CacheSpan(0x7a2170, 0x4000)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mcentral.go:101 +0x700 fp=0x3fff9b7c090 sp=0x3fff9b7c040
runtime.mCache_Refill(0x3ff970d0000, 0x15, 0x7b7fa8)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/mcache.go:118 +0xf4 fp=0x3fff9b7c0d0 sp=0x3fff9b7c090
runtime.mallocgc.func2()
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/malloc.go:611 +0x34 fp=0x3fff9b7c0f0 sp=0x3fff9b7c0d0
runtime.systemstack(0x3fff9b7c1a8)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/asm_arm64.s:251 +0xb8 fp=0x3fff9b7c100 sp=0x3fff9b7c0f0
runtime.mallocgc(0x180, 0x5208a0, 0x8800000000, 0x4000000000)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/malloc.go:612 +0x874 fp=0x3fff9b7c1e0 sp=0x3fff9b7c100
runtime.newobject(0x5208a0, 0xb32b0)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/malloc.go:742 +0x78 fp=0x3fff9b7c210 sp=0x3fff9b7c1e0
runtime.malg(0x8000, 0xb325c)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/proc1.go:2165 +0x28 fp=0x3fff9b7c250 sp=0x3fff9b7c210
runtime.mpreinit(0x793a60)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/os1_linux.go:188 +0x24 fp=0x3fff9b7c270 sp=0x3fff9b7c250
runtime.mcommoninit(0x793a60)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/proc1.go:114 +0x114 fp=0x3fff9b7c2c0 sp=0x3fff9b7c270
runtime.schedinit()
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/proc1.go:57 +0x74 fp=0x3fff9b7c310 sp=0x3fff9b7c2c0
runtime.rt0_go(0x3fff9b7f690, 0x3fff9b7f699, 0x0, 0x3fff9b7f69e, 0x3fff9b7f6af, 0x3fff9b7f6be, 0x3fff9b7f6d6, 0x3fff9b7f6ea, 0x3fff9b7f6fa, 0x3fff9b7f708, ...)
    /home/jperrin/code/github/go-linux-arm64-bootstrap/src/runtime/asm_arm64.s:73 +0xbc fp=0x3fff9b7c340 sp=0x3fff9b7c310

I've tried a number of variations on the build theme, but they all fail in the same manner.
I think this is go assuming/forcing a 4K PAGESIZE rather than the 64K PAGESIZE we have on linux/arm64, but I don't know enough to verify this. Any help would be greatly appreciated.

The text was updated successfully, but these errors were encountered:

vbatts · 2015-07-27T18:00:23Z

also, the same "address space conflict" can be seen when compiling go1.5beta2 from gcc-5.1.1 (which has gccgo supporting go1.4.2) natively on arm64.

ianlancetaylor · 2015-07-27T18:09:04Z

Looks like a page size problem. The current runtime assumes a 8192 byte page size. Your arm64 system seems to be using a 65,536 byte page size. Does that sound right?

jperrin · 2015-07-27T18:11:38Z

That's correct, yes.

[jperrin@apma3 bin]$ getconf PAGESIZE
65536

rsc · 2015-07-27T18:22:07Z

Please try this patch:

diff --git a/src/runtime/arch1_arm64.go b/src/runtime/arch1_arm64.go
index 1a3165c..29a87db 100644
--- a/src/runtime/arch1_arm64.go
+++ b/src/runtime/arch1_arm64.go
@@ -8,7 +8,7 @@ const (
    thechar        = '7'
    _BigEndian     = 0
    _CacheLineSize = 32
-   _PhysPageSize  = 4096*(1-goos_darwin) + 16384*goos_darwin
+   _PhysPageSize  = 65536
    _PCQuantum     = 4
    _Int64Align    = 8
    hugePageSize   = 0

jperrin · 2015-07-27T18:33:08Z

That patch works for me. Thanks!

davecheney · 2015-07-27T18:44:37Z

Ubuntu kernels still stick with 4k pages, but it looks like we're in the minority.

I'd be happy to set the page size to 64k on arm64 for linux, which is what we have done for ppc64 I believe.

On 28 Jul 2015, at 04:33, Jim Perrin notifications@github.com wrote:

That patch works for me. Thanks!

—
Reply to this email directly or view it on GitHub.

gopherbot · 2015-07-28T18:00:27Z

CL https://golang.org/cl/12747 mentions this issue.

aclements · 2015-07-28T21:07:10Z

Does this mean we now won't release memory to the system on arm64 because of the work-around we have in place in scavengelist for #9993?

jefby · 2015-07-29T06:57:51Z

Hello,@rsc，my arm64 kernel is also use 64k PAGESIZE , how to dynamic check the PAGESIZE and set it ??

davecheney · 2015-07-29T07:01:58Z

% getconf PAGESIZE

To the best of my knowledge you cannot change the page size the kernel
uses, it is set when compiling the kernel.

On Wed, Jul 29, 2015 at 4:57 PM, jefby notifications@github.com wrote:

Hello,@rsc https://github.com/rsc，my arm64 kernel is also use 64k
PAGESIZE , how to dynamic check the PAGESIZE and set it ??

—
Reply to this email directly or view it on GitHub
#11886 (comment).

jefby · 2015-07-29T07:18:29Z

@davecheney I'm sorry ,my means is that dynamic check the system PAGESIZE in code src/runtime/arch1_arm64.go , and set its value correctly not force it to some value.

mwhudson · 2015-07-29T08:20:20Z

My understanding from the last time this went around is that the page size
needs to be a constant. But maybe I'm wrong...

On 29 July 2015 at 19:18, jefby notifications@github.com wrote:

@davecheney https://github.com/davecheney I'm sorry ,my means is that
dynamic check the system PAGESIZE in code src/runtime/arch1_arm64.go , and
set its value correctly not force it to some value.

—
Reply to this email directly or view it on GitHub
#11886 (comment).

davecheney · 2015-07-29T08:47:05Z

It can be recovered by a syscall, or from the auxv variable passed to the
program.

The issue we probably cannot address for 1.5 is the page size value is used
in several constant expressions which would become considerably more
expensive if the value wasn't a constant.

@khr would probably know more about this.

Thanks

Dave

On Wed, 29 Jul 2015 17:18 jefby notifications@github.com wrote:

@davecheney https://github.com/davecheney I'm sorry ,my means is that
dynamic check the system PAGESIZE in code src/runtime/arch1_arm64.go , and
set its value correctly not force it to some value.

—
Reply to this email directly or view it on GitHub
#11886 (comment).

aclements · 2015-07-29T16:14:05Z

The issue we probably cannot address for 1.5 is the page size value is used
in several constant expressions which would become considerably more
expensive if the value wasn't a constant.

I don't think replacing _PhysPageSize with a variable would actually be very expensive. Unlike _PageSize, it's not used in many places.

I think it's much too late to do this for 1.5, though we could query the physical page size from the OS at initialization time and at least check that it's <= _PhysPageSize.

davecheney · 2015-07-29T22:05:19Z

That's good to know. Phys page size is available via a syscall, or in the
auxv vector on startup on all the linux platforms we support.

To be clear, I would not want to see this happen for 1.5.

On Thu, 30 Jul 2015 02:14 Austin Clements notifications@github.com wrote:

The issue we probably cannot address for 1.5 is the page size value is used
in several constant expressions which would become considerably more
expensive if the value wasn't a constant.

I don't think replacing _PhysPageSize with a variable would actually be
very expensive. Unlike _PageSize, it's not used in many places.

I think it's much too late to do this for 1.5, though we could query the
physical page size from the OS at initialization time and at least check
that it's <= _PhysPageSize.

—
Reply to this email directly or view it on GitHub
#11886 (comment).

4ad · 2015-09-10T08:11:08Z

I am 100% sure I set arm64 to use 64k pages. I am not sure how we ended up with 4k pages again. Oh well.

minux · 2015-09-10T22:31:14Z

Merged into #10180.

ianlancetaylor changed the title ~~Linux/Arm64 bootstrap issue, runtime: address space conflict~~ runtime: address space conflict on Linux/Arm64 Jul 27, 2015

ianlancetaylor added this to the Go1.5 milestone Jul 27, 2015

rsc closed this as completed in 7a63ab1 Jul 28, 2015

eliasnaur mentioned this issue Apr 6, 2016

runtime: "fatal error: missed stack barrier" failures on the darwin/arm64 android/arm64 builders #15138

Closed

golang locked and limited conversation to collaborators Sep 22, 2016

gopherbot added the FrozenDueToAge label Sep 22, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: address space conflict on Linux/Arm64 #11886

runtime: address space conflict on Linux/Arm64 #11886

jperrin commented Jul 27, 2015

vbatts commented Jul 27, 2015

ianlancetaylor commented Jul 27, 2015

jperrin commented Jul 27, 2015

rsc commented Jul 27, 2015

jperrin commented Jul 27, 2015

davecheney commented Jul 27, 2015

gopherbot commented Jul 28, 2015

aclements commented Jul 28, 2015

jefby commented Jul 29, 2015

davecheney commented Jul 29, 2015

jefby commented Jul 29, 2015

mwhudson commented Jul 29, 2015

davecheney commented Jul 29, 2015

aclements commented Jul 29, 2015

davecheney commented Jul 29, 2015

4ad commented Sep 10, 2015

minux commented Sep 10, 2015 via email

runtime: address space conflict on Linux/Arm64 #11886

runtime: address space conflict on Linux/Arm64 #11886

Comments

jperrin commented Jul 27, 2015

vbatts commented Jul 27, 2015

ianlancetaylor commented Jul 27, 2015

jperrin commented Jul 27, 2015

rsc commented Jul 27, 2015

jperrin commented Jul 27, 2015

davecheney commented Jul 27, 2015

gopherbot commented Jul 28, 2015

aclements commented Jul 28, 2015

jefby commented Jul 29, 2015

davecheney commented Jul 29, 2015

jefby commented Jul 29, 2015

mwhudson commented Jul 29, 2015

davecheney commented Jul 29, 2015

aclements commented Jul 29, 2015

davecheney commented Jul 29, 2015

4ad commented Sep 10, 2015

minux commented Sep 10, 2015 via email