runtime: support linux/arm 64kB page size #12480
Comments
|
What is the concrete bug here? This reads to me like syncthing has a bug. Or, what is Go doing wrong? |
|
see also #10180 for the linker enhancement and more discussion Basically runtime assumes 4KB pages in its memory allocator, |
|
Some newer Synology NAS like the DS715+ are affected as well as the kernel is using 64k pages. |
|
Other ARM-based NAS like ("Western Digital My Cloud Mirror")[http://www.wdc.com/de/products/products.aspx?id=1640] have a PAGESIZE of 32768. Popular golang projects like |
|
Austin and Rick are here at GopherCon, i'll ask them what the consequences On Tue, Jul 12, 2016 at 8:12 AM, Roland Moriz notifications@github.com
|
|
/cc @aclements @RLH this is the issue we spoke about at GopherCon. |
|
CL https://golang.org/cl/25021 mentions this issue. |
|
@prodigy7, @tseel, @rmoriz, you should be able to cherry-pick https://go-review.googlesource.com/c/25021 on to Go 1.7 to get larger pages on your NASes. You'll have to rebuild the toolchain, but it should work. The next release should work without any additional effort, since it will detect whatever the OS page size is and use that directly. There may still be linker issues with Go on large page machines, as documented in #10180. Since this has been reduced to solving the remaining problems in #10180, I'm going to go ahead and close this as a dup. |
Currently we assume the physical page size on ARM is 4kB. While this is usually true, the architecture also supports 16kB and 64kB physical pages, and Linux (and possibly other OSes) can be configured to use these larger page sizes. With Go 1.6, such a configuration could potentially run, but generally resulted in memory corruption or random panics. With current master, this configuration will cause the runtime to panic during init on Linux when it checks the true physical page size (and will still cause corruption or panics on other OSes). However, the assumed physical page size only has to be a multiple of the true physical page size, the scavenger can now deal with large physical page sizes, and the rest of the runtime can deal with a larger assumed physical page size than the true size. Hence, there's little disadvantage to conservatively setting the assumed physical page size to 64kB on ARM. This may result in some extra memory use, since we can only return memory at multiples of the assumed physical page size. However, it is a simple change that should make Go run on systems configured for larger page sizes. The following commits will make the runtime query the actual physical page size from the OS, but this is a simple step there. Updates #12480. Change-Id: I851829595bc9e0c76235c847a7b5f62ad82b5302 Reviewed-on: https://go-review.googlesource.com/25021 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Minux Ma <minux@golang.org>
Currently the physical page size assumed by the runtime is hard-coded. On Linux the runtime at least fetches the OS page size during init and sanity checks against the hard-coded value, but they may still differ. On other OSes we wouldn't even notice. Add support on all OSes to fetch the actual OS physical page size during runtime init and lift the sanity check of PhysPageSize from the Linux init code to general malloc init. Currently this is the only use of the retrieved page size, but we'll add more shortly. Updates #12480 and #10180. Change-Id: I065f2834bc97c71d3208edc17fd990ec9058b6da Reviewed-on: https://go-review.googlesource.com/25050 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
Now that the runtime fetches the true physical page size from the OS, make the physical page size used by heap growth a variable instead of a constant. This isn't used in any performance-critical paths, so it shouldn't be an issue. sys.PhysPageSize is also renamed to sys.DefaultPhysPageSize to make it clear that it's not necessarily the true page size. There are no uses of this constant any more, but we'll keep it around for now. Updates #12480 and #10180. Change-Id: I6c23b9df860db309c38c8287a703c53817754f03 Reviewed-on: https://go-review.googlesource.com/25022 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
Currently we assume the physical page size on ARM is 4kB. While this is usually true, the architecture also supports 16kB and 64kB physical pages, and Linux (and possibly other OSes) can be configured to use these larger page sizes. With Go 1.6, such a configuration could potentially run, but generally resulted in memory corruption or random panics. With current master, this configuration will cause the runtime to panic during init on Linux when it checks the true physical page size (and will still cause corruption or panics on other OSes). However, the assumed physical page size only has to be a multiple of the true physical page size, the scavenger can now deal with large physical page sizes, and the rest of the runtime can deal with a larger assumed physical page size than the true size. Hence, there's little disadvantage to conservatively setting the assumed physical page size to 64kB on ARM. This may result in some extra memory use, since we can only return memory at multiples of the assumed physical page size. However, it is a simple change that should make Go run on systems configured for larger page sizes. The following commits will make the runtime query the actual physical page size from the OS, but this is a simple step there. Updates golang#12480. Change-Id: I851829595bc9e0c76235c847a7b5f62ad82b5302 Reviewed-on: https://go-review.googlesource.com/25021 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Minux Ma <minux@golang.org>
Currently the physical page size assumed by the runtime is hard-coded. On Linux the runtime at least fetches the OS page size during init and sanity checks against the hard-coded value, but they may still differ. On other OSes we wouldn't even notice. Add support on all OSes to fetch the actual OS physical page size during runtime init and lift the sanity check of PhysPageSize from the Linux init code to general malloc init. Currently this is the only use of the retrieved page size, but we'll add more shortly. Updates golang#12480 and golang#10180. Change-Id: I065f2834bc97c71d3208edc17fd990ec9058b6da Reviewed-on: https://go-review.googlesource.com/25050 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
Now that the runtime fetches the true physical page size from the OS, make the physical page size used by heap growth a variable instead of a constant. This isn't used in any performance-critical paths, so it shouldn't be an issue. sys.PhysPageSize is also renamed to sys.DefaultPhysPageSize to make it clear that it's not necessarily the true page size. There are no uses of this constant any more, but we'll keep it around for now. Updates golang#12480 and golang#10180. Change-Id: I6c23b9df860db309c38c8287a703c53817754f03 Reviewed-on: https://go-review.googlesource.com/25022 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
Hi,
I bought last week a Zyxel NAS540 which uses a kernel with 64k pages. Now, after some days of investigating how I can run own software on the NAS (the NAS itself is open for access from outside), I found out that
It would nice, if Go would be supplemented with a build for ARM/64k. Another discussion related to that topic in context with the software syncthing could also found here: syncthing/syncthing#928
I compiled syncthing like here https://forum.syncthing.net/t/request-for-64k-binaries/5555/9 described but when I start syncthing, I got errors like that and I think its a 64k problem:
The text was updated successfully, but these errors were encountered: