Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build: all.bash takes too long #26473

Open
rsc opened this issue Jul 19, 2018 · 15 comments
Open

build: all.bash takes too long #26473

rsc opened this issue Jul 19, 2018 · 15 comments
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Backlog

Comments

@rsc
Copy link
Contributor

rsc commented Jul 19, 2018

My target for all.bash is to keep it around 3-4 minutes. Usually when it gets to 5 minutes I spend some time trimming it back down. Maybe 3-4 minutes is no longer attainable, but right now we're at 7 minutes, which is much longer than I'd hope.

I filed a few bugs to try to help: #26469, #26472, #26470, #26471.

Speeding all.bash may also help speed the trybots, which also take longer than I'd hope.
@rsc rsc added this to the Go1.12 milestone Jul 19, 2018
@bradfitz bradfitz added Testing An issue that has been verified to require only test changes, not just a test failure. help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Jul 19, 2018
This was referenced Jul 22, 2018
Closed
Closed
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/147358 mentions this issue: strings: lower running time of TestCompareStrings

gopherbot pushed a commit that referenced this issue Nov 4, 2018
@randall77
strings: lower running time of TestCompareStrings
4bb9b61 
At each comparison, we're making a copy of the whole string.
Instead, use unsafe to share the string backing store with a []byte.

It reduces the test time from ~4sec to ~1sec on my machine
(darwin/amd64).  Some builders were having much more trouble with this
test (>3min), it may help more there.

Fixes #26174
Fixes #28573
Fixes #26155
Update #26473

Change-Id: Id5856fd26faf6ff46e763a088f039230556a4116
Reviewed-on: https://go-review.googlesource.com/c/147358
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/152817 mentions this issue: cmd/cover: run tests in parallel, don't change source directory

gopherbot pushed a commit that referenced this issue Dec 6, 2018
@ianlancetaylor
cmd/cover: run tests in parallel, don't change source directory
4a801cd 
This speeds up the cmd/cover testsuite by about 40% on my laptop.

Updates #26473
Updates #28386

Change-Id: I853b1b3b8c98dc89440f7b7bf5c0ade1d3d66802
Reviewed-on: https://go-review.googlesource.com/c/152817
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@andybons andybons modified the milestones: Go1.12, Go1.13 Feb 12, 2019
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176900 mentions this issue: cmd/fix: mark tests as parallel

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176901 mentions this issue: cmd/objdump: mark tests as parallel

gopherbot pushed a commit that referenced this issue May 13, 2019
@josharian
cmd/objdump: mark tests as parallel
ed7a92b 
Speeds up

go test -short -count=1 cmd/objdump

on my machine from 1.7s to 1.3s.

Not much, but as the backpacking saying goes,
take care of the ounces and the pounds will take care of themselves.

Updates #26473

Change-Id: I59fe9a179e48537c7d82cbba72cde9f92b42a029
Reviewed-on: https://go-review.googlesource.com/c/go/+/176901
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176937 mentions this issue: cmd/compile: mark a few more tests as parallel

gopherbot pushed a commit that referenced this issue May 13, 2019
@josharian
cmd/fix: mark tests as parallel
3378683 
This speeds up

go test -short -count=1 cmd/fix

on my machine from about 8s to about 0.05s.

Updates #26473

Change-Id: I698ee20704ae0aee874ba642e7b0e070ddc99194
Reviewed-on: https://go-review.googlesource.com/c/go/+/176900
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue May 13, 2019
@josharian
cmd/compile: mark a few more tests as parallel
5d98330 
Reduces the time on my machine for

go clean -cache; go test -short -count=1 cmd/compile/internal/gc

from 4.7s to 3.7s.

Updates #26473

Change-Id: I9f9573675ffd6519da63961f48f61260ae4717fd
Reviewed-on: https://go-review.googlesource.com/c/go/+/176937
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/176957 mentions this issue: runtime: count the scavenger's M as idle when sleeping

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177097 mentions this issue: runtime: mark newly-mapped memory as scavenged

@rsc rsc mentioned this issue May 15, 2019
Closed
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177398 mentions this issue: cmd/go: cut 'go test -short cmd/go' time by about half

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177399 mentions this issue: runtime: disable some tests in -quick mode

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177417 mentions this issue: misc/cgo: disable testcarchive, testplugin during -short test (including all.bash)

gopherbot pushed a commit that referenced this issue May 16, 2019
@rsc
cmd/go: cut 'go test -short cmd/go' time by about half
dccd5da 
Was 50 seconds on unloaded Mac laptop; now 27.
Still longer than I would like, but every little bit helps.

For #26473.

Change-Id: Id4be016ee1555cbc3512eca0ae10236d7f06bd02
Reviewed-on: https://go-review.googlesource.com/c/go/+/177398
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue May 16, 2019
@rsc
runtime: disable some tests in -quick mode
018d9b4 
Speeds up the "go test runtime -cpu=1,2,4 -short -quick" phase of all.bash.

For #26473.

Change-Id: I090f5a5aa754462b3253a2156dc31fa67ce7af2a
Reviewed-on: https://go-review.googlesource.com/c/go/+/177399
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
gopherbot pushed a commit that referenced this issue May 16, 2019
@rsc
misc/cgo: disable testcarchive, testplugin during -short test (includ…
a6a0ed3 
…ing all.bash)

These tests take 20 seconds each to rebuild the entire world
in their respective modes. That's too much for short mode,
including all.bash on developer machines.

Keep doing it on builders and if someone runs 'go test' by hand
in that directory.

For #26473.

Change-Id: I3dc6955bc3aa7a20fd170efcde72a7d19b37bdbf
Reviewed-on: https://go-review.googlesource.com/c/go/+/177417
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177559 mentions this issue: all: shorten some tests

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177558 mentions this issue: misc/cgo/test: consolidate tests into fewer cgo source files

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177577 mentions this issue: test: skip cross-arch codegen tests in all.bash

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177597 mentions this issue: cmd/api: read std package info once, not per goos-goarch-cgo

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/177599 mentions this issue: misc/cgo/errors: consolidate test work

gopherbot pushed a commit that referenced this issue May 16, 2019
@mknyszek
runtime: mark newly-mapped memory as scavenged
4e7bef8 
On most platforms newly-mapped memory is untouched, meaning the pages
backing the region haven't been faulted in yet. However, we mark this
memory as unscavenged which means the background scavenger
aggressively "returns" this memory to the OS if the heap is small.

The only platform where newly-mapped memory is actually unscavenged (and
counts toward the application's RSS) is on Windows, since
(*mheap).sysAlloc commits the reservation. Instead of making a special
case for Windows, I change the requirements a bit for a sysReserve'd
region. It must now be both sysMap'd and sysUsed'd, with sysMap being a
no-op on Windows. Comments about memory allocation have been updated to
include a more up-to-date mental model of which states a region of memory
may be in (at a very low level) and how to transition between these
states.

Now this means we can correctly mark newly-mapped heap memory as
scavenged on every platform, reducing the load on the background
scavenger early on in the application for small heaps. As a result,
heap-growth scavenging is no longer necessary, since any actual RSS
growth will be accounted for on the allocation codepath.

Finally, this change also cleans up grow a little bit to avoid
pretending that it's freeing an in-use span and just does the necessary
operations directly.

Fixes #32012.
Fixes #31966.
Updates #26473.

Change-Id: Ie06061eb638162e0560cdeb0b8993d94cfb4d290
Reviewed-on: https://go-review.googlesource.com/c/go/+/177097
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
gopherbot pushed a commit that referenced this issue May 21, 2019
@rsc
test: skip cross-arch codegen tests in all.bash
4f248e9 
The test/codegen tests check all architectures
mentioned in the test file, but this requires
building at least the runtime for that architecture.
This CL changes the test to only check the local
architecture, leaving checking of other architectures
to the relevant builders, as usual.

This cuts 'go run run.go codegen' by 12r 78u 21s.

After this change, all.bash runs in ~4:40 on my laptop.

For #26473.

Change-Id: Ia0354d1aff2df2949f838528c8171410bc42dc8b
Reviewed-on: https://go-review.googlesource.com/c/go/+/177577
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue May 21, 2019
@rsc
cmd/api: read std package info once, not per goos-goarch-cgo
b864818 
Cuts api test time from 12.7r 26.2u 14.2s to 7.5r 12.1u 2.2s.

After this change, all.bash runs in ~4:36 on my laptop.

For #26473.

Change-Id: I4211e6afcd7ab61a4ed2c9a2aa5ac1ea04982695
Reviewed-on: https://go-review.googlesource.com/c/go/+/177597
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue May 21, 2019
@rsc
misc/cgo/errors: consolidate test work
798e0b3 
Build a single binary containing all the TestPointerChecks
instead of building many small binaries,
each with its own cgo+compile+link invocation.
This cuts 'go test -run=TestPointerChecks'
from 6.7r 35.5u 26.1s to 2.1r 2.1u 1.4s.

Move as many cgo checks as possible into fewer test files
for TestReportsTypeErrors too.
This cuts 'go test -run=TestReportsTypeErrors'
from 2.1r 6.7u 6.7s to 1.5r 2.5u 2.5s.

After this change, all.bash runs in ~4:30 on my laptop.

For #26473.

Change-Id: I3787448b03689a1f62dd810957ab6013bb75582f
Reviewed-on: https://go-review.googlesource.com/c/go/+/177599
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue May 22, 2019
@rsc
misc/cgo/test: consolidate tests into fewer cgo source files
b0e238a 
Each different file that does import "C" must be compiled
and analyzed separately by cgo. Having fewer files import "C"
reduces the cost of building the test. This is especially important
because this test is built and run four different times (with different
settings) during all.bash.

go test -c in this directory used to take over 20 seconds on my laptop.
Now it takes under 5 seconds.

Removes 23.4r 29.0u 21.5s from all.bash.

For #26473.

Change-Id: Ie7cb7b0d9d6138ebd2eb548d0d8ea6e409ae10b9
Reviewed-on: https://go-review.googlesource.com/c/go/+/177558
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
gopherbot pushed a commit that referenced this issue May 22, 2019
@rsc
all: shorten some tests
06b0bab 
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.

After this change, all.bash is under 5 minutes again on my laptop.

For #26473.

Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@ianlancetaylor ianlancetaylor mentioned this issue Aug 21, 2019
Closed
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@dmitshur dmitshur mentioned this issue May 13, 2020
Open
@qiulaidongfeng qiulaidongfeng mentioned this issue Jan 20, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@bradfitz @rsc @andybons @gopherbot