Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools: build failing on windows-amd64-longtest #35186

Closed
bcmills opened this issue Oct 26, 2019 · 5 comments
Closed

x/tools: build failing on windows-amd64-longtest #35186

bcmills opened this issue Oct 26, 2019 · 5 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows release-blocker Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Go1.14

Comments

@bcmills
Copy link
Contributor

bcmills commented Oct 26, 2019 •
edited by dmitshur
Loading

The windows-amd64-longtest builder is failing in numerous ways on x/tools.

Looks like some of the same out-of-memory issues as in #33951, but there may be other failures too.

https://build.golang.org/log/591874cbe6ad7d7b59ad1f9ef2b3d34e81af20a4

CC @ianthehat @matloob @stamblerre
@bcmills bcmills added OS-Windows NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 26, 2019
@gopherbot gopherbot added this to the Unreleased milestone Oct 26, 2019
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Oct 26, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Dec 12, 2019 •
edited
Loading

I believe this is a release-blocker via #11811.

(CC @cagedmantis @dmitshur @toothrot)

@bcmills bcmills modified the milestones: Unreleased, Go1.14 Dec 12, 2019
@dmitshur dmitshur mentioned this issue Dec 20, 2019
Closed
@dmitshur dmitshur self-assigned this Jan 2, 2020
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/213557 mentions this issue: dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM

@dmitshur
Copy link
Contributor

dmitshur commented Jan 7, 2020 •
edited
Loading

I've spun up a bunch of VMs/laptops with Windows 7/10 (WindowsBuild was very helpful), some with 8 GB RAM and one with 16 GB.

8 GB

On 8 GB configurations, I was able to get go test golang.org/x/tools/... to pass, but could not get go test -race golang.org/x/tools/... (with -race) to pass even once, despite multiple attempts and test caching helping out:

tools race oom 1

tools race oom 2

All 7-8~ GB RAM were used up during a large portion of the time.

16 GB

With 16 GB, go test -race golang.org/x/tools/... (with -race) passed on first try.

image

RAM usage still reached nearly all 15~16 GB RAM used up some of the time.

I've sent CL 213557 which bumps the RAM from 7.2 to 14.4 on windows-amd64-{longtest,race} builders. I think it's a reasonable next step to try, since some other builders already get 16 GB. /cc @bradfitz

gopherbot pushed a commit to golang/build that referenced this issue Jan 7, 2020
@dmitshur
dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM
05d2263 
Start using n1-highcpu-16 machine type instead of n1-highcpu-8
for windows-amd64-longtest and windows-amd64-race builders.

The primary motivation of this change is to increase amount of
RAM available on the race builder from 7.2 GB to 14.4 GB.
CL 208500 has helped by going from 3.6 to 7.2, but empirically
8 GB is not enough for all tests to pass, while 16 GB can be.
We already use 16 GB for some other race and/or long builders.

Updates golang/go#35186
Updates golang/go#33951
Updates golang/go#33986

Change-Id: I8031a517e6e44ce5e5faba085e7c781a1fcab92e
Reviewed-on: https://go-review.googlesource.com/c/build/+/213557
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@dmitshur
Copy link
Contributor

dmitshur commented Jan 7, 2020 •
edited
Loading

The aforementioned change has had an effect. Both windows-amd64-longtest and -race have been passing on x/tools since the builder change was deployed this morning. I'll keep watching a bit more, and then close this issue if the pattern continues.

There have been instances where x/tools has passed on all builders with tip and on release-branch.go1.13:

image

image

The freebsd-amd64-race builder still fails due to low memory occasionally, but that's outside of scope of this issue. I'll use issue #36444 to track increasing its memory to match Linux and Windows if needed.

@dmitshur dmitshur mentioned this issue Jan 7, 2020
Closed
@dmitshur
Copy link
Contributor

dmitshur commented Jan 13, 2020 •
edited
Loading

This issue is resolved now, based on the current state of the windows-amd64-longtest column on https://build.golang.org/?repo=golang.org%2fx%2ftools:


(Newer windows-amd64-longtest builds are on top.)

@dmitshur dmitshur mentioned this issue Jan 13, 2020
Closed
@golang golang locked and limited conversation to collaborators Jan 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows release-blocker Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
3 participants
@dmitshur @bcmills @gopherbot