cmd/go: cannot link with CGO_ENABLED=1 on mips64

[matoro]

What version of Go are you using (go version)?

$ go version
go version go1.19.1 linux/mips64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="mips64"
GOBIN=""
GOCACHE="/home/matoro/.cache/go-build"
GOENV="/home/matoro/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="mips64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/matoro/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/matoro/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_mips64"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
GOMIPS64="hardfloat"
AR="ar"
CC="mips64-unknown-linux-gnu-gcc"
CXX="mips64-unknown-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD="/home/matoro/go-sqlite3/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -mabi=64 -mhard-float -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1657790444=/tmp/go-build -gno-record-gcc-switches"

What did you do?

go is unable to bootstrap itself, nor link any binaries when CGO_ENABLED=1. No problems with CGO_ENABLED=0. It errors out with thousands of errors like this, one for every single symbol:

+ ./cmd/dist/dist bootstrap -a
Building Go toolchain1 using /usr/lib/go-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/mips64.
# cmd/trace
/var/tmp/portage/dev-lang/go-1.19.1/work/go/pkg/tool/linux_mips64/link: running mips64-unknown-linux-gnu-gcc failed: exit status 1
/var/tmp/portage/dev-lang/go-1.19.1/temp/go-link-3757955373/go.o: in function `internal/cpu.Initialize':
/usr/lib/go/src/internal/cpu/cpu.go:123:(.text+0x14): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -f
PIC
/usr/lib/go/src/internal/cpu/cpu.go:124:(.text+0x30): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with 
-fPIC
/var/tmp/portage/dev-lang/go-1.19.1/temp/go-link-3757955373/go.o: in function `internal/cpu.Initialize':
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0x40): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile 
with -fPIC
/var/tmp/portage/dev-lang/go-1.19.1/temp/go-link-3757955373/go.o: in function `internal/cpu.doinit':
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:23:(.text+0x54): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompil
e with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:23:(.text+0x64): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompil
e with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0x78): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompil
e with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0x84): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompil
e with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0x90): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompil
e with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0xa4): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompil
e with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0xb8): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:22:(.text+0xc4): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:27:(.text+0xcc): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu_mips64x.go:27:(.text+0xe4): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/var/tmp/portage/dev-lang/go-1.19.1/temp/go-link-3757955373/go.o: in function `internal/cpu.Initialize':
/usr/lib/go/src/internal/cpu/cpu.go:125:(.text+0x100): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/var/tmp/portage/dev-lang/go-1.19.1/temp/go-link-3757955373/go.o: in function `internal/cpu.processOptions':
/usr/lib/go/src/internal/cpu/cpu.go:148:(.text+0x130): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:180:(.text+0x34c): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:187:(.text+0x368): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x384): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x38c): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x3a4): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x3bc): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x3c4): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x3dc): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x3f4): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x3fc): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x414): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:175:(.text+0x41c): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:163:(.text+0x43c): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:163:(.text+0x444): relocation R_MIPS_HI16 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:163:(.text+0x45c): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC
/usr/lib/go/src/internal/cpu/cpu.go:163:(.text+0x474): relocation R_MIPS_26 against `a local symbol' cannot be used when making a shared object; recompile with -fPIC

Attached are two logs, the first from trying to bootstrap itself (which is where the above series of errors comes from) and the second from attempting to build the go-sqlite tests. Building the package succeeds (because it's just a static library) but building the tests fails because they have dynamic executables that need to link.

o00p.log.gz
go-sqlite-logs.log

System toolchain being used: mips64-unknown-linux-gnu-gcc 12.2.0, binutils 2.39, kernel 5.19, big-endian n64 ABI.

Shell access to this machine can be provided on request!

[cherrymui]

cannot be used when making a shared object

This error is weird. We are building an executable, not a shared object...

What C linker do you use and what version? Does setting CGO_LDFLAGS=-no-pie or CGO_LDFLAGS=-Wl,-no-pie help?

[matoro]

cannot be used when making a shared object

This error is weird. We are building an executable, not a shared object...

What C linker do you use and what version? Does setting CGO_LDFLAGS=-no-pie or CGO_LDFLAGS=-Wl,-no-pie help?

This is with bfd from binutils-2.39. I can try that, but another curious thing I've discovered is that the compile time shoots up almost 10x with CGO_ENABLED=1.

Time to compile (and hit this error) with CGO_ENABLED=1: ~8 hours
Time to compile with CGO_ENABLED=0: ~45 mins

[matoro]

cannot be used when making a shared object

This error is weird. We are building an executable, not a shared object...

What C linker do you use and what version? Does setting CGO_LDFLAGS=-no-pie or CGO_LDFLAGS=-Wl,-no-pie help?

@cherrymui Your comment about the linker made me curious if that might be it, and it turned out it was! Reverting to binutils-2.38 resolves the issue, so this is a regression in binutils-2.39. I will report it upstream there.

Also, for reference, here are the exact compile times:

CGO_ENABLED=0: 2022-09-15T15:29:59 >>> dev-lang/go-1.19.1: 43 minutes, 40 seconds
CGO_ENABLED=1: 2022-09-17T15:20:07 >>> dev-lang/go-1.19.1: 7 hours, 13 minutes, 36 seconds

That is twice as long as it takes to compile Rust from source on the same hardware.

2022-09-15T10:13:58 >>> dev-lang/rust-1.63.0-r1: 4 hours, 10 minutes, 25 seconds

But since that's only incidental to the ticket and this is technically identified as a binutils issue, I will close.

[matoro]

FYI, I bisected this to bminor/binutils-gdb@65daf5b and opened a bug in binutils upstream at https://sourceware.org/bugzilla/show_bug.cgi?id=29592.