cmd/go: remove -buildmode=shared (not c-shared)

[rsc]

-buildmode=shared has been broken since modules, and it is apparently unused.
See #47257 (comment).
Let's remove it.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[electricface]

I really want to use this buildmode=shared mode.

[scott-cotton]

@rsc how has it been broken since modules?

[ainar-g]

Just to clarify, this is not related to --buildmode=plugin, right?

[ianlancetaylor]

@electricface Why do you want to use it? Are you using it today?

@scott-cotton You can't use -buildmode=shared with a module and then use -linkshared to link against it from other modules. The whole idea of -buildmode=shared doesn't fit very well with modules. Each separate program can be linked against different versions of dependent modules, so if it did work people would in practice wind up with many different versions of the shared libraries of the same modules.

@ainar-g Correct, this is not about -buildmode=plugin, although plugins have their own, different, problems.

[electricface]

@ianlancetaylor
I am a debian-baseed Linux Desktop Environment developer. Our team have developed lot of daemon programs with golang. But these programs have significantly larger binary file size and use more ram than C or C++ programs. We currently working on these two problems. An idea is making some modules pluggable, which means only load modules that we need.

As other C or C++ programs are using shared-library, we are researching on some shared-library-like mechanism of golang. Inspired by buildmode=plugin and goloader, we found a way that bases on buildmode=shared to make modules pluggable recently, you can found a demo here. This solution has not been using in production environment, because of some problems hasn't been solved yet. I found out that programs built with -linkeshared can not be debugged using gdb and dlv.

After this commit, our "unoffical plugin" solution can work fine with go 1.14.9~1.16.7. Programs built in this way can run stably. But this won't work on go 1.17. Program will exit with a "fatal error: unreachable method called. linker bug?"

Plan of remove buildmode=shared make me feel bad.

[ianlancetaylor]

@electricface It sounds like you want plugins, which are supported (though not very well) by -buildmode=plugin. We aren't going to support -buildmode=shared just to support plugins, since we already have -buildmode=plugin.

[electricface]

@ianlancetaylor
I know that using static linking can eliminate the problems caused by the mismatched version of the dependent dynamic library.
But in the linux desktop system ecosystem, C or C++ languages are mostly used. These languages generally use dynamic libraries to save memory. The main reason for not using -buildmode=plugin is that static linking cannot save memory. For example, if the modules are divided into smaller ones and a lot of modules are added, each module occupies a relatively large amount of memory, and the overall occupancy will eventually become unacceptably large. Compared with C or C++ languages, they only use less memory to implement these functions, while the golang implementation uses several times the memory.
There is even an idea in our team that we must change the development language if we can't reduce the memory. I personally like to use golang for development, so I plan to study this part of the technology.

[rsc]

"I want to use it" is different from "I am using it today".
We believe that it is not possible to use it today, and furthermore it is difficult to provide.
We shouldn't advertise something that doesn't actually work.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[mewmew]

Just to confirm, this proposal suggests to remove --buildmode=shared while keeping --buildmode=c-shared? I am using --buildmode=c-shared for a number of projects and would be very sad to see it go.

[ianlancetaylor]

This proposal is only about -buildmode=shared.

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[rasky]

@rsc not sure if this is a good place to give feedback about the proposal process, but this is the first time (since the proposal process exists) that I missed an issue. I was mildly interested in this (not enough now to vehemently protest), but even if I'm subscribed to the proposal process issue and I normally read minutes every week, I missed this. I took 10 days off for vacation, and those 10 days covered the only 2 emails where this issue was mentioned, before today when it was approved. Again, I don't mind about this specific issue at this point but I wanted to raise a general point.

Maybe a proposal should stay a minimum time in the proposal process before being closed, so that even people that don't work on Go full time have time to notice and react even with personal life happening in-between.

[zhouguangyuan0718]

To limit binary file size, I use command go install -buildmode=shared std to install libstd.so. And I build my application by go build -linkshared. I have been using this in production environment for a long time. I hope I can still use it in the furture.
Can we keep -buildmode=shared std, it's useful.

[Jason7602]

Our company has been using the shared mode to reduce the binary size for a long time. The plugin mode is not useful in some scenarios cause the entire runtime package needs to be compiled and packed. So we strongly hope that shared mode can be reserved.

[kiap]

@ianlancetaylor

Some folks contacted me from ITF USA
Saying the GPL / LGPL problem with Go can easily be fixed in the most conservative way:

Sending messages from process A (EULA or any license) to process B (GPL licensed), embracing 100% separation

I guess that's doable,

Thanks for @ianlancetaylor 's help and other folks for their input.

[pjanx]

@rsc Is there any issue or discussion to track about your -o proposal here?

My so far hypothetical use case is a multiprocess architecture, with a bunch of small helper programs that possibly depend on C libraries and can be installed separately. Again, it's about size.

I get a feeling that Linux distributions could make use of a "go-rt" package for "std" and link their stuff against that. It would save a bit of bandwidth for utilities.

[paralin]

I've been working on plugin loading and splitting components cleanly (and automatically) across multiple IPC-connected processes, the code is available here: controller-bus.

It could sure use some improvement on the c-shared plugin loading side, (although I know that's unrelated to this issue)

[laoshaw]

After reading this whole thread it's not clear to me about -buildmode=shared 's future. Can I summarize it as:

1. plugin and c-shared will stay.
2. shared will be limited to use cases as below only:

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

[ianlancetaylor]

@laoshaw There has not been a final decision but what you describe is likely. More precisely, it's likely that we will only permit a single -buildmode=shared library to be included in a build; it doesn't necessarily have to be the Go standard library although that will presumably be the common case.

[mlaventure]

Just wanted to check on the status of this. Since the issue is closed, is there another issue/location where I can follow the evolution of this? I don't see it explicitly in the 1.19 beta 1 change log so I'm guessing at best it'll be for 1.20?

[ianlancetaylor]

@mlaventure I think the status is that we developed an idea (a single shared library) but stalled out on implementing it. Nothing has changed yet and as far as I know there is no new proposal.

[mlaventure]

@ianlancetaylor thanks for the update. Is there somewhere I can track the progress on this? In the meantime we'll just have to transition to a different language for our systems with stricter memory constraints unfortunately.

[ianlancetaylor]

@mlaventure I don't know of a place to track progress. Sorry.

You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

[mlaventure]

@mlaventure I don't know of a place to track progress. Sorry.

You should change to a different language if it makes sense for you, of course, but I don't understand the comment. The existing -buildmode=shared continues to work for now.

Unfortunately it doesn't work for me, the compiler panics when I try to build a package with -buildmode=shared -linkshared:

panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
cmd/link/internal/loader.(*Loader).LoadSyms(0xc00079a000, 0x670a56)
	/usr/local/go/src/cmd/link/internal/loader/loader.go:2210 +0x9f
cmd/link/internal/ld.(*Link).loadlib(0xc0000da000)
	/usr/local/go/src/cmd/link/internal/ld/lib.go:577 +0x46e
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0x0, 0x0}, {0x679c5b, ...}, ...})
	/usr/local/go/src/cmd/link/internal/ld/main.go:249 +0xd8b
main.main()
	/usr/local/go/src/cmd/link/main.go:69 +0x1005

This is using go 1.17.11.

[ianlancetaylor]

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

[bcmills]

That is #47455, which could perhaps be reopened — but, to be honest, I would be surprised if it gets much debugging effort until we figure out the future of -buildmode=shared.

[mlaventure]

You wouldn't normally use both -buildmode=shared and -linkshared together. Anyhow perhaps that should be a different issues if there isn't already one open for it. Thanks.

It used to be the only way I had found to build a shared library and avoid the /usr/local/go/pkg/tool/linux_amd64/link: cannot implicitly include runtime/cgo in a shared library error. I think there was an issue for it around go 1.7, but the issue seems to still be there. The packages I am building do not depend on CGO.

[gopherbot]

Change https://go.dev/cl/417194 mentions this issue: misc/cgo/testshared: run tests only in GOPATH mode

[omenos]

I've been looking at -linkshared application building recently to see if it's a viable solution for Linux distribution packaging of Go software. The main workflow concept is this:

• Distributions provide a -buildmode=shared package that includes libstd.so (in Fedora this is golang-shared).

  GOROOT=$(pwd) PATH=$(pwd)/bin:$PATH go install -buildmode=shared -v -x std
  

• Applications are built with -linkshared to link against this global libstd.so for the Go standard library.

The idea behind this is to enable updates to Go within the distribution without requiring mass rebuilds of applications due to stdlib patches/fixes. It has the side benefit of creating smaller binaries, but the gains are dependent on other factors as well (e.g. minimal dependencies). However, I do not fully understand the Go build process and the way the compiler works to know if this is even a feasible goal.

• As it stands right now this isn't possible because Go attempts to rebuild libstd.so on each build run, which doesn't lend confidence to a stable shared object being produced, and users building software as non-root generally do not have write access to the global GOROOT.
• I've been told that it's possible for things to change between patch releases of Go (e.g. 1.19.0->1.19.1) that might introduce reliability issues in this process.
• Dependencies using assembly may run into issues
  • For example, trying to build terraform:

    $ GOROOT=/tmp/golang go build -o ./bin/terraform -mod=vendor -linkshared .
    # github.com/klauspost/compress/zstd/internal/xxhash
    asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00092 (/home/mroche/Documents/code/tools/terraform/vendor/github.com/klauspost/compress/zstd/internal/xxhash/xxhash_amd64.s:120)ADDQ	R15, AX
    asm: assembly failed
    

• If my understanding is correct, this wouldn't remove the need to rebuild applications if there's a security fix to the compiler itself, only the stdlib.

Does this sound like a use case that could be (or maybe shouldn't be at all) supported by Go tooling?

[bcmills]

@omenos, to my knowledge the Go compiler doesn't currently have any notion of “ABI boundaries”, and in particular it can do inlining of function bodies across packages. So there is a substantial risk of subtle bugs when picking up changes to the standard library and toolchain — particularly if a function fixed in a patch release was inlined into the caller when it was compiled.

[gizahNL]

Distributions might choose to do

go build -buildmode=shared -o libgo1.17.2.so std golang.org/x/...
go build -linkshared=libgo1.17.2.so cmd1
go build -linkshared=libgo1.17.2.so cmd2
go build -linkshared=libgo1.17.2.so cmd3

This sounds like a great solution. I know I'm late to the game and perhaps it's a bit much to ask ;) What would really help in my use case (single repo/module many separate applications) is a build command that can build multiple binaries in one go, and intelligently bundles all shared dependencies (or just anything not in their respective main packages) into a shared library.
So basically your solution, with automation so the risk of operator mistake is limited.

[logrusorgru]

+1 for shared libraries support with module aware versions

I'm using plugins with their dependencies. And they are very fat. I just want to move the dependencies to a shared library. And I don't want to care about a compatibility etc.

Also, root installation is not acceptable for me too. Currently I'm using separate Go compiler (unlike /usr/lib/go).

P.S. only linux + amd64, don't care any other OS or architecture