runtime: bsdthread_register fatal error (CylancePROTECT) #18171
Comments
|
It will be helpful to know what does bsdthread_register return.
Could you please apply this patch to Go 1.8beta source code and
post what does the new println say?
https://gist.github.com/minux/ef054728f5f9b8e3685b88f5c88d1ad3
(I don't post the patch inline because github and/or Gmail will
reformat the message and make it harder to apply the patch.)
Thanks.
|
|
@minux I think we should just include the return value in error message permanently. That would make it easier to debug in future. if r := bsdthread_register(); r != 0 {
...
throw(fmt.Sprintf("runtime: bsdthread_register error(%d)", r))
}WDYT? |
|
Yeah, we can do that, but I'm still interested in this particular case
because this is obviously system dependent (so far we only got
one such issue, so it's not widespread....)
|
|
Is the go command binary itself crashing? We have built the go binary for 1.8 beta 1 on Macs without problems. I'm wondering what is different about your system. |
|
It may work to try but I don't know that would be necessary. I wonder if there's something on your system intercepting process execution and doing an implicit DYLD insertion. |
|
FWIW, I can generate this panic 100% of the time (on macOS Sierra) using both 1.7.4 and tip by running an executable under dtruss: GO_LDFLAGS=-linkmode=external does not fix it. |
|
dtrace will insert libSystem.dylib into the process, so using dtrace on
non-cgo Go binaries will result in a crash. This is known.
|
|
What does |
|
Register a thread creation callback function to the kernel.
A process can only register one. When cgo is not used (internal linking),
Go doesn't pthread_create to create threads, and registers its own
thread creation callback function.
See https://golang.org/src/runtime/sys_darwin_amd64.s for details.
|
|
We still don't know why @ksaldana1's system is force-inserting libSystem.dylib into ordinary process execution. Maybe we can make all binaries expect libSystem.dylib, but I think it's too late for Go 1.8. |
|
Updating to go 1.8rc1 I've noticed this issue as well on OSX 10.11.6. Reverting back to 1.7.4 does not reproduce the issue. I did not experienced the issue with 1.8beta1, but I did recently have a antivirus software installed on my system within the past few days. Not sure if that is related. |
|
I rebuilt Go from source on both tip and |
|
Just thought I'd chime in, just ran into this as well in the upgrade to 1.8. I was running CylanceProtect and it would throw the same errors. Strangely 1.7.x had no issues. |
|
I'm also running into the issue. I have Cylance installed as well. |
|
I also have the same problem and I have Cylance installed. In case the information is useful: OSX El Capitan |
|
I can confirm that I am having the same error on macOS Sierra, Go 1.8, running Cylance protect. |
|
I managed to get around this problem by asking the IT support to change the Cylance policy to (MAC - BG). |
|
You can try to have your Cylance administrator add the following Memory Protection exclusions: |
|
If you need to pass along to your Cylance administrators, this article indicates the directories that need to be excluded to work around Go 1.8 fatal errors. |
|
I'm curious why this started in Go 1.8. The fact that @jasdel rebuilding from source doesn't have the problem is also interesting, although that didn't help @ksaldana1. I wonder if it means our build has changed in some way. At a glance, I can't see anything different about the Go 1.8 binaries vs the Go 1.7.5 binaries. bin/go should be using cgo in both versions, which should make it not call bsdthread_register at all. Perhaps @jamiseichenauer was over-conservative and the go command does not need to be on the whitelist. tool/compile and tool/link don't use cgo (because they don't have network code in them), and they do try to call bsdthread_register. We can close this issue forever by making all Go binaries built for OS X implicitly import "runtime/cgo" (when CGO_ENABLED=1). Then Go will always use libpthread for thread creation and not call bsdthread_register (again assuming cgo is enabled). |
I don't want to play the game of trying to make antivirus happy. Nobody ever seems to win that game. |
|
@rsc We have a contact with Cylance, anything in particular you'd recommend reaching out to them about? |
|
@polds, thanks for the offer. Before we ping Cylance, can people having this problem try the following? I've attached go-issue-18171.zip with six binaries, three from Go 1.7.6 and three from Go 1.8.2. @andredasilvapinto, @j-mroz, @polds, anyone else having problems, can you download the zip file and then run: and post the resulting shell transcript? On my non-Cylanced system, I just get help messages. But the expectation is that under Cylance some of these binaries will fail. Specifically, I expect gofmt1.8.2 to fail but not any of the others. Confirming or disproving that assumption would be useful for asking good questions of Cylance. Thanks. |
|
So of interesting note, I just installed Cylance on a VM (had to disable it since it's been messing with Go) I did not see the errors in Cylance 1.0.x. When I upgraded to Version 2.0.x it started throwing those errors. @rsc Here is the output of those commands, successful results stripped (you were right about gofmt failing): |
|
@rsc Here is my output (OS X El Cap 10.11.6 with Cylance 1.2.1418.609): Appears that |
|
@polds, thanks, OK, so then here's my question for Cylance. As background, on the Mac, C programs using threads use libSystem.dylib to interact with the operating system to manage those threads. Go programs that call out to C libraries do the same. But Go programs that don't call out to C libraries do everything themselves, to avoid any dependencies on C when not needed. Using networking on a Mac means calling a C library, so on a Mac it's the basic file I/O-only Go programs that don't use the usual C threads. This has been true from Go 1.0 and before. As more background, we also know from experience that some Mac programs force dynamic loading of libraries ahead of ordinary program start, including libSystem.dylib. For example if you run under dtruss, you can't avoid having libSystem.dylib. If libSystem.dylib is forced into a Go binary expecting to manage threads itself, the two do not get along: Go's bsdthread_register system call fails, causing output like the errors above. We've also seen this with interposition software that sets the DYLD_INSERT_LIBRARIES environment variable. Now, about Cylance. The failing gofmt binary from Go 1.8.2 (gofmt1.8.2 in go-issue-18171.zip) is behaving for all the world like Cylance is doing some kind of early dynamic library loading that is registering for creating threads before Go can. The question is why gofmt from Go 1.7.6 (gofmt1.7.6 in the same zip file) works fine. On the Go side, nothing about this arrangement changed from Go 1.7 to Go 1.8. Does Cylance somehow know about or detect Go binaries and avoid doing the interposition, and perhaps the detector recognizes Go 1.7 binaries but not Go 1.8 binaries? Is there any advice Cylance would give us on what we can do to make our newer binaries be treated like the Go 1.7 binaries are treated? An alternative long-term fix would be to use libSystem.dylib even for non-C-using Go binaries, but that's technically quite difficult for us and wouldn't be something we could do retroactively to Go 1.8 or even the upcoming Go 1.9. Thanks very much. |
|
By the way, gofmt works if I run it via lldb for some reason, doesn't crash ... |
|
I've done a git bisect on golang repository using all.bash script to check for the issue 276803d is the first bad commit :040000 040000 621194da05c8a090f3aadb7b36a5d2769384f20a 75b4f6cac52c869353962f88f72a568a86348573 M src |
|
So, my contact no longer works at Cylance, I'm going through alternative "slow channels" but not having high hopes. |
|
I wonder if Go should just use libSystem.dylib always on macOS. I would
imagine that doing anything else is not supported by Apple.
…On May 31, 2017 11:52 AM, "Peter Olds" ***@***.***> wrote:
So, my contact no longer works at Cylance, I'm going through alternative
"slow channels" but not having high hopes.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#18171 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AGGWBzhOBjLmUfjplknF4Yt1THhjUlOLks5r_YytgaJpZM4LDD37>
.
|
|
Moving sections SELFROSECT and SMACHOPLT to their old position after the SPCLNTAB fixes the issue with bsdthread_register. But not all test have passed, for some reason that I haven't checked net/http test failed. I've used latest commit form master that I had as a base 4aa5d2e. Why on earth that would change anything? |
|
@DemiMarie, I agree we might want to do that in the long term, but it's tricky to bootstrap (see #18171 (comment) and #18171 (comment) above). @j-mroz, thanks very much for finding that. Can you send a CL, even if it breaks net/http? Maybe someone else can figure out what's wrong with net/http. Also network tests on macOS are flaky under some configurations, apparently due to problems with the macOS network stack. So your change might be fine. Thanks everyone. |
|
CL https://golang.org/cl/44870 mentions this issue. |
|
I've also experienced net and log/syslog tests failure - "signal: broken pipe", but then it all worked. Seems to be random, and not reproducible all the time. Anyway, I've posted a CL as you suggested . |
|
CC @crawshaw |
|
One of our contacts at Cylance has been working on this issue, and has targeted the 1450 release for resolving it. I just got added the Early Adopter release for the 1450... Preliminary results: this issue hasn't appeared when running 1.8.3 with CylancePROTECT 2.0.1.1450.505 🎊 |
|
FWIW: El Capitan and go 1.8.3 with CylancePROTECT (now at 2.0.1450.506) not producing the error where this same machine was producing it before a Cylance update. |
|
Thanks so much @ascondment, @bru7us, and the engineers at Cylance who got this fixed. |
|
@rsc @abscondment @bru7us I am experiencing the same issue but do not have the update mentioned for cylance. Mac OSX Sierra 10.12.5 Do you know when this update will get pushed to Sierra? |
|
@nilator83 1450 was released on June 20th, you may need to ask your Cylance admin to enable it for your zone |
What version of Go are you using (
go version)?go version go1.8beta1
What operating system and processor architecture are you using (
go env)?What did you do?
Any call to go build/tools fails with
What did you expect to see?
Working builds.
I have gone through all threads related to this error and have tried their solutions (disabling anti-virus, messing with DYLD environment variables, adding -ldflags -linkmode=external to build) and am still running into this issue.
Note, I am running an old version of OSX -- Yosemite v10.10.5.
I have tried both building from source and using the installer. Same effect either way.
Downgrading to 1.7.4 does not run into this error.
The text was updated successfully, but these errors were encountered: