Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

interrupt triggering SIGILL #74

Closed
srenatus opened this issue Apr 27, 2021 · 7 comments
Closed

interrupt triggering SIGILL #74

srenatus opened this issue Apr 27, 2021 · 7 comments

Comments

@srenatus
Copy link
Contributor

srenatus commented Apr 27, 2021

👉 main.go, go.sum and go.mod are here

Running go run main.go, iteration 388 gives me

2021/04/27 21:02:57 387: trap: wasm trap: interrupt
wasm backtrace:
    0:   0x21 - <unknown>!<wasm function 0>
SIGILL: illegal instruction
PC=0x33f6001a m=3 sigcode=1
instruction bytes: 0xf 0xb 0x55 0x48 0x89 0xe5 0xff 0xd2 0x48 0x89 0xec 0x5d 0xc3 0x0 0x0 0x0

goroutine 0 [idle]:
runtime: unknown pc 0x33f6001a
stack: frame={sp:0x70000e951a10, fp:0x0} stack=[0x70000e8d2350,0x70000e951f50)
000070000e951910:  0000000000000000  0000000000000000
000070000e951920:  0000000000000000  0000000000000000
000070000e951930:  0000000000000000  0000000000000000
000070000e951940:  0000000000000000  0000000000000000
000070000e951950:  0000000000000000  0000000000000000
000070000e951960:  0000000000000000  0000000000000000
000070000e951970:  0000000000000000  0000000000000000
000070000e951980:  0000000000000000  0000000000000000
000070000e951990:  0000000000000000  0000000000000000
000070000e9519a0:  0000000000000000  0000000000000000
000070000e9519b0:  0000000000000000  0000000000000000
000070000e9519c0:  0000000000000000  0000000000000000
000070000e9519d0:  0000000000000000  0000000000000000
000070000e9519e0:  0000000000000000  0000000000000000
000070000e9519f0:  0000000000000000  0000000000000000
000070000e951a00:  0000000000000000  0000000000000000
000070000e951a10: <000070000e951a20  0000000033f60022
000070000e951a20:  000070000e951b00  00000000047b5ccc
000070000e951a30:  0000000000000000  50878a5e3aac6a03
000070000e951a40:  50878a5e3aac6b33  000070000e951b50
000070000e951a50:  000070000e951a30  000070000e951cd0
000070000e951a60:  00000000043bf9e0  5087fa5e30422dbd
000070000e951a70:  0000000000000000  0000037f00001f80
000070000e951a80:  0000000000000000  0000000000000000
000070000e951a90:  0000000000000000  0000000000000000
000070000e951aa0:  000000c0001de8a0  000000c0001db898
000070000e951ab0:  0000000000000000  000070000e951b18
000070000e951ac0:  0000000005fd3790  000070000e951cd0
000070000e951ad0:  5831d1b433c600b4  000070000e951cd0
000070000e951ae0:  0000000000000008  000070000e951da8
000070000e951af0:  000070000e951d20  000070000e951b50
000070000e951b00:  000070000e951bc0  00000000043bfa9f
runtime: unknown pc 0x33f6001a
stack: frame={sp:0x70000e951a10, fp:0x0} stack=[0x70000e8d2350,0x70000e951f50)
000070000e951910:  0000000000000000  0000000000000000
000070000e951920:  0000000000000000  0000000000000000
000070000e951930:  0000000000000000  0000000000000000
000070000e951940:  0000000000000000  0000000000000000
000070000e951950:  0000000000000000  0000000000000000
000070000e951960:  0000000000000000  0000000000000000
000070000e951970:  0000000000000000  0000000000000000
000070000e951980:  0000000000000000  0000000000000000
000070000e951990:  0000000000000000  0000000000000000
000070000e9519a0:  0000000000000000  0000000000000000
000070000e9519b0:  0000000000000000  0000000000000000
000070000e9519c0:  0000000000000000  0000000000000000
000070000e9519d0:  0000000000000000  0000000000000000
000070000e9519e0:  0000000000000000  0000000000000000
000070000e9519f0:  0000000000000000  0000000000000000
000070000e951a00:  0000000000000000  0000000000000000
000070000e951a10: <000070000e951a20  0000000033f60022
000070000e951a20:  000070000e951b00  00000000047b5ccc
000070000e951a30:  0000000000000000  50878a5e3aac6a03
000070000e951a40:  50878a5e3aac6b33  000070000e951b50
000070000e951a50:  000070000e951a30  000070000e951cd0
000070000e951a60:  00000000043bf9e0  5087fa5e30422dbd
000070000e951a70:  0000000000000000  0000037f00001f80
000070000e951a80:  0000000000000000  0000000000000000
000070000e951a90:  0000000000000000  0000000000000000
000070000e951aa0:  000000c0001de8a0  000000c0001db898
000070000e951ab0:  0000000000000000  000070000e951b18
000070000e951ac0:  0000000005fd3790  000070000e951cd0
000070000e951ad0:  5831d1b433c600b4  000070000e951cd0
000070000e951ae0:  0000000000000008  000070000e951da8
000070000e951af0:  000070000e951d20  000070000e951b50
000070000e951b00:  000070000e951bc0  00000000043bfa9f

goroutine 1 [syscall]:
runtime.cgocall(0x40b7740, 0xc00019bcf0, 0xc0001de8a0)
        /nix/store/2dgrrcr9z6571yygbzwcds3ss1vid4sk-go-1.16/share/go/src/runtime/cgocall.go:154 +0x5b fp=0xc00019bcc0 sp=0xc00019bc88 pc=0x400407b
github.com/bytecodealliance/wasmtime-go._Cfunc_go_wasmtime_func_call(0x5fd4db0, 0xc0001de890, 0xc0001de8a0, 0xc0001db898, 0x0)
        _cgo_gotypes.go:715 +0x49 fp=0xc00019bcf0 sp=0xc00019bcc0 pc=0x40abf29
github.com/bytecodealliance/wasmtime-go.(*Func).Call.func3.1(0xc0000a5c60, 0xc0001de890, 0xc0001de8a0, 0xc0001db898, 0xc38b091e922abf01)
        /Users/stephan/go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.26.0/func.go:505 +0x117 fp=0xc00019bd30 sp=0xc00019bcf0 pc=0x40b44f7
github.com/bytecodealliance/wasmtime-go.(*Func).Call.func3(0xc0001db898)
        /Users/stephan/go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.26.0/func.go:505 +0x4f fp=0xc00019bd70 sp=0xc00019bd30 pc=0x40b456f
github.com/bytecodealliance/wasmtime-go.enterWasm(0xc0000b8900, 0xc00019be98, 0x0)
        /Users/stephan/go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.26.0/func.go:589 +0xf0 fp=0xc00019bdd0 sp=0xc00019bd70 pc=0x40b0630
github.com/bytecodealliance/wasmtime-go.(*Func).Call(0xc0000a5c60, 0x0, 0x0, 0x0, 0x0, 0xc0000b0d50, 0x0, 0x0)
        /Users/stephan/go/pkg/mod/github.com/bytecodealliance/wasmtime-go@v0.26.0/func.go:500 +0x27d fp=0xc00019bed0 sp=0xc00019bdd0 pc=0x40aff7d
main.main()
        /Users/stephan/Scratch/wasmtime/sigill/main.go:35 +0x1f4 fp=0xc00019bf88 sp=0xc00019bed0 pc=0x40b6954
runtime.main()
        /nix/store/2dgrrcr9z6571yygbzwcds3ss1vid4sk-go-1.16/share/go/src/runtime/proc.go:225 +0x256 fp=0xc00019bfe0 sp=0xc00019bf88 pc=0x4037356
runtime.goexit()
        /nix/store/2dgrrcr9z6571yygbzwcds3ss1vid4sk-go-1.16/share/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00019bfe8 sp=0xc00019bfe0 pc=0x4066d21

rax    0xffffffffffff7fff
rbx    0x0
rcx    0x8
rdx    0x33f60000
rdi    0xffffffffffff7fff
rsi    0x5fd30b0
rbp    0x70000e951a10
rsp    0x70000e951a10
r8     0x33f6001c
r9     0xc0001db898
r10    0xc0001de8a0
r11    0x0
r12    0x70000e951b50
r13    0x70000e951a30
r14    0x70000e951cd0
r15    0x43bf9e0
rip    0x33f6001a
rflags 0x10246
cs     0x2b
fs     0x0
gs     0x0
exit status 2

Some environment information,

  • macos big sur 11.2.3
  • go version go1.16 darwin/amd64

I'll gladly supply more information, or do some guided poking -- I'm afraid I don't really know where to start. 😅


Follow-up to #60.

@alexcrichton
Copy link
Member

Ok thanks! I tried this again and let it run in the background and indeed got the fault eventually. I mostly try to investigate these sorts of issues by tracking them down in a debugger if I can, and afterwards just thinking hard about them. (unfortunately these are often pretty gnarly to investigate)

That being said after thinking about this and seeing it in a debugger I think I know the issue of what's going on here. The problem is that wasmtime needs to initialize trap handlers per-thread and while it's doing that for the first thread the Go runtime is presumably migrating the main goroutine to a different OS thread and wasmtime isn't re-initializing (currently that's an unsafe operation in Rust and requires a manual opt-in to initialize it).

I'll look into fixing this in Wasmtime in the near future!

alexcrichton added a commit to alexcrichton/wasmtime that referenced this issue Apr 28, 2021
Platforms Wasmtime supports may have per-thread initialization that
needs to run before WebAssembly. For example Unix needs to setup a
sigaltstack and macOS needs to set up mach ports. In bytecodealliance#2757 this
per-thread setup was moved out of the invocation of a wasm function,
relying on the lack of Send for Store to initialize the thread at Store
creation time and never worry about it later.

This conflicted with [wasmtime's desired multithreading
story](bytecodealliance#2812) so a new
[`Store::notify_switched_thread` was
added](bytecodealliance#2822) to
explicitly indicate a Store has moved to another thread (if it unsafely
did so).

It turns out though that it's not always easy to determine when a
`Store` moves to a new thread. For example the Go bindings for Wasmtime
are generally unaware when a goroutine switches OS threads. This led to
bytecodealliance/wasmtime-go#74 where a SIGILL
was left uncaught, making it appear that traps aren't working properly.

This commit revisits the decision in bytecodealliance#2757 and moves per-thread
initialization back into the path of calling into WebAssembly. This is
differently from before, though, where there's still only one TLS access
on the path of calling into WebAssembly, unlike before where it was a
separate access. This allows us to get the speed benefits of bytecodealliance#2757 as
well as the flexibility benefits of not having to explicitly move a
store between threads.

With this new ability this commit deletes the recently added
`Store::notify_switched_thread` method since it's no longer necessary.
@alexcrichton
Copy link
Member

I've verified that bytecodealliance/wasmtime#2863 fixes the issue on my end at least

@srenatus
Copy link
Contributor Author

Thanks a lot, both for fixing this so quickly and providing insights. 😃

alexcrichton added a commit to bytecodealliance/wasmtime that referenced this issue Apr 28, 2021
* Bring back per-thread lazy initialization

Platforms Wasmtime supports may have per-thread initialization that
needs to run before WebAssembly. For example Unix needs to setup a
sigaltstack and macOS needs to set up mach ports. In #2757 this
per-thread setup was moved out of the invocation of a wasm function,
relying on the lack of Send for Store to initialize the thread at Store
creation time and never worry about it later.

This conflicted with [wasmtime's desired multithreading
story](#2812) so a new
[`Store::notify_switched_thread` was
added](#2822) to
explicitly indicate a Store has moved to another thread (if it unsafely
did so).

It turns out though that it's not always easy to determine when a
`Store` moves to a new thread. For example the Go bindings for Wasmtime
are generally unaware when a goroutine switches OS threads. This led to
bytecodealliance/wasmtime-go#74 where a SIGILL
was left uncaught, making it appear that traps aren't working properly.

This commit revisits the decision in #2757 and moves per-thread
initialization back into the path of calling into WebAssembly. This is
differently from before, though, where there's still only one TLS access
on the path of calling into WebAssembly, unlike before where it was a
separate access. This allows us to get the speed benefits of #2757 as
well as the flexibility benefits of not having to explicitly move a
store between threads.

With this new ability this commit deletes the recently added
`Store::notify_switched_thread` method since it's no longer necessary.

* Fix a test compiling
@alexcrichton
Copy link
Member

Ok with bytecodealliance/wasmtime#2863 merged I'm gonna close this and this'll get into the packages on the next wasmtime version bump which should be soon-ish

@srenatus
Copy link
Contributor Author

srenatus commented May 4, 2021

Heya. Thanks again for fixing this!

Are there any plans for a release including this? 😄

@alexcrichton
Copy link
Member

Ah I was waiting on a wasmtime version for this but I realized I can publish 0.26.1, so I've pushed that up.

@srenatus
Copy link
Contributor Author

srenatus commented May 4, 2021

Thank you! It looks great on my end. Thanks again 👏

mchesser pushed a commit to mchesser/wasmtime that referenced this issue May 24, 2021
* Bring back per-thread lazy initialization

Platforms Wasmtime supports may have per-thread initialization that
needs to run before WebAssembly. For example Unix needs to setup a
sigaltstack and macOS needs to set up mach ports. In bytecodealliance#2757 this
per-thread setup was moved out of the invocation of a wasm function,
relying on the lack of Send for Store to initialize the thread at Store
creation time and never worry about it later.

This conflicted with [wasmtime's desired multithreading
story](bytecodealliance#2812) so a new
[`Store::notify_switched_thread` was
added](bytecodealliance#2822) to
explicitly indicate a Store has moved to another thread (if it unsafely
did so).

It turns out though that it's not always easy to determine when a
`Store` moves to a new thread. For example the Go bindings for Wasmtime
are generally unaware when a goroutine switches OS threads. This led to
bytecodealliance/wasmtime-go#74 where a SIGILL
was left uncaught, making it appear that traps aren't working properly.

This commit revisits the decision in bytecodealliance#2757 and moves per-thread
initialization back into the path of calling into WebAssembly. This is
differently from before, though, where there's still only one TLS access
on the path of calling into WebAssembly, unlike before where it was a
separate access. This allows us to get the speed benefits of bytecodealliance#2757 as
well as the flexibility benefits of not having to explicitly move a
store between threads.

With this new ability this commit deletes the recently added
`Store::notify_switched_thread` method since it's no longer necessary.

* Fix a test compiling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants