Profiler can deadlock on OS X #13294
Labels
Comments
|
I started some work to address this on my threading branch |
|
Can we move that work onto master? |
|
@vtjnash Is this fixed? (Or is this still not due to one of your todo's in |
|
no, i think keno wanted us to add a guard around calls to |
Keno
added a commit
that referenced
this issue
Jan 21, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
Keno
added a commit
that referenced
this issue
Jan 21, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
Keno
added a commit
that referenced
this issue
Jan 21, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
Keno
added a commit
that referenced
this issue
Jan 21, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
JeffBezanson
pushed a commit
that referenced
this issue
Jan 28, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
KristofferC
pushed a commit
that referenced
this issue
Apr 11, 2020
The deadlock here happens when the main thread is trying to register a new JIT object while the profiler is triggered. The main thread will acquire the objectmap lock and then get suspended. Then the profiler thread will attempt to stackwalk, but in order to do that effectively, it needs to look at the object map to find the unwind info for the function on the stack. We don't have the same problem on linux because the main thread runs the backtracing. On linux and mac the registration process works differently and on linux at least the backtracing happens on a different thread so we may not have this problem. However, #13294 says that it is also a problem on OS X. We should keep an eye out for it. For the moment, just try to fix this by terminating the stack unwind when we fail to acquire the lock. That's not ideal, because it reduces the quality of the profiler info, but only in situations where we would previous deadlock. This entire code needs a rewrite, but for now, I'm just hoping to get CI to stopping deadlocking on us.
Sacha0
pushed a commit
that referenced
this issue
Oct 23, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If we happen to stop a thread while it is trying to register a dynamic fde, the profiler thread will dead lock trying to acquire a lock held by the blocked thread.
The text was updated successfully, but these errors were encountered: