Breakpoints fail on worker threads

[weswitt]

Running VsCode 1.19.1 & C/C++ Version 0.14.5 on Windows 10. Attempting remote debugging to an Ubuntu 16.04 machine using GDB. Debugging does work, but breakpoints on threads other than the main thread do not work properly. The breakpoint is hit, but when it does so the debugger does not switch threads and set the context correctly. The debugging is useless because of this.

Attempting the same thing locally on the same Ubuntu machine works so this seems to be a remote debugging specific bug. This appears to be a regression as this used to work correctly.

[pieandcakes]

@weswitt So you set a breakpoint, it gets hit on a thread other than main and the debugger doesn't have the right thread in context?

[pieandcakes]

Can you also tell me if you are doing gdb <-> gdbserver or SSH remote to gdb debugging?

[weswitt]

Correct, the thread context is wrong. You can't step or examine locals.

{
"version": "0.2.0",
"configurations": [
{
"name": "Launch",
"type": "cppdbg",
"request": "launch",
"program": "/home/wesw/dev/weswpc/homeauto/server/bin/homeauto",
"args": ["-p", "/home/wesw/dev/weswpc/homeauto/server", "-c", "/home/wesw/dev/weswpc/homeauto/server/homeauto.conf"],
"sourceFileMap": {
"/home/wesw/dev/weswpc/homeauto": "d:/dev/src/homeauto",
"/home/wesw/mosquitto-1.4.10": "s:/dev/mosquitto-1.4.10"
},
"pipeTransport": {
"pipeCwd": "d:/tools/git/usr/bin",
"pipeProgram": "d:/tools/git/usr/bin/ssh.exe",
"pipeArgs": [ "-X", "-i", "d:/tools/ssh-private.key", "wesw@192.168.1.165" ],
"debuggerPath": "/usr/bin/gdb"
},
"cwd": ".",
"linux": { "MIMode": "gdb" },
"setupCommands": [ { "text": "-enable-pretty-printing" } ],
"showDisplayString": true
}
]
}

[pieandcakes]

when you say it is wrong, does it have the wrong stack line selected on the wrong thread or is it not showing which thread is selected? When you say you can't examine locals, what are you seeing?

I tried this with my multithreaded app sample that's in this repo and I am seeing the right thread selected with the right frame and values in my locals window.

You can try this sample app too and see if you see the same behavior: https://github.com/Microsoft/vscode-cpptools/tree/master/Code%20Samples/Fib.

If you don't can you provide me a repro application so I can debug the exact scenario?

[weswitt]

here is a picture of what happens:

[pieandcakes]

@weswitt That picture shows it still is running (isn't in breakmode) so locals/callstack should not show anything. If its supposed to be at a breakpoint, then we aren't getting the right event. If you enable "logging": { "engineLogging": true } and repro and send me your log that would be appreciated.

[weswitt]

The process has stopped at the bp. Look at the debug console. It says hit Breakpoint 3" and I know for a fact that it did because I caused to do so.

[pieandcakes]

If you enable logging, we can see what gdb is sending back to us when the breakpoint is hit. I suspect something that it is sending isn't being processed correctly and therefore we don't know that it is stopped. What is happening here is we don't know that the breakpoint is hit and we aren't notifying VSCode that a breakpoint is hit and it should be in stop mode.

[weswitt]

Thanks you. How do I enable GDB logging?

[pieandcakes]

Add "logging": { "engineLogging": true } to your launch.json.

[weswitt]

I think what is happening is that it is stopping before the BP hits and is not continued:

=thread-selected,id="5"
1: (1649) ->=library-loaded,id="/lib/x86_64-linux-gnu/libnss_files.so.2",target-name="/lib/x86_64-linux-gnu/libnss_files.so.2",host-name="/lib/x86_64-linux-gnu/libnss_files.so.2",symbols-loaded="0",thread-group="i1"
1: (1655) ->"Stopped due to shared library event:\n"
1: (1655) ->" Inferior loaded /lib/x86_64-linux-gnu/libnss_files.so.2\n"
1: (1655) ->*stopped,reason="solib-event",added=[library="/lib/x86_64-linux-gnu/libnss_files.so.2"],thread-id="6",stopped-threads="all",core="1"
Stopped due to shared library event:
Inferior loaded /lib/x86_64-linux-gnu/libnss_files.so.2

[pieandcakes]

that stopped event reason "solib-event" says it is a library load event that caused it to stop. Stopping on library load events should be disabled at this point. Can you provide more of the log?

[weswitt]

Here is more. Coping text from the console is not easy at all. Would be nice if the log could be directed to a file.

1: (1591) ->"[New Thread 0x7ffff0bdc700 (LWP 8038)]\n"
-12-28 17:55:48 MQTT: publishing [r=yes] [WIT=thread-created,id="5",group-id="i1"
1: (1591) ->*running,thread-id="5"
[New Thread 0x7ffff0bdc700 (LWP 8038)]
1: (1591) ->"[Thread 0x7ffff13dd700 (LWP 8037) exited]\n"
1: (1591) ->=thread-exited,id="4",group-id="i1"
[Thread 0x7ffff13dd700 (LWP 8037) exited]
1: (1637) ->"[New Thread 0x7fffeb7fe700 (LWP 8041)]\n"
[New Thread 0x7fffeb7fe700 (LWP 8041)]
1: (1637) ->*running,thread-id="6"
1: (1641) ->"[Switching to Thread 0x7fffeb7fe700 (LWP 8041)]\n"
1: (1641) ->"Stopped due to shared library event (no libraries added or removed)\n"
[Switching to Thread 0x7fffeb7fe700 (LWP 8041)]
1: (1641) ->*stopped,reason="solib-event",thread-id="6",stopped-threads="all",core="1"
Stopped due to shared library event (no libraries added or removed)
1: (1641) <--exec-continue
1: (1645) ->^running
1: (1645) ->*running,thread-id="all"
1: (1645) ->=thread-selected,id="5"
1: (1645) ->(gdb)
=thread-selected,id="5"
1: (1649) ->=library-loaded,id="/lib/x86_64-linux-gnu/libnss_files.so.2",target-name="/lib/x86_64-linux-gnu/libnss_files.so.2",host-name="/lib/x86_64-linux-gnu/libnss_files.so.2",symbols-loaded="0",thread-group="i1"
1: (1655) ->"Stopped due to shared library event:\n"
1: (1655) ->~" Inferior loaded /lib/x86_64-linux-gnu/libnss_files.so.2\n"
1: (1655) ->*stopped,reason="solib-event",added=[library="/lib/x86_64-linux-gnu/libnss_files.so.2"],thread-id="6",stopped-threads="all",core="1"
Stopped due to shared library event:
Inferior loaded /lib/x86_64-linux-gnu/libnss_files.so.2

[weswitt]

Further diagnosis shows that this is caused by the debugee outputting text to STDOUT. It seems that if enough text from other threads than the thread being debugged can cause this. If I disable my apps logging to STDOUT then the debugger behaves correctly. I suspect that when debugging remotely the timing / slowness is different enough from the faster local scenario that this happens.

[pieandcakes]

@weswitt Enabling logging to a file is a little bit more involved. To do it, open ~/.vscode/extensions/ms-vscode.cpptools-0.14.5/package.json and locate the line that says "program": "./debugAdapters/OpenDebugAD7", and add right after it "args": [ "--engineLogging=~/output.log" ],. Save the file and reload VS Code to take the changes. Now when you are done debugging, there should be a log file ~/output.log.

[pieandcakes]

I'm interested in seeing if calls to -gdb-set stop-on-solib-events are in the log as this would cause it to stop on library loads, which it shouldn't do during normal debugging. At this point it looks like we're not expecting this stopping event so it seems to be in a bad state but I don't know how to reproduce it on my side.

When you say you are outputting text to STDOUT, how much text are you generating?

[pieandcakes]

@weswitt i'm still unable to reproduce this. If you can provide me a sample app, please let me know.

[pieandcakes]

Thank you for reporting an issue. We do not have enough information at this time to act upon this issue. Please try our latest version and if it is still occuring, provide more information including configurations, logs and version information and reopen this issue.