ANSI color code interpretation broken in terminal

[oliversalzburg]

• VSCode Version: Code 1.14.2 (cb82feb, 2017-07-19T23:34:09.706Z)
• OS Version: Windows_NT ia32 10.0.15063
• Extensions:

Extension	Author (truncated)	Version
vscode-eslint	dba	1.2.11
pgsql	dou	0.0.8
EditorConfig	Edi	0.9.4
JSDocTagComplete	Hoo	0.0.2
vscode-esformatter	how	1.5.2
output-colorizer	IBM	0.1.2
vscode-puglint	mrm	2.3.0

---

Steps to Reproduce:

When I issue certain commands in the terminal, color code interpretation seems to break at certain points in the output:

I don't know what it is specifically about the output that breaks the interpreter, but I can reproduce this at will with the CLI application I am using there.

To my knowledge, this worked previously and this works fine in other terminals.

[Tyriar]

@oliversalzburg can you setup a small project/set of files that demonstrates the issue?

[oliversalzburg]

I'm having a really hard time isolating the issue due to how complex the scenario appears to be. However this is where I'm at right now: https://github.com/oliversalzburg/vscode-31326

To give some insight into what is happening. release will spawn git to create a commit. git will try to sign the commit with gpg2. gpg2 will ask gpg-agent for the key, which then spawns pinentry, a GUI application, to ask me for the password.

Now, if you don't have gpg signing of commits enabled, you won't see the issue.

Simple output to the console will also not break the terminal. For example, when git generated output and it gets printed to the console by the simple-git wrapper.

What is especially interesting here is that all git output is captured and the You need a passphrase... string is actually from gpg-agent. I have no idea how it is able to write to the console. But, when it does, it breaks the VS Code terminal.

[Tyriar]

@rprichard any ideas why this would be happening?

[rprichard]

My guess: node.js turns ENABLE_VIRTUAL_TERMINAL_PROCESSING on, then sends escape sequences to the console to set the color. node.js runs gpg, which I'm guessing is an MSYS2/Cygwin-based program. The Cygwin runtime sets the console mode for various reasons, and incidentally turns ENABLE_VIRTUAL_TERMINAL_PROCESSING back off.

[rprichard]

@oliversalzburg Does this issue reproduce in an ordinary Windows console? e.g. using Start -> Run, run this command:

c:\msys64\usr\bin\bash.exe --login

Don't use mintty, and don't use ConEmu. If you have ConEmu configured to handle all new console windows, you may have to disable that option temporarily.

I see the same behavior if I run this JavaScript code (Windows 10 v2 console, 10.0.10563, with MSYS2 gpg.exe in the PATH):

// FWIW, I'm using node v8.1.0
const { spawnSync } = require("child_process");
process.stdout.write("\x1b[32m" + "color" + "\x1b[0m" + "\n");
spawnSync("gpg.exe", ["--foo-bar"], {stdio: "inherit"});
process.stdout.write("\x1b[32m" + "color" + "\x1b[0m" + "\n");

FWIW, the standalone ConoutMode test program in the winpty repository might be helpful for debugging this issue. It can show whether ENABLE_VIRTUAL_TERMINAL_PROCESSING flag is enabled, or set/unset the flag. It might not behave as expected if you run it from cmd.exe / powershell.exe, though, because those shells modify ENABLE_VIRTUAL_TERMINAL_PROCESSING themselves.

Here are some comments I made the last time I looked into this VT mode flag: rprichard/winpty#92 (comment).

[oliversalzburg]

@rprichard Seems like you're right and this is not specific to VS Code.

When I run the command from cmd.exe, I see the same issue:

No idea why it works in ConEmu and I could have sworn this worked fine in VS Code previously. I even ran into a similar issue today, which happens when I cancel running unit tests:

All of this seems new to me. But maybe a Windows 10 update is to blame?

[rprichard]

No idea why it works in ConEmu ...

By default, ConEmu injects a hook DLL into processes attached to its hidden underlying console (which it calls the "RealConsole"). The ConEmu DLL overrides functions like WriteFile and WriteConsole{A,W} to interpret VT escape sequences. This ConEmu ability preceded the official Win10 v2 console support for VT escapes, which is controlled by the ENABLE_VIRTUAL_TERMINAL_PROCESSING mode flag. My guess is that the ability is unaffected by the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag.

... I could have sworn this worked fine in VS Code previously. ... But maybe a Windows 10 update is to blame?

Lots of possibilities:

• The ENABLE_VIRTUAL_TERMINAL_PROCESSING flag only works with the Win10 v2 console. It's an error to enable the flag with the v1 legacy console, which is enabled by checking "Use legacy console (requires relaunch)" in the console properties dialog.
• IIRC, the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag was added in 10.0.10586 and wasn't present in the initial 10.0.10240 release.
• The behavior of both cmd.exe and powershell.exe has changed w.r.t. the flag. (For cmd, see https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/15617610--re-enable-enable-virtual-terminal-processing-by. For powershell, see Win10 New Console: Enable ENABLE_VIRTUAL_TERMINAL_PROCESSING by default (or with a flag) rprichard/winpty#92 (comment) and Use correct win32 console mode w/ external commands PowerShell/PowerShell#2991.) (I'm guessing this one doesn't apply, but maybe it would if node.js used cmd.exe when it invoked gpg.)
• Someone had to add ENABLE_VIRTUAL_TERMINAL_PROCESSING support to node.js. I looked over the libuv src/win/tty.c code, and it looks like it does its own VT escape processing if it can't enable ENABLE_VIRTUAL_TERMINAL_PROCESSING. If you use an older node.js, then escape processing will work regardless of the ENABLE_VIRTUAL_TERMINAL_PROCESSING setting.

[Tyriar]

Closing as it looks to be at the Windows level 😃