Crash when trying to load an Assembly in a hostfxr environment after calling FreeConsole()

[WerWolv]

Description

Hi
I'm currently experiencing a crash in one of my applications which uses the hostfxr environment to load .NET Assemblies. The application is generally a GUI application, however we use the CONSOLE subsystem to have support for a command line interface. When run from the regular explorer, the application detects this and calls FreeConsole() to hide the console window.

This works fine for the most part, however we noticed that FreeConsole() apparently leaves the stdout, stderr and stdin handles in an invalid state which later on causes a crash here when the JIT tries to duplicate the stdout handle. The code there seems to do some checks already but doesn't seem to catch this problem.

Reproduction Steps

The latest nightly release of ImHex triggers this issue at launch on Windows 10 sometimes. It's not consistent but it happens often.
It can be downloaded from https://imhex.download/#nightly

Alternatively, a simple application which uses the CONSOLE subsystem, calls FreeConsole() and then loads an Assembly through the hostfxr functions should reproduce the issue on Windows 10.

Expected behavior

stdout should simply not be used in this case instead of causing a crash

Actual behavior

The call to _dup() at

runtime/src/coreclr/jit/ee_il_dll.cpp

Line 92 in de0ab15

int jitstdoutFd = _dup(_fileno(procstdout()));

causes a abort to happen.

Regression?

No response

Known Workarounds

Interestingly enough, this issue seems to only affect Windows 10 (and potentially even older versions of Windows). It does not happen on Windows 11.

Configuration

• .NET 7.0.400
• Windows 10
• x64
• It's specific to Windows 10 (and potentially before), not Windows 11

Other information

• My application:
  https://github.com/WerWolv/ImHex
• The code that calls FreeConsole(): https://github.com/WerWolv/ImHex/blob/master/main/source/window/win_window.cpp#L238-L251
• The code that initializes the .NET runtime and loads an Assembly: https://github.com/WerWolv/ImHex/blob/master/plugins/script_loader/source/loaders/dotnet/dotnet_loader.cpp#L156
• Stacktrace:
  

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hi
I'm currently experiencing a crash in one of my applications which uses the hostfxr environment to load .NET Assemblies. The application is generally a GUI application, however we use the CONSOLE subsystem to have support for a command line interface. When run from the regular explorer, the application detects this and calls FreeConsole() to hide the console window.

This works fine for the most part, however we noticed that FreeConsole() apparently leaves the stdout, stderr and stdin handles in an invalid state which later on causes a crash here when the JIT tries to duplicate the stdout handle. The code there seems to do some checks already but doesn't seem to catch this problem.

Reproduction Steps

The latest nightly release of ImHex triggers this issue at launch on Windows 10 sometimes. It's not consistent but it happens often.
It can be downloaded from https://imhex.download/#nightly

Alternatively, a simple application which uses the CONSOLE subsystem, calls FreeConsole() and then loads an Assembly through the hostfxr functions should reproduce the issue on Windows 10.

Expected behavior

stdout should simply not be used in this case instead of causing a crash

Actual behavior

The call to _dup() at

runtime/src/coreclr/jit/ee_il_dll.cpp

Line 92 in de0ab15

int jitstdoutFd = _dup(_fileno(procstdout()));

causes a abort to happen.

Regression?

No response

Known Workarounds

Interestingly enough, this issue seems to only affect Windows 10 (and potentially even older versions of Windows). It does not happen on Windows 11.

Configuration

• .NET 7.0.400
• Windows 10
• x64
• It's specific to Windows 10 (and potentially before), not Windows 11

Other information

• My application:
  https://github.com/WerWolv/ImHex
• The code that calls FreeConsole(): https://github.com/WerWolv/ImHex/blob/master/main/source/window/win_window.cpp#L238-L251
• The code that initializes the .NET runtime and loads an Assembly: https://github.com/WerWolv/ImHex/blob/master/plugins/script_loader/source/loaders/dotnet/dotnet_loader.cpp#L156
• Stacktrace:
  

Author:	WerWolv
Assignees:	-
Labels:	area-CodeGen-coreclr, untriaged, needs-area-label
Milestone:	-

[jkotas]

Does the release build of the JIT need to do this I/O initialization unconditionally during startup? It is unnecessary work unless there needs to be something printed to the console.

[jakobbotsch]

Can you see what the value of _fileno(procstdout()) is when the crash happens? It looks like there is already some error detection of the value above, so I suppose the value is not one of the detected cases (or there is some race condition, likely the code should reuse stdoutFd).

Does the release build of the JIT need to do this I/O initialization unconditionally during startup? It is unnecessary work unless there needs to be something printed to the console.

It would presumably be possible to refactor it to be initialized lazily, though there are a number of places we make use of jitstdout. I don't know how expensive this initialization is compared to the rest of the initialization.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[BruceForstall]

I marked it .NET 9 for now, but we can adjust that when we get a repro and/or fix.

[PerikiyoXD]

Can you see what the value of _fileno(procstdout()) is when the crash happens?

Invalid bytes. It can be any value. Actually, I can't debug it as the variables are optimized away, says VS.
Although, stdoutFd equals 1, thus entering the if ((stdoutFd != -1) && (stdoutFd != -2) && (errno != EINVAL)) condition and trying to duplicate the file descriptor (Which is INVALID).

I guessed that there's a race condition too. Must be noted that we launch plugins on paralell to the execution.

Quick summary of the situation

In this case we're under a IMAGE_SUBSYSTEM_WINDOWS_CUI executable process.

This crash is probably caused by FreeConsole.

FreeConsole call does close the standard streams.

We're getting a crash trying to load hostfxr.

hostfxr tries to duplicate a file descriptor, through a closed handle.

There are times (just like this) where a CUI/CLI application wants to detach from the console by using FreeConsole and that invalidates the streams that were used on hostfxr initialization.

Note

This is kind of a python and pythonw dilemma in the end, but definitely there should be a safe-check of the std handles when accessing and duplicating handles.

[PerikiyoXD]

It would definitely need a safe check on procstdout()

[jakobbotsch]

So are you saying procstdout() is returning nullptr or some other failure value?

This code runs just once at start up, so a race condition seems unlikely if you can readily reproduce it.

[jkotas]

So are you saying procstdout() is returning nullptr or some other failure value?

procstdout returns a FILE* that points to closed OS handle. Attempt to duplicate the closed OS handle fails.

[jakobbotsch]

procstdout returns a FILE* that points to closed OS handle. Attempt to duplicate the closed OS handle fails.

Does it mean there is no safe way to use stdout from the VM and JIT? From MSDN on _fileno it does not sound like FreeConsole should close stdout:

If stdout or stderr is not associated with an output stream (for example, in a Windows application without a console window), the file descriptor returned is -2. In previous versions, the file descriptor returned was -1. This change allows applications to distinguish this condition from an error.

There's also this SO post that seems related, but doesn't give much insight: https://stackoverflow.com/questions/60833724/allocconsole-and-stdin-stdout

[jkotas]

Does it mean there is no safe way to use stdout from the VM and JIT?

If some other component in the process closes the underlying OS handle for standard output, using stdout is not going to work. That is by design. Nothing to fix there.

This issue highlighted that the JIT initialization duplicates OS file handles that are not going to be used in 99.99% cases. I think it is something we should fix. OS handles and OS syscalls are not free.

[jakobbotsch]

If some other component in the process closes the underlying OS handle for standard output, using stdout is not going to work. That is by design. Nothing to fix there.

This issue highlighted that the JIT initialization duplicates OS file handles that are not going to be used in 99.99% cases. I think it is something we should fix. OS handles and OS syscalls are not free.

I agree that we should change it to be lazily initialized. In practice it'll fix this issue too. Still, I am curious about the underlying issue here, and how we should be detecting whether or not stdout is a valid handle that can be used.
Even if we make initialization lazy the JIT is still going to crash the process if DOTNET_JitDisasm is turned on for one of these processes.

[WerWolv]

As far as I understand it, the issue is caused by FreeConsole() only closing the underlying kernel handles and not the file descriptors themselves, therefor leaving them in an undefined state. Normally this shouldn't be possible.

I found this post on StackOverflow that mostly describes the issue and how to work around it, however some of the code seems to be missing and some functions aren't available to us since we're using MinGW: https://stackoverflow.com/questions/12676312/freeconsole-behaviour-on-windows-8

[jkotas]

the issue is caused by FreeConsole() only closing the underlying kernel handles

I believe that this issue was fixed in more recent Windows versions.

how we should be detecting whether or not stdout is a valid handle

There is no way to reliably detect this. If the original handle was closed, the handle value could have been later reused for a different object.

[WerWolv]

I believe that this issue was fixed in more recent Windows versions.

This is absolutely possible yeah. I was never able to reporduce the issue on Windows 11 but it's pretty consistent when using a Windows 10 VM.

There is no way to reliably detect this. If the original handle was closed, the handle value could have been later reused for a different object.

I don't know how viable this is but maybe it would make sense to be able to pass in a stream that is used then when calling into the hostfxr interface. And just default to stdout if none is passed in.

[jakobbotsch]

I don't know how viable this is but maybe it would make sense to be able to pass in a stream that is used then when calling into the hostfxr interface. And just default to stdout if none is passed in.

The optimization @jkotas suggested above will in practice fix the issue: we can avoid any I/O operations until the JIT actually needs the handle. The JIT does not need to do anything with stdout in normal scenarios -- only when certain environment variables are set.