Unobserved task exception when server is not reachable

[Callum-Shipton]

What version of gRPC and what language are you using?

Grpc.Net.Client = 2.43.0
Grpc.HealthCheck = 2.43.0

What operating system (Linux, Windows,...) and version?

Windows 10, Build 19041

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Version: 6.0.201

What did you do?

Ran Vigor health check example with no server running (only client), modified to capture UnobservedTaskException

static async Task Main(string[] args)
{
    TaskScheduler.UnobservedTaskException += (sender, e) => {
        Console.WriteLine("Detected unobserved task exception: " + e.Exception);
    };

    using var channel = GrpcChannel.ForAddress("https://localhost:5001");
    var client = new Health.HealthClient(channel);

    Console.WriteLine("Watching health status");

    var watchTask = Task.Run(async () =>
    {
        try
        {
            var response = await client.CheckAsync(new HealthCheckRequest { Service = "" });
            Console.WriteLine($"{DateTime.Now}: Service is {response.Status}");
        }
        catch (Exception)
        {
            Console.WriteLine("Detected expected exception");
        }
    });

    try
    {
        await watchTask;
    }
    catch (Exception)
    {
        Console.Write("Detected unexpected exception from watchTask");
    }


    Console.WriteLine();
    Console.WriteLine("Waiting for finalizers");

    // Provoke the garbage collector to find the unobserved exception.
    GC.Collect();
    // Wait for any failed tasks to be garbage collected
    GC.WaitForPendingFinalizers();

    Console.WriteLine("Finished");
}

What did you expect to see?

I expected to see the only exception thrown from awaiting client.CheckAsync when it was refused by the target machine

Watching health status
Detected expected exception

Waiting for finalizers
Finished

What did you see instead?

I saw the expected exception but also an UnobservedTaskException after GC had collected a task created within client.CheckAsync

Watching health status
Detected expected exception

Waiting for finalizers
Detected unobserved task exception: System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(CancellationToken cancellationToken)"))
 ---> Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(CancellationToken cancellationToken)")
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.PickAsync(PickContext context, Boolean waitForReady, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   at Client.Program.<>c__DisplayClass0_0.<<Main>b__1>d.MoveNext() in C:\Users\Callum.Shipton\source\repos\grpc-dotnet\examples\Vigor\Client\Program.cs:line 48
   --- End of inner exception stack trace ---
Finished

Anything else we should know about?

I have ran this with both unary and streaming calls and they both exhibit the unexpected exception very repeatedly.

I was going to raise this in Grpc.Core but a similar issue was closed (grpc/grpc#22097) for only using Grpc.Net.Client which this also does.
Looks like the issue also affects Grpc.Core (grpc/grpc#24421) but has gone stale
Might also be related to (dotnet/runtime#61411) but I didn't dig deep enough to find out

[JamesNK]

Thanks for the reproduction. It took me a while but I was able to figure out what was going on and have a fix: #1660