RpcException - PurgeAllInstancesAsync operation was canceled - HttpClient.Timeout of 100 seconds elapsing

[CGSK]

When trying to purge all the instances for a specified period of time timeout occurs.

Exception while executing function: Functions.PurgeDurableFunctionsHistory Result: Failure
Exception: System.OperationCanceledException: The PurgeAllInstancesAsync operation was canceled.
---> Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="", DebugException="System.OperationCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.")
---> System.OperationCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: The operation was canceled.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at System.Net.Http.Http2Connection.Http2Stream.<>c.b__88_0(Object s, CancellationToken cancellationToken)
at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout)
--- End of inner exception stack trace ---
at Grpc.Net.Client.Internal.GrpcCall2.ResolveException(String summary, Exception ex, Nullable1& status, Exception& resolvedException)
at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetException(Exception exception, Task1& taskField)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetException(Exception exception, Task1& taskField)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetException(Exception exception, Task1& taskField)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetException(Exception exception, Task1& taskField)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetException(Exception exception, Task1& taskField)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext()
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.SetException(Exception exception, Task1& taskField)
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder1.AsyncStateMachineBox1.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---

--- End of inner exception stack trace ---
at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.PurgeInstancesCoreAsync(PurgeInstancesRequest request, CancellationToken cancellation)
--- End of inner exception stack trace ---
at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.PurgeInstancesCoreAsync(PurgeInstancesRequest request, CancellationToken cancellation)
at MyFuncApp.Functions.App.Doc.PurgeDurableFunctionsHistory.RunAsync(TimerInfo myTimer, DurableTaskClient client, CancellationToken cancellationToken) in C:\Users\VM251KE\source\repos\MyFuncApp\Services\MessageProcessor\MyFuncApp.Functions.App.Doc\PurgeDurableFunctionsHistory.cs:line 63
at MyFuncApp.Functions.App.Doc.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\VM251KE\source\repos\MyFuncApp\Services\MessageProcessor\MyFuncApp.Functions.App.Doc\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 123
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack: at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.PurgeInstancesCoreAsync(PurgeInstancesRequest request, CancellationToken cancellation)
at MyFuncApp.Functions.App.Doc.PurgeDurableFunctionsHistory.RunAsync(TimerInfo myTimer, DurableTaskClient client, CancellationToken cancellationToken) in C:\Users\VM251KE\source\repos\MyFuncApp\Services\MessageProcessor\MyFuncApp.Functions.App.Doc\PurgeDurableFunctionsHistory.cs:line 63
at MyFuncApp.Functions.App.Doc.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\VM251KE\source\repos\MyFuncApp\Services\MessageProcessor\MyFuncApp.Functions.App.Doc\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 123
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

Other Info:
We are using isolated plan
Host.json already has a functionTimeout of 10 hours
We are using azure storage queue provider

Question:
How to increase the Grpc http timeout used by PurgeAllInstancesAsync?

[lilyjma]

@CGSK - could you please provide a simple sample for reproducing this?

[CGSK]

@lilyjma Timeout only happens if we have plenty of instances to delete which is obviously the case in our prod.

[Function("PurgeDurableFunctionsHistory")]
public async Task RunAsync([TimerTrigger("%PurgeDurableFunctionsHistory:Schedule%")] TimerInfo myTimer, [DurableClient] DurableTaskClient client, CancellationToken cancellationToken)
{
PurgeResult result = null;
if (!int.TryParse(Environment.GetEnvironmentVariable("PurgeDurableFunctionsHistoryDays"), out int deletePurgeHistoryDays))
{
deletePurgeHistoryDays = 30;
}

var orchStatus = new List<OrchestrationRuntimeStatus>
{
	OrchestrationRuntimeStatus.Completed,
};
var purgeFilter = new PurgeInstancesFilter(
	DateTime.SpecifyKind(DateTime.MinValue, DateTimeKind.Utc),
	DateTime.SpecifyKind(DateTime.UtcNow.AddDays(deletePurgeHistoryDays * -1), DateTimeKind.Utc),
	orchStatus);

result = await client.PurgeAllInstancesAsync(purgeFilter, cancellationToken).ConfigureAwait(false);

}

[lilyjma]

@CGSK - Could you try the following?

Rather than increasing the timeout, try and split purging into multiple steps, making each step small enough to succeed. For example, purge data older than 12 months, then purge data older than 11 months, eventually reaching the 30 days target. You may need to adjust the numbers. This workaround is more scalable than increasing the timeout.

[CGSK]

@lilyjma In our case, we started a new task hub with just 7 days old data, the purge is failing with timeout. Is there any way we can configure the http or function execution timeout of the actual sidecar process? If PurgeAllInstancesAsync makes a http call from the sidecar process to purge task hub and if the host's timeout configuration is not pushed to sidecar process, this is very constraining. I wonder this sidecar http timeout is specific to DurableTaskClient.PurgeAllInstancesAsync (Grpc) implementation or this is even an issue with any custom http client calls (for ex. using System.Net.Http.HttpClient).

[CGSK]

@lilyjma This is even happening in the latest NuGet version.

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask.SqlServer" Version="1.3.0" />

[ciranmc]

@CGSK @lilyjma Did we ever get a fix or workaround for this? I am facing exactly the same issue when running Azure Function locally but with the WaitForInstanceCompleteAsync method. Seems like a fundamental issue as these methods are designed for long running processes. There has to be some easy way to configure the httpClient timeout via services collection or something. Any help would be greatly appreciated.

[m-mcmaster]

@lilyjma. This still seems to be an issue even after upgrading to 1.2.2.

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.2.2" />

InnerException1: Status(StatusCode="Cancelled", Detail="", DebugException="System.OperationCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.") --> InnerException2: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. --> InnerException3: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. --> InnerException4: The operation was canceled. --> InnerException5: The operation was canceled.

I am generation 50k+ Instance entries each night within the timespan of an hour. Other than making the timespan very, very small or doing the instance iteration myself, how are we supposed to clean up large volumes of entries now?