Elasticsearch configuration from recipe occasionally throws System.ObjectDisposedException during site setup

[porgabi]

Describe the bug

When the Elasticsearch feature is enabled and its configuration is run in a setup recipe, sometimes the logs contain the following exception after site setup:

2024-07-30 15:25:41.9221|Default|https://localhost/|00-cabdda5987d6837cfe2013beb2030800-b4467ad8d6782fdc-00||OrchardCore.Environment.Shell.Scope.ShellScope|ERROR|Error while executing the background job 'elastic-index-rebuild' after the end of the request on tenant 'Default'. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'IServiceProvider'.
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.OrchardCoreBuilderExtensions.<>c.<AddDataAccess>b__0_4(ShellScope scope)
   at OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync()
   at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute, Boolean activateShell)
   at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute, Boolean activateShell)
   at OrchardCore.Search.Elasticsearch.Core.Services.ElasticIndexingService.ProcessContentItemsAsync(String indexName)
   at OrchardCore.Search.Elasticsearch.Core.Recipes.ElasticIndexRebuildStep.<>c__DisplayClass0_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at OrchardCore.BackgroundJobs.HttpBackgroundJob.<>c__DisplayClass0_0.<<ExecuteAfterEndOfRequestAsync>b__1>d.MoveNext()    at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.OrchardCoreBuilderExtensions.<>c.<AddDataAccess>b__0_4(ShellScope scope)
   at OrchardCore.Environment.Shell.Scope.ShellScope.BeforeDisposeAsync()
   at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute, Boolean activateShell)
   at OrchardCore.Environment.Shell.Scope.ShellScope.UsingAsync(Func`2 execute, Boolean activateShell)
   at OrchardCore.Search.Elasticsearch.Core.Services.ElasticIndexingService.ProcessContentItemsAsync(String indexName)
   at OrchardCore.Search.Elasticsearch.Core.Recipes.ElasticIndexRebuildStep.<>c__DisplayClass0_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at OrchardCore.BackgroundJobs.HttpBackgroundJob.<>c__DisplayClass0_0.<<ExecuteAfterEndOfRequestAsync>b__1>d.MoveNext()

This seems to happen only when the elastic-index-rebuild step is run after content items have been created, and it doesn't happen every single time either. Depending on the environment, it can happen frequently or pretty much none at all, so it's hard to pinpoint what the exact cause of the problem may be. The fact that this kept happening on VMs suggests it could be a performance-related issue.

Orchard Core version

1.8.3

To Reproduce

Steps to (maybe) reproduce the behavior:

1. Add Elasticsearch configuration (steps such as ElasticIndexSettings, Settings that contain ElasticSettings, elastic-index-rebuild, and Queries) to a setup recipe.
2. Run the aforementioned configuration after creating some content items in a previous step.

Expected behavior

There should be either no such exception or a consistent appearance of the exception.

[github-actions]

Thank you for submitting your first issue, awesome! 🚀 We're thrilled to receive your input. If you haven't completed the template yet, please take a moment to do so. This ensures that we fully understand your feature request or bug report. On what happens next, see the docs.

If you like Orchard Core, please star our repo and join our community channels.

[Piedone]

Do you have a workaround, perhaps?

[github-actions]

We triaged this issue and set the milestone according to the priority we think is appropriate (see the docs on how we triage and prioritize issues).

This indicates when the core team may start working on it. However, if you'd like to contribute, we'd warmly welcome you to do that anytime. See our guide on contributions here.

[porgabi]

Do you have a workaround, perhaps?

As far as I have seen, this issue can be avoided by ensuring in the recipe that the content item creation steps run after the Elasticsearch config ones.

[Piedone]

You've only seen this in a setup recipe, not otherwise, right?

[porgabi]

Yes.

[gvkries]

Exactly the same error can no longer occur in main, because the ElasticIndexingService no longer uses a separate ShellScope. That doesn't mean a similar bug doesn't still exist, but I haven't been able to find it yet.