Memory Usage

[ShaneCourtrille]

@jtkech @sebastienros We're dealing with an extremely high memory usage problem and I'm starting to look at core dumps to figure out why System.String and System.Byte[] instances are taking so much space. I was wondering if you could look at the worst of the core dump from below and tell me if anything else seems overly high considering we only have ~380 active tenants. We have a lot more tenants that aren't being accessed so they shouldn't be loaded if I'm not mistaken?

00007f073462ac30   553894    101916496 OrchardCore.DisplayManagement.Descriptors.ShapeDescriptorIndex
00007f07339c79b0  1308888    104711040 System.Threading.Tasks.Task`1[[Microsoft.AspNetCore.Mvc.Razor.Compilation.CompiledViewDescriptor, Microsoft.AspNetCore.Mvc.Razor]]
00007f0733ff9e28  2621437    105496392 System.ValueTuple`3[[System.String, System.Private.CoreLib],[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]
00007f073395db48   596417    114512064 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelBindingMessageProvider
00007f07324548a0  1530050    122404000 GraphQL.Types.FieldType
00007f072ab5c758    11857    128565446 System.Char[]
00007f0733ec7ac0  2701704    131673784 Microsoft.AspNetCore.Routing.Patterns.RoutePatternPart[]
00007f0733ee5640  4409352    141099264 Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor
00007f072ad769f0  2932893    164242008 Microsoft.Extensions.DependencyInjection.ServiceDescriptor
00007f0733993c98   663918    169963008 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadata
00007f0730107cc0   593599    178735088 Microsoft.Extensions.DependencyInjection.ServiceDescriptor[]
00007f0733ec66b8  1896533    182067168 Microsoft.AspNetCore.Routing.Patterns.RoutePattern
00007f0730b47018     2984    184403456 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory+ServiceDescriptorCacheItem, Microsoft.Extensions.DependencyInjection]][]
00007f0733ffd880  1598062    217350096 Microsoft.AspNetCore.Routing.Matching.Candidate[]
00007f072b2e1d10  4746272    350762368 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
00007f0733fef580  7987282    383389536 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]]
00007f072ab18080  6139275    385820040 System.Int32[]
00007f072aa452a8 18066459    433595016 System.Object
00007f0733fef630  1722425    515843344 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]][]
00007f072aa4b110  6107112    681516720 System.Object[]
00007f072ab1d2e0 14743936   1273771010 System.String
00007f072ab7e7f0  2837388   1942957181 System.Byte[]
00005601997a1f50    10208   3341303608      Free

[jtkech]

We're dealing with an extremely high memory usage problem

How much?

and I'm starting to look at core dumps to figure out why System.String and System.Byte[] instances are taking so much space. I was wondering if you could look at the worst of the core dump from below and tell me if anything else seems overly high considering we only have ~380 active tenants.

Yes tenants are isolated sites, we share for example shape descriptors, but they have their own routes, they run in containers having their own services and so on. That said, nothing so high that I can see from the above dump for 380 tenants.

We have a lot more tenants that aren't being accessed so they shouldn't be loaded if I'm not mistaken?

Unless they have been requested at least once. You may need to implement a tenant auto release, meaning each non default tenant can release itself after a certain amount of inactivity, e.g. through a tenant background task, something that we started to support in #13633.

That said, even if a tenant is released, it still holds its settings having a Name, TenantId, VersionId and so on. How many tenants do you have? Maybe you need to distribute them across multiple servers/instances, each one serving a limited set of tenants, this is the goal of #13633.

Also be sure you are not in debug mode, the GC may behave differently, or in dev mode e.g. if the ASPNETCORE_ENVIRONMENT env var is set to Development.