Error: "The entity type 'DataProtectionKey' requires a primary key" when using .Net 7 previews

[Webreaper]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I have implemented DataProtectionKey storage using an EFCore DB context, and it has been working fine for nearly a year.

In June, I upgraded my application to .Net 7 preview 5 and EFCore 7 preview 5, and now see the following exception:

Migrations failed with exception: System.InvalidOperationException: The entity type 'DataProtectionKey' requires a primary key to be defined. If you intended to use a keyless entity type, call 'HasNoKey' in 'OnModelCreating'. For more information on keyless entity types, see https://go.microsoft.com/fwlink/?linkid=2141943.

The webserver then terminates within IHost.Run with the following (similar) exception:

[22:03:19.707-0001-INF] Damselfly Webserver terminated with exception: The entity type 'DataProtectionKey' requires a primary key to be defined. If you intended to use a keyless entity type, call 'HasNoKey' in 'OnModelCreating'. For more information on keyless entity types, see https://go.microsoft.com/fwlink/?linkid=2141943.

I am using the SQLite DB provider.

This only happens on Linux (deployed in an Ubuntu docker container). If I run the same code, against the same .db file, on OSX, the migrations complete without issue. I have not tried Windows.

The error suggests using the [Keyless] attribute to resolve this error, but I don't know enough about DataProtectionKeys to know if it would cause any issues.

Note that the problem persists with .Net 7 preview 6 and EFCore preview 6.

Expected Behavior

The expected behaviour is that the db.Database.Migrate(); call completes without issue and the application starts normally.

Steps To Reproduce

I declare my DataProtectionKeys in my DbContext here: https://github.com/Webreaper/Damselfly/blob/develop/Damselfly.Core/Models/ImageContext.cs#L43

public DbSet<DataProtectionKey> DataProtectionKeys { get; set; }

I add configure the DataProtection here: https://github.com/Webreaper/Damselfly/blob/master/Damselfly.Web/Startup.cs#L73

services.AddDataProtection().PersistKeysToDbContext<ImageContext>();

I can't find anywhere in the documentation that suggests I need to do something explicit to create a key.

The DataProtectionKeys were added in 2021, in this Migration: https://github.com/Webreaper/Damselfly/blob/master/Damselfly.Migrations.Sqlite/Migrations/20210922112630_AddDataProtectionKeys.cs:

   modelBuilder.Entity("Microsoft.AspNetCore.DataProtection.EntityFrameworkCore.DataProtectionKey", b =>
                {
                    b.Property<int>("Id")
                        .ValueGeneratedOnAdd()
                        .HasColumnType("INTEGER");

                    b.Property<string>("FriendlyName")
                        .HasColumnType("TEXT");

                    b.Property<string>("Xml")
                        .HasColumnType("TEXT");

                    b.HasKey("Id");

                    b.ToTable("DataProtectionKeys");
                });

Exceptions (if any)

Migrations failed with exception: System.InvalidOperationException: The entity type 'DataProtectionKey' requires a primary key to be defined. If you intended to use a keyless entity type, call 'HasNoKey' in 'OnModelCreating'. For more information on keyless entity types, see https://go.microsoft.com/fwlink/?linkid=2141943.
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelValidator.ValidateNonNullPrimaryKeys(IModel model, IDiagnosticsLogger`1 logger)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelValidator.Validate(IModel model, IDiagnosticsLogger`1 logger)
   at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelValidator.Validate(IModel model, IDiagnosticsLogger`1 logger)
   at Microsoft.EntityFrameworkCore.Sqlite.Infrastructure.Internal.SqliteModelValidator.Validate(IModel model, IDiagnosticsLogger`1 logger)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelRuntimeInitializer.Initialize(IModel model, Boolean designTime, IDiagnosticsLogger`1 validationLogger)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelSource.GetModel(DbContext context, ModelCreationDependencies modelCreationDependencies, Boolean designTime)
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.CreateModel(Boolean designTime)
   at Microsoft.EntityFrameworkCore.Internal.DbContextServices.get_Model()
   at Microsoft.EntityFrameworkCore.Infrastructure.EntityFrameworkServicesBuilder.<>c.<TryAddCoreServices>b__8_4(IServiceProvider p)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite, RuntimeResolverContext, ServiceProviderEngineScope, RuntimeResolverLock)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite, RuntimeResolverContext, ServiceProviderEngineScope, RuntimeResolverLock)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite, RuntimeResolverContext, ServiceProviderEngineScope, RuntimeResolverLock)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite, RuntimeResolverContext, ServiceProviderEngineScope, RuntimeResolverLock)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite, RuntimeResolverContext, ServiceProviderEngineScope, RuntimeResolverLock)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite, RuntimeResolverContext, ServiceProviderEngineScope, RuntimeResolverLock)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitScopeCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type, ServiceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider)

.NET Version

.Net 7 preview 5 and preview 6

Anything else?

The issue only happens when running the containerised deployment on Linux. Running exactly the same code on OSX works fine.

[Webreaper]

Thanks for moving. I originally went to open this in EFCore, but then decided it was more of a aspnetcore issue. 🤣

Just to sanity check, this morning I ran dotnet ef migrations add to see if there was some fundamental change within the DataProtectionKeys structure that might be teased out into a migration to fix this issue, but (as probably expected) it didn't have any effect.

[Webreaper]

Okay, so additional info. I've managed to workaround this by explicitly adding the key with my OnModelCreating method:

   protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            base.OnModelCreating(modelBuilder);

            // Potential fix for https://github.com/dotnet/aspnetcore/issues/43187
            var dpk = modelBuilder.Entity<DataProtectionKey>();
            dpk.HasKey(x => x.Id);

With this, the app starts up correctly, as the key has been correctly added to the entity.

So I guess the questions are:

1. Should developers need to explicitly declare the key in their OnModelCreating code? if so, I've never seen that in the documentation for that function (e.g., it's not mentioned in the Configure ASP.NET Core Data Protection docs).
2. Why does it work without the explicit key definition on OSX? Is there some attribute introspection that's failing on Linux (or in Docker) that's affecting the entity key declaration? I wondered if something was being stripped out as part of the single-file publishing/trimming process, but I've tried turning off /p:PublishSingleFile=true /p:PublishTrimmed=true just in case, and it didn't seem to make a difference.

Let me know if you need any more details.

[ajcvickers]

@Webreaper Sounds like this could be related to trimming. I suspect when the Id property is not referenced directly, then on platforms that turn on trimming by default it is getting trimmed out.

[Webreaper]

Yes, it could be feasible. I do deploy as a single-file executable, and usually have trimming etc enabled:

# dotnet publish Damselfly.Web -r $runtime -f net${dotnetversion} -c Release --self-contained true /p:Version=$version /p:PublishSingleFile=true /p:PublishTrimmed=true /p:IncludeNativeLibrariesForSelfExtract=true

but I disabled trimming to see if it would have an effect (or would fix this problem).

Also, I had trimming disabled for Preview 5 due to a related bug (dotnet/runtime#70758) but disabling trimming doesn't prevent this issue from happening.

[ajcvickers]

@Webreaper Are you sure Microsoft.AspNetCore.DataProtection.EntityFrameworkCore is not being trimmed?

[Webreaper]

If trimming is not enabled in the build/publish, how would it be trimmed? Or is there some magic I'm missing?

[ajcvickers]

I don't know enough about trimming to answer, but I suspect it is worth you investigating.

[Webreaper]

Maybe @richlander might have an idea?

[Webreaper]

Also, if this was a trimming error, it wouldn't explain why it doesn't happen before EFCore 7 preview 5.

[ajcvickers]

@Webreaper What happens if you remove this line:

 dpk.HasKey(x => x.Id);

And replace it with:

Console.WriteLine(new DataProtectionKey().Id);

Or anything else that references the Id property without configuring it in EF model? Does the code still work like it does when using dpk.HasKey(x => x.Id);? If so, then that implies that just referencing the Id property is enough to have EF find it. If it doesn't then, we're going to have to start reducing the complexity in the repro until we can figure out the root cause.

[Webreaper]

Sure. Will give it a try.

[richlander]

Can you share the dotnet build command you are using and any relevant properties ion your project file?

[Webreaper]

Here's the dotnet publish command:

https://github.com/Webreaper/Damselfly/blob/develop/scripts/makeserver.sh#L33

It evaluates to:

dotnet publish Damselfly.Web -r linux-x64 -f net7.0 -c Release --self-contained true /p:Version=3.1.0 
/p:EnableCompressionInSingleFile=true /p:PublishSingleFile=true /p:PublishTrimmed=true 
/p:IncludeNativeLibrariesForSelfExtract=true

I have tried removing the trimming, as follows, but it didn't prevent the error:

dotnet publish Damselfly.Web -r linux-x64 -f net7.0 -c Release --self-contained true /p:Version=3.1.0 
/p:PublishSingleFile=true /p:IncludeNativeLibrariesForSelfExtract=true

I also have Directory.Build.props, and here's the project file which doesn't have anything fancy in terms of properties. Let me know if I can point you at anything else.

Note that this commit seemed to fix the problem per earlier comments. So it's not a huge issue, but it's somewhat confusing as I don't know why it should be needed (I would presume that DataProtectionKey.Id is already explicitly marked with [Key]).

[richlander]

Do you get the error just the following command?

dotnet publish Damselfly.Web -r linux-x64 -f net7.0 -c Release

[Webreaper]

Will try. Can't remember if my deployment scripts will break if it's not a single file deployment, so I might have to tweak.

[Webreaper]

Okay, so that's confirmed. This breaks:
'''
dotnet publish Damselfly.Web -r linux-x64 -f net7.0 -c Release --self-contained true /p:Version=3.1.0
/p:PublishSingleFile=true /p:IncludeNativeLibrariesForSelfExtract=true
'''
This works:

dotnet publish Damselfly.Web -r linux-x64 -f net7.0 -c Release

So it is to do with single-file deployment and/or trimming. What combination do you want me to try next?

[Webreaper]

Okay, so adding self-contained true works.

Then adding /p:PublishSingleFile=true, and it's also fine.

Adding /p:PublishTrimmed=true causes it to break. So it looks like it is related to trimming.

[richlander]

Perfect. That's what we wanted to find out. Let's see if we can get some help.

@agocke @vitek-karas

[ajcvickers]

I think the behavior here is correct. The application is mapping a type (DataProtectionKey) from one of our libraries (Microsoft.AspNetCore.DataProtection) as an entity type in the application's model. This is a very unusual thing to do, but in this case is not totally unreasonable because DataProtectionKey could conceptually be considered an entity type.

From a trimming perspective, we don't have any knowledge that the Id property is going to be mapped by convention by EF Core, and hence trimming it out is reasonable since it is never explicitly used anywhere in the code. This can be fixed by explicitly mapping that the Id property as a key, which causes it to be referenced and not trimmed.

[Webreaper]

Makes sense. I guess the thing that confuses me is that this issue doesn't happen at with .Net 6, and (I'm pretty certain, but can confirm if you need to) that it didn't happen with the .Net 7 before preview 5. So that implies a change in behaviour that might confuse people when 7 GA is released.

Perhaps as you guys are turning the screws on the trimming efficiency it's resulted in the change? But possibly you might be able to put some hints to stop it getting trimmed? After all, I'm not doing anything explicit with the Id field, I'm just using the standard EFCore Data Protection Keys functions - which made this error super confusing from my perspective at first because it's not even referenced within my app - it's something internal to Asp.Net identity.

I assumed that in Aspnetcore Identity this property is already being explicitly mapped as a key....

Edit: just realised that last comment may have been directed at Rich and the identity folks, not me. 😊

[roji]

I guess the thing that confuses me is that this issue doesn't happen at with .Net 6, and (I'm pretty certain, but can confirm if you need to) that it didn't happen with the .Net 7 before preview 5. So that implies a change in behaviour that might confuse people when 7 GA is released.

That is likely because of this change in .NET 7.

[richlander]

this can be fixed by explicitly mapping that the Id property as a key

That was directed at you @Webreaper.

[Webreaper]

Got it.

I'm fine with this and the fix works well. I'm just wondering if there's any hints or docs you could put together because other people will get caught by this; while the fix to add the reference is straightforward, it might catch people unawares to have to add a HasKey call for a model they may not even be aware they're using. Although hopefully googling this issue will help, or I can write an SO question and answer for it.... 😁

[Webreaper]

Oh, and I think the "on Linux" part may be misleading; I may not see it on Mac because that's my Dev env so I don't usually publish as a single file there. I'll test it when I get 5 mins, and if it's consistent I'll change the issue title to "when publishing trimmed" so finding the issue and solution is easier for anyone who is hit by it.

[roji]

We do plan to make the experience better when trimming. For example, we could annotate EF APIs to preserve all public properties on entity CLR types. We're not sure we'll be able to do significant trimming work for 7.0, but at least some guidance/documentation does make sense.

[Webreaper]

Additional info on the cause of this here, for anyone landing on this issue: https://devblogs.microsoft.com/dotnet/announcing-dotnet-7-preview-7/#trimming-and-nativeaot-all-assemblies-trimmed-by-default

[ajcvickers]

Transferring this back to ASP.NET since this is not an issue in EF. It's possible that DataProtectionKey.Id should be marked so as not to be trimmed.

[Webreaper]

@ajcvickers is there an issue to track the trimming support improvements that @roji mentions above? Would be good to link them here - and I'd like to track so that I can remove the HasKey call from my code when they're available.

[ajcvickers]

@Webreaper Trimming improvements for EF Core are unlikely to have an impact on this, at least in short to medium term. EF Core knows nothing of DataProtectionKey.

[Webreaper]

Oh, yes - good point. Presumably it's up to the identity/aspnetcore folks to decorate it appropriately. Or something.

Although since it's clearly declared as a key where it's defined, I was assuming that @roji was talking about doing some magic to automatically do that for any entity... :)

[ajcvickers]

Although since it's clearly declared as a key where it's defined

Where is is clearly declared as a key?

[Webreaper]

Where is is clearly declared as a key?

Dammit, you got me. I'd assumed it was..... But apparently not: https://github.com/dotnet/aspnetcore/blob/main/src/DataProtection/EntityFrameworkCore/src/DataProtectionKey.cs#L14

Presumably, if they explicitly decorated that with [Key] the problem wouldn't happen?

[ajcvickers]

@Webreaper That would not help unless anything attributed with [Key] in any library is not trimmed, which I doubt is the case.

[Webreaper]

Ah, okay. Is there a way to make it so that something decorated with [Key] automatically gets DynamicDependency applied? (I'm hand-waving again here, as I haven't delved into the trimming hints decorators). But you get my drift?

[ajcvickers]

That's not something I can answer.

[roji]

This is all something I'm planning to explore, though I don't think keys should be handled in a special way here: basically public properties on entity types shouldn't get trimmed, we should be able to do that by annotating modelBuilder.Entity, DbSet<>, etc.

[JamesNK]

The problem here is nothing in the DataProtection EntityFramework library references DataProtectionKey.Id.

There are two three solutions:

1. DataProtection EntityFramework dynamically includes DataProtectionKey.Id to stop it from being trimmed.
2. EntityFramework adds annotations to prevent properties from being trimmed from entities. e.g. DbSet.Add (and other EF methods that interact with entities) adds [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicProperties)].
3. DataProtection EntityFramework explicitly specifies the key using HasKey(u => u.Id) in OnModelCreating. I'm guessing EF has a convention that if no key is specified, then an appropriate Id property on the entity is used.

Number 2 is more complicated, but it's the better fix. Because EntityFramework isn't annotated, all trimmed libraries that use EF are going to have to ensure entities aren't trimmed manually (i.e. fix number 1). Ideally, EF should be annotated to do it for you.

Generally, EF entities are defined in user apps and aren't trimmed. But people who enable full trimming of all assemblies will also see problems like this.

Correctly annotating EF will take a lot of time. Here is a short-term fix (number 1) for DataProtection EntityFramework - #43204. There is only one entity, so it's simple to fix.

@Webreaper You should be able to remove your workaround after upgrading to .NET 7 RC1.

[Webreaper]

Awesome. Thanks James!