LibraryImportAttribute for p/invoke source generation

[AaronRobinsonMSFT]

Background and Motivation

The DllImportAttribute used in P/Invoke scenarios relies on a built-in system of dynamic IL generation to support marshalling of arguments to to a native environment. A new attribute that would permit the static IL generation
is proposed. Scenario details and overall design can be found here.

Proposed API

The proposed shape for LibraryImportAttribute is based on the existing DllImportAttribute, but looks to remove the inconsistent or platform-specific aspects.

This attribute is intended to be emitted by the p/invoke source generator rather than added to the runtime libraries.

Related proposal: #46838

namespace System.Runtime.InteropServices
{
    /// <summary>
    /// Specifies how strings should be marshalled
    /// </summary>
    public enum StringMarshalling
    {
        None = 0,
        Utf8,   // UTF-8
        Utf16,  // UTF-16, machine-endian
    }

    /// <summary>
    /// Attribute used to indicate a Source Generator should create a function for marshaling
    /// arguments instead of relying on the CLR to generate an IL Stub at runtime.
    /// </summary>
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
    public sealed class LibraryImportAttribute : Attribute
    {
        /// <summary>
        /// Constructor.
        /// </summary>
        /// <param name="libraryName">Name of the library containing the import</param>
        public LibraryImportAttribute(string libraryName);

        /// <summary>
        /// Library to load.
        /// </summary>
        public string LibraryName { get; }

        /// <summary>
        /// Indicates the name or ordinal of the entry point to be called.
        /// </summary>
        /// <see cref="System.Runtime.InteropServices.DllImportAttribute.EntryPoint"/>
        public string? EntryPoint { get; set; }

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="MarshalStringsUsing" /> must not be specified.
        /// </remarks>
        public StringMarshalling StringMarshalling { get; set; }

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="StringMarshalling" /> must not be specified.
        /// The type should be one that conforms to usage with the attributes:
        /// <see cref="System.Runtime.InteropServices.MarshalUsingAttribute"/>
        /// <see cref="System.Runtime.InteropServices.NativeMarshallingAttribute"/>
        /// </remarks>
        public Type? MarshalStringsUsing { get; set; }

        /// <summary>
        /// Indicates whether the callee sents an error (SetLastError on Windows or errorno
        /// on other platforms) before returning from the attributed method.
        /// </summary>
        /// <see cref="System.Runtime.InteropServices.DllImportAttribute.SetLastError"/>
        public bool SetLastError { get; set; }
    }
}

Alternate names for the attribute:

• ForeignFunctionAttribute
• GeneratedDllImportAttribute
• InteropImportAttribute
• NativeLibraryImportAttribute
• PInvokeImportAttribute
  ...

Usage Examples

Usage of LibraryImportAttribute would be similar to that of DllImportAttribute, but would also require a source generator be referenced during build. The attributed method and its containing types must also be partial. The source generator can be found in DllImportGenerator.

[LibraryImport("NativeBinary")]
internal static partial int Sum(int a, int b);

[LibraryImport("NativeBinary", StringMarshalling = StringMarshalling.Utf16)]
internal static partial string GetValue(string name);

[LibraryImport("NativeBinary", MarshalStringsUsing = typeof(MyStringMarshal.Wtf8String))]
internal static partial string GetValue(string name);

namespace MyStringMarshal
{
    struct Wtf8String { ... }
}

Analyzer/Generator diagnostics

• Fixer up for converting supported signatures from DllImportAttribute to LibraryImportAttribute.
• Generator will emit errors for invalid usage or missing support - prototyped diagnostics can be found here.
• Guidance for the Type to use with MarshalStringsUsing is covered by the diagnostics in Attributes and Analyzer Diagnostics to support custom struct marshalling in the DllImport Source Generator #46838

Risks

Attributes are typically static metadata so this will increase the amount of metadata for a compiled assembly. The DllImportAttribute is actually a pseudo-attribute and does not have the same metadata cost as a typical .NET Attribute like the one proposed - note that the resulting generated code will vastly out-weigh the cost of this new metadata.

In terms of UX, a user would hit a build failure if one uses the attribute but doesn't reference a source code generator. There are also differences in compatibility/support between the built-in system and the proposed source generator - documented here - which users may hit. Users will get a generator diagnostic if trying to use the generator in these cases.

[AaronRobinsonMSFT]

/cc @stephentoub @jkoritzinsky @elinor-fung @jkotas @tannergooding

[jkotas]

We will need number of other APIs to support source generated interop. Have you filled this one because you need it to get in sooner than the rest, or can it wait for the rest of the APIs to support source generated interop?

I think it is ok to add this attribute before API review as internal to CoreLib if you would like to get the source generated interop (partially) enabled for shipping bits to prove that it works well and to make it easier to fine tune the shape of the APIs required to support it.

Does it also need the constructor (and maybe the Value property) so that GeneratedDllImportAttribute("kernel32.dll") works?

These compatibility differences would make up the bulk of analyzer rules.

I am not following how the compatibility differences relate to analyzer rules. I would expect that we are going to have:

• Auto-fixer suggests to use GeneratedDllImportAttribute
• Source generator generates errors for unsupported constructs
• (Optional) Analyzer generates for supported constructs that are likely wrong

[stephentoub]

The proposed shape for GeneratedDllImportAttribute is identical to the existing DllImportAttribute.

That certainly helps with migration, which is great. On the flip side, DllImport has been around since the beginning of .NET, and for such things, it's rare we don't find something about it we wish we could have done differently. Are there any things here it would be valuable to redesign, as we now have that chance?

[stephentoub]

Attributes are typically static metadata so this will increase the amount of metadata for a compiled assembly. The DllImportAttribute is actually a pseudo-attribute and does not have the same metadata cost as a typical .NET Attribute like the one proposed.

I'd have thought a much bigger cost would be all of the code actually spit into the compiled assembly by the generator...?

[rseanhall]

Was #33742 (comment) addressed somewhere else? Why does this attribute need to be part of .NET instead of the source generator?

[AaronRobinsonMSFT]

We will need number of other APIs to support source generated interop. Have you filled this one because you need it to get in sooner than the rest, or can it wait for the rest of the APIs to support source generated interop?

@jkotas This is the first of several API issues - see "New .NET APIs" in #43060.

Does it also need the constructor (and maybe the Value property) so that GeneratedDllImportAttribute("kernel32.dll") works?

Oops. Copied the API from the wrong document - updated above.

• Auto-fixer suggests to use GeneratedDllImportAttribute

That is my mistake, added.

• Source generator generates errors for unsupported constructs
• (Optional) Analyzer generates for supported constructs that are likely wrong

I was including both of the above as compatibility between source gen and the built-in. I agree both should do that. I will update the proposal to clarify.

Are there any things here it would be valuable to redesign, as we now have that chance?

@stephentoub This v1 proposal is for specifically for providing what is needed in .NETCoreApp. We have some ideas on additional support but those could be added in an update. I don't know if we should add them here at the moment. I believe the design is flexible enough to expand.

I'd have thought a much bigger cost would be all of the code actually spit into the compiled assembly by the generator...?

Absolutely, but I was interpreting this as the specific API itself and not how other tools respond to it. The result of the source generator will very much impact the size and is planned to be investigated see "Prototype exit criteria" in #43060.

Why does this attribute need to be part of .NET instead of the source generator?

@rseanhall Adding this to .NETCoreApp permits its use in .NETCoreApp. Providing this as a NuGet would make a cumbersome library dependency to remove during compilation - which isn't possible in source generators and would therefore need to be a post-process step.

[jkotas]

This v1 proposal is for specifically for providing what is needed in .NETCoreApp. We have some ideas on additional support but those could be added in an update. I don't know if we should add them here at the moment. I believe the design is flexible enough to expand.

I think question is about what to drop from the initial proposal, not about what to add to it. For example, do we like PreserveSig? It is very COM specific and it is not a very good name either. Would it make sense to drop it for now and replace it with something more appropriate in future iterations? Run a similar thought process for each of the bools.

[AaronRobinsonMSFT]

I think question is about what to drop from the initial proposal, not about what to add to it.

Ah. If I recall the desire was to have the same API surface area. I could have missed the rational for that suggestion, but when it was originally proposed most of the above properties were removed since they didn't have a lot of value. We can definitely remove some of them if keeping the surface area the same isn't a goal.

/cc @jkoritzinsky @elinor-fung

[AaronRobinsonMSFT]

@stephentoub @jkotas At present the following two fields are explicit errors in the prototype and have been removed from above. The PreserveSig is still being used, but I could see an argument for changing the name. The others all seemed reasonable to us and were implemented. All semantic differences and limitations on the other fields are fully documented here.

  /// <summary>
  /// Enables or disables best-fit mapping behavior when converting Unicode characters
  /// to ANSI characters.
  /// </summary>
  /// <see cref="System.Runtime.InteropServices.DllImportAttribute.BestFitMapping"/>
  public bool BestFitMapping;

  /// <summary>
  /// Enables or disables the throwing of an exception on an unmappable Unicode character
  /// that is converted to an ANSI "?" character.
  /// </summary>
  /// <see cref="System.Runtime.InteropServices.DllImportAttribute.ThrowOnUnmappableChar"/>
  public bool ThrowOnUnmappableChar;

[jkotas]

PreserveSig is still being used

I see a very few uses of it in our own libraries.

HRESULT marshaling always had a questionable behavior around IErrorInfo. It assumes that whatever thread-local IErrorInfo* happens to be there is valid for failing HRESULT that is not always true, and tends to lead to bogus exception to be thrown intermittently on error. At least some (if not all) uses of it in our own libraries seem to suffer from this problem. If we keep it as is, we will be rolling this problem into future.

ExactSpelling

We are introducing analyzers to prefer ExactSpelling=true (#35695). Should we make ExactSpelling=true to be the default? Or maybe even drop it completely from GeneratedDllImportAttribute?

[rseanhall]

Providing this as a NuGet would make a cumbersome library dependency to remove during compilation - which isn't possible in source generators and would therefore need to be a post-process step.

I don't understand. Isn't the source generator going to be consumed though a NuGet package by .NETCoreApp? Wouldn't that use PrivateAssets="all", which removes any problematic consequences from that reference?

The attributes are the UI for the source generator. Its ability to evolve will be significantly limited without corresponding changes in the runtime if the attributes live there.

[AaronRobinsonMSFT]

Isn't the source generator going to be consumed though a NuGet package by .NETCoreApp? Wouldn't that use PrivateAssets="all", which removes any problematic consequences from that reference?

@rseanhall I don't know how it would be able to compile then. For example if project (A) applies this new attribute to a function declaration then it will by necessity need to reference an assembly that contains the attribute. That will be the C# that the developer writes and we can't change that attribute usage. The source generator will read that attribute during source generation and emit additional source code, but the originally written source will still exist. This means that even after source generation that attribute will exist in the final assembly and require a reference to the assembly it contains. The source generator doesn't need to be referenced but the containing assembly will need to be.

We could apply a ConditionalAttribute to the type which could then be removed in some circumstances but that is undesirable in my opinion. Understanding how these functions were defined is helpful for various tools. It also isn't clear the benefit of removing them is other than less metadata.

Its ability to evolve will be significantly limited without corresponding changes in the runtime if the attributes live there.

I don't think this is true. The current design is around indicating/propagating the current DllImport mechanism to any source generator. This can be observed in the minimal scope for the proposed API. This means that the information convey is enough to enable any source generator to consume the same attribute. It isn't forcing any policy on those source generators it is as intended to be the minimal metadata. Third party and even the eventual default source generator will be able to provide additional attributes if desired.

[jkotas]

I don't know how it would be able to compile then.

I believe that @rseanhall meant that a private local copy of the attribute can be emitted by the source generator, similar to how C# compiler produces a private local copies of attributes today. For example, if you compile the fragment below against .NET Framework, you will see a private local copy of Nullable attributes in the binary since they do not exist in .NET Framework:

using System;
#nullable enable
public class Test { public object? foo() => null; }

We should clarify the versioning story for the interop source generator - whether the interop source generator is going to be coupled with the .NET runtime version or whether it is going to be independent. The question to answer is: Once a new version of the interop source generator ships in .NET 7, is it going to be possible to use .NET 7 version of the interop source generator to target .NET 6?

[AaronRobinsonMSFT]

@jkotas Ah. That doesn't/won't work with source generators and types that have fields or state. The issue is that during the parsing pass the attribute is marked as an "error" because during compilation the attribute type isn't known. This means that the fields passed into that type are in an error state and making inspection difficult. If the attribute was simple a marker then this would be possible. In fact this is how I originally tried to make this work and then discovered the issues with the compiler not knowing the type prior to passing the parsed code into the source generator.

[ericstj]

Does dotnet/roslyn#49753 address that?

[AaronRobinsonMSFT]

Does dotnet/roslyn#49753 address that?

Yes. That is exactly the issue we faced.

I don't think this changes this particular API though. I say that because it is a primitive for all source generators to hook into since the goal is to advance the existing P/Invoke system. The current proposal is a minimum enough for all basic scenarios, but source generators should be able to extend the limited functionality with a tailored experience if desired.

[jkotas]

it is a primitive for all source generators to hook into since the goal is to advance the existing P/Invoke system

I see this attribute as a trigger for the source generator that comes with the platform. I expect that we are going to only have one of those. I think the eventual other source generators should invent their own triggering attribute. If we had multiple source generators attached to the same attribute, the system does not compose well, e.g. you cannot use both of them in the same project.

[AaronRobinsonMSFT]

I think the eventual other source generators should invent their own triggering attribute.

They can extend the functionality for sure, but it can also be a base for all source generators.

If we had multiple source generators attached to the same attribute, the system does not compose well, e.g. you cannot use both of them in the same project.

Agree - multiple source generators are likely not a good idea. However there is no reason to think we would require our source generator to be always on. The plan was the .NET one would be an option and users could pull down the official .NET one or consume another if desired. I don't think making ours special or inbox is a good idea as it bloats the environment and to me indicates this one is best. It works for our platform needs and is grows to accommodate the biggest user base but there could be users that want a more tailored experience and triggering off the same code generator permits the code to remain as is and try different generators from the broader community. It seems similar to any of the Diagnostic attributes that are not just used by VS but any potential managed debugger.

[jkotas]

I doubt that community would be interested in building source generators that achieve nearly identical outcomes as the default PInvoke source generator. I agree with you that there may be interop source generators that deliver significantly distinct features, but they will likely want to have their custom own attributes and flags to control the generation and so having this attribute will not help much.

It would be useful to have a paragraph about versioning and composability in the design doc.

[RussKie]

We are introducing analyzers to prefer ExactSpelling=true (#35695). Should we make ExactSpelling=true to be the default? Or maybe even drop it completely from GeneratedDllImportAttribute?

In Windows Forms we generally prefer to use ExactSpelling however there are few cases where we prefer code readability and set ExactSpelling=false.

[AaronRobinsonMSFT]

If the proposal at #51156 is accepted then removing the CallingConvention field would be possible and preferable.

[elinor-fung]

The description has been updated to use LibraryImport (with a list of some other names that were proposed).
We also opted to remove Windows-centric properties like PreserveSig as previously suggested.

[AaronRobinsonMSFT]

This looks good @elinor-fung. I think the SetLastError field's doc may need to be updated. I could also see an argument o rename it to something less Windows-centric.

[RussKie]

I'm finding MarshalStringsUsing too verbose. Could we consider something shorter, e.g. StringsMarshaller?

[jkoritzinsky]

The idea for the name is for parity with the MarshalUsing attribute's naming and to keep our naming scheme consistent.

[bartonjs]

Video

The main thing that came up in discussion is that the string encoding being an open Type parameter is not very good ease-of-use/user-experience. We probably want something with the convenience of the CharSet enum, but perhaps a new one specific to this use case. (So this would be a sibling property to the MarshalStringsUsing property, just accelerating the common cases)

[teo-tsirpanis]

@bartonjs in case you don't know, the livestream is down.

[bartonjs]

@teo-tsirpanis Yeah. @terrajobst's connection dropped and so I just stepped in and wrote the notes to wrap up the meeting.

[elinor-fung]

Current thoughts on the string marshalling:

namespace System.Runtime.InteropServices
{
    public enum StringMarshalling
    {
        Utf8,   // UTF-8
        Utf16,  // UTF-16, machine-endian
    }

    public sealed class LibraryImportAttribute : Attribute
    {
        ...

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="MarshalStringsUsing" /> must not be specified.
        /// </remarks>
        public StringMarshalling StringMarshalling { get; set; }

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="StringMarshalling" /> must not be specified.
        /// The type should be one that conforms to usage with the attributes:
        /// <see cref="System.Runtime.InteropServices.MarshalUsingAttribute"/>
        /// <see cref="System.Runtime.InteropServices.NativeMarshallingAttribute"/>
        /// </remarks>
        public Type? MarshalStringsUsing { get; set; }
    }
}

Not promoting ANSI as first-class via the enum - users can go through MarshalStringsUsing.

[bartonjs]

We renamed the MarshalStringsUsing property to StringMarshallingCustomType, and renamed the 0 value of StringMarshalling from None to Custom to better describe the relationship of the two properties. Otherwise, looks good as proposed.

namespace System.Runtime.InteropServices
{
    /// <summary>
    /// Specifies how strings should be marshalled
    /// </summary>
    public enum StringMarshalling
    {
        Custom = 0,
        Utf8,   // UTF-8
        Utf16,  // UTF-16, machine-endian
    }

    /// <summary>
    /// Attribute used to indicate a Source Generator should create a function for marshaling
    /// arguments instead of relying on the CLR to generate an IL Stub at runtime.
    /// </summary>
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
    public sealed class LibraryImportAttribute : Attribute
    {
        /// <summary>
        /// Constructor.
        /// </summary>
        /// <param name="libraryName">Name of the library containing the import</param>
        public LibraryImportAttribute(string libraryName);

        /// <summary>
        /// Library to load.
        /// </summary>
        public string LibraryName { get; }

        /// <summary>
        /// Indicates the name or ordinal of the entry point to be called.
        /// </summary>
        /// <see cref="System.Runtime.InteropServices.DllImportAttribute.EntryPoint"/>
        public string? EntryPoint { get; set; }

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="StringMarshallingCustomType" /> must not be specified.
        /// </remarks>
        public StringMarshalling StringMarshalling { get; set; }

        /// <summary>
        /// Indicates how to marshal string parameters to the method.
        /// </summary>
        /// <remarks>
        /// If this field is specified, <see cref="StringMarshalling" /> must not be specified.
        /// The type should be one that conforms to usage with the attributes:
        /// <see cref="System.Runtime.InteropServices.MarshalUsingAttribute"/>
        /// <see cref="System.Runtime.InteropServices.NativeMarshallingAttribute"/>
        /// </remarks>
        public Type? StringMarshallingCustomType { get; set; }

        /// <summary>
        /// Indicates whether the callee sents an error (SetLastError on Windows or errorno
        /// on other platforms) before returning from the attributed method.
        /// </summary>
        /// <see cref="System.Runtime.InteropServices.DllImportAttribute.SetLastError"/>
        public bool SetLastError { get; set; }
    }
}

[naine]

Custom seems a bit inconsistent with the mutually exclusive relationship of the two properties, since it implies you should do StringMarshalling = Custom, StringMarshallingCustomType = typeof(...).

IMO, the attribute descriptions should be updated as follows if that name will be used:

        /// <remarks>
        /// If this field is set to a value other than <see cref="Custom" />, <see cref="StringMarshallingCustomType" /> must not be specified.
        /// </remarks>
        public StringMarshalling StringMarshalling { get; set; }

        /// <remarks>
        /// If this field is specified, <see cref="StringMarshalling" /> must not be specified, or must be set to <see cref="Custom" />.
        /// </remarks>
        public Type? StringMarshallingCustomType { get; set; }