Internal error while xml serializing a byte[] property set to null

[lscorcia]

Hi, I am evaluating the porting of some .net 4.7.2 code to .net core 3 and found this problem that I think is a compatibility issue in XML Serialization.

You can find the repro project at https://github.com/lscorcia/net-core-serialization-issue.

Basically we have a WCF Connected Service which takes a couple of objects as parameters. One of them contains a byte[] property, declared as nullable in the service definition. This is the relevant generated proxy snippet:

/// <remarks/>
[System.Xml.Serialization.SoapElementAttribute(DataType="base64Binary", IsNullable=true)]
public byte[] contenuto
{
  get
  {
    return this.contenutoField;
  }
  set
  {
    this.contenutoField = value;
  }
}

When invoking the remote service, the request serialization is ok when the property is set to some value, but fails when the value is null: it throws a System.ServiceModel.CommunicationException in System.Private.ServiceModel.dll. Here is the full stack trace:

System.ServiceModel.CommunicationException
  HResult=0x80131500
  Message=There was an error in serializing body of message : 'There was an error generating the XML document.'.  Please see InnerException for more details.
  Source=System.Private.ServiceModel
  StackTrace:
   at System.ServiceModel.Dispatcher.XmlSerializerOperationFormatter.SerializeBody(XmlDictionaryWriter writer, MessageVersion version, String action, MessageDescription messageDescription, Object returnValue, Object[] parameters, Boolean isRequest)

Inner Exception 1:
InvalidOperationException: There was an error generating the XML document.
   at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id) in /_/src/System.Private.Xml/src/System/Xml/Serialization/XmlSerializer.cs:line 373
   at System.ServiceModel.Dispatcher.XmlSerializerOperationFormatter.SerializeBody(XmlDictionaryWriter writer, MessageVersion version, String action, MessageDescription messageDescription, Object returnValue, Object[] parameters, Boolean isRequest)

Inner Exception 2:
InvalidOperationException: Internal error.
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.WritePrimitiveValue(TypeDesc typeDesc, Object o, Boolean isElement, String& stringValue) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 1165
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.WritePrimitive(WritePrimitiveMethodRequirement method, String name, String ns, Object defaultValue, Object o, TypeMapping mapping, Boolean writeXsiType, Boolean isElement, Boolean isNullable) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 991
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.WriteElement(Object o, ElementAccessor element, Boolean writeAccessor) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 405
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.WriteElements(Object o, Object enumSource, ElementAccessor[] elements, TextAccessor text, ChoiceIdentifierAccessor choice, Boolean writeAccessors, Boolean isNullable) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 194
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.WriteStructMethod(StructMapping mapping, String n, String ns, Object o, Boolean isNullable, Boolean needType) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 598
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.<>c__DisplayClass20_0.<CreateXmlSerializationWriteCallback>b__0(Object o) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 456
   at System.Xml.Serialization.XmlSerializationWriter.WriteReferencedElement(String name, String ns, Object o, Type ambientType) in /_/src/System.Private.Xml/src/System/Xml/Serialization/XmlSerializationWriter.cs:line 1350
   at System.Xml.Serialization.XmlSerializationWriter.WriteReferencedElements() in /_/src/System.Private.Xml/src/System/Xml/Serialization/XmlSerializationWriter.cs:line 1370
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.GenerateMembersElement(Object o, XmlMembersMapping xmlMembersMapping) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 1371
   at System.Xml.Serialization.ReflectionXmlSerializationWriter.WriteObject(Object o) in /_/src/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs:line 74
   at System.Xml.Serialization.XmlSerializer.SerializeUsingReflection(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id) in /_/src/System.Private.Xml/src/System/Xml/Serialization/XmlSerializer.cs:line 383
   at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id) in /_/src/System.Private.Xml/src/System/Xml/Serialization/XmlSerializer.cs:line 350

This type of call was working fine in the original .net 4.7.2 software.
I can supply the service WSDL if needed, but I don't think that's relevant - the WCF proxy class looks ok and the error is in the early request creation.

I tried source-stepping in the framework code, but it is heavily optimized and the debugger is not that helpful. It looks like the null value of the array prevents the recognition of the parameter as an array and that it ends up on the wrong code path, but I might be easily mistaken on this.

Please let me know if you need any additional information.

[lscorcia]

The issue is still present on the recently released .net core 3.1.0 . I have updated the example project in the repository.

[StephenBonikowsky]

@lscorcia Thanks for reporting this, we will be investigating and will follow-up when we have more info.

[HongGit]

Moving this to .NET 6.0 milestone, if needed, we will consider to backport to 3.1.

[GidanMX2]

I have the same issue on 2.2, I have to pass an empty byte[] instead of null to bypass the problem

[ankit57]

I am facing same issue in 3.1.0. Is there any update on the fix or workaround?

[lscorcia]

The issue is still present on .net core 5. I have updated the example project in the repository.
Maybe one day this will be fixed and I will no longer have a sad smile when the .net teams wonder why .net core adoption rates are lower than expectations (https://visualstudiomagazine.com/articles/2021/03/16/ef-core.aspx).

[danmoseley]

@lscorcia are you willing to debug/investigate a little further to see why .NET Core is behaving differently?

[lscorcia]

Sure, let me know what I can do!

[danmoseley]

This isn't actually my area (that's @HongGit team see area-owners.md) but I was assuming you could attach a debugger and see what's going on. Often when there's a regression from .NET Framework it's quite fixable, although again I'm not familiar with this library.

[danmoseley]

Instructions for building and debugging are linked from the README.

[lscorcia]

So, if I understand correctly, the process would be to create a debug build from the current repo. This would mean preparing the build requirements (vs2019 c++ workload, cmake, etc), cloning the repo at the .net 5 tag, building the full runtime (not just the libraries), then find out how to connect a vs2019 debugger to the built environment and then finally try to understand where the crash comes from. It's going to be a very long process as I will need to do that in a VM as I can't risk borking my work environment, and my machine is a crappy laptop. I will try, but I can't guarantee I will be successful. If you know of any prebuilt debug build somewhere it's probably going to be much easier.

[danmoseley]

@lscorcia I should not have said building. I was thinking you could simply attach Visual Studio (any recent version) choosing the ".NET 5" debugger engine, set a function breakpoint on System.Xml.Serialization.ReflectionXmlSerializationWriter.WritePrimitiveValue (or on InvalidOperationException and hit ignore if it hits any in another library). When the breakpoints hits, the debugger should be able to pull down sources and symbols automatically. Then you could maybe look at the local state and make some inferences.

@HongGit is there someone on your team who could give more specific pointers? It seems this is an upgrade blocker for @lscorcia .

[lscorcia]

Hi @danmoseley , I did what you suggested and I was able to source-step inside the framework code. The exception is triggered at

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs

Lines 1160 to 1163 in 2f740ad

	else
	{
	throw new InvalidOperationException(SR.XmlInternalError);
	}

when the method WritePrimitiveValue is invoked with the following parameters:

• typeDesc = System.Byte[] (FormatterName = "ByteArrayBase64")
• o = null

Now, at

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs

Line 1096 in 2f740ad

else if (o is byte[] && typeDesc.FormatterName == "ByteArrayHex")

we see an explicit check for a different Formatter, so the code does not enter that path, and also at

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs

Lines 1156 to 1159 in 2f740ad

	else if (o is byte[])
	{
	// we deal with byte[] specially in WritePrimitive()
	}

there is a check for byte[], but it only checks the value object type, not the typeDesc, therefore it does not enter that path either (o is an object with null value).

This latter if block has a comment that suggests that the byte[] should be dealt with in the caller method WritePrimitive, but then the exception is raised and it never reaches that point:

runtime/src/libraries/System.Private.Xml/src/System/Xml/Serialization/ReflectionXmlSerializationWriter.cs

Line 1039 in 2f740ad

else if (o is byte[] a)

I am not familiar with the design of this serialization logic, but maybe the issue is that those checks are being performed on the value instead of using the type descriptor typeDesc? The fix could be to simply check the typeDesc in case the value is null?

[danmoseley]

What would be interesting is to now figure out what .NET Framework is doing differently. Again using a debugger but you might have to step some more or get creative with how you break in, to find the equivalent point (since it's not throwing)

[danmoseley]

Never mind, it seems this codepath may be new in .NET Core: dotnet/corefx#11369

I think we need an area expert to advise now.