[iOS / Native AOT] Bad stack trace due to unsupported marshalling behavior

[tipa]

Package

Sentry

.NET Flavor

.NET

.NET Version

8.0.2

OS

iOS

SDK Version

4.2.1

Self-Hosted Sentry Version

No response

Steps to Reproduce

Throw an exception in the AppDelegate.FinishedLaunching method in an NativeAot-compiled app.

To run a Native AOT compiled app locally on a physical device use dotnet publish /t:Run /p:_DeviceName=DEVICEID where DEVICEID can be queried using xcrun xctrace list devices.

There might be more prerequisites, like the managed exception to be marshalled to native code and back to managed.

Expected Result

Crash with useful strack trace including managed C# function from where it originated

Actual Result

Thread 0 Crashed:
0   libsystem_kernel.dylib          0x39fe95fbc         __pthread_kill
1   libsystem_pthread.dylib         0x3e519567c         pthread_kill
2   libsystem_c.dylib               0x3210abb8c         abort
3   MyXYTestApp                     0x20499ce8c         xamarin_assertion_message (runtime.m:1461)
4   MyXYTestApp                     0x20499d5d4         xamarin_process_managed_exception (runtime.m:2441)
5   MyXYTestApp                     0x20499d458         xamarin_process_managed_exception_gchandle (runtime.m:1123)
6   MyXYTestApp                     0x2049c6508         -[App application:didFinishLaunchingWithOptions:] (registrar.mm:2760)
7   UIKitCore                       0x31562361c         -[UIApplication _handleDelegateCallbacksWithOptions:isSuspended:restoreState:]
8   UIKitCore                       0x315622784         -[UIApplication _callInitializationDelegatesWithActions:forCanvas:payload:fromOriginatingProcess:]
9   UIKitCore                       0x315621768         -[UIApplication _runWithMainScene:transitionContext:completion:]
10  UIKitCore                       0x3156213b4         -[_UISceneLifecycleMultiplexer completeApplicationLaunchWithFBSScene:transitionContext:]
11  UIKitCore                       0x31559dedc         _UIScenePerformActionsWithLifecycleActionMask
12  UIKitCore                       0x3156253b0         __101-[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]_block_invoke
13  UIKitCore                       0x31554ce44         -[_UISceneLifecycleMultiplexer _performBlock:withApplicationOfDeactivationReasons:fromReasons:]
14  UIKitCore                       0x31554b8bc         -[_UISceneLifecycleMultiplexer _evalTransitionToSettings:fromSettings:forceExit:withTransitionStore:]
15  UIKitCore                       0x31554b224         -[_UISceneLifecycleMultiplexer uiScene:transitionedFromState:withTransitionContext:]
16  UIKitCore                       0x31554b0f4         __186-[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]_block_invoke
17  UIKitCore                       0x31554affc         +[BSAnimationSettings(UIKit) tryAnimatingWithSettings:fromCurrentState:actions:completion:]
18  UIKitCore                       0x31554a884         _UISceneSettingsDiffActionPerformChangesWithTransitionContextAndCompletion
19  UIKitCore                       0x31554a534         -[_UIWindowSceneFBSSceneTransitionContextDrivenLifecycleSettingsDiffAction _performActionsForUIScene:withUpdatedFBSScene:settingsDiff:fromSettings:transitionContext:lifecycleActionType:]
20  UIKitCore                       0x3158ce26c         __64-[UIScene scene:didUpdateWithDiff:transitionContext:completion:]_block_invoke.225
21  UIKitCore                       0x3155496b8         -[UIScene _emitSceneSettingsUpdateResponseForCompletion:afterSceneUpdateWork:]
22  UIKitCore                       0x315549528         -[UIScene scene:didUpdateWithDiff:transitionContext:completion:]
23  UIKitCore                       0x315661c94         -[UIApplication workspace:didCreateScene:withTransitionContext:completion:]
24  UIKitCore                       0x315661a2c         -[UIApplicationSceneClientAgent scene:didInitializeWithEvent:completion:]
25  FrontBoardServices              0x3419176d0         -[FBSScene _callOutQueue_didCreateWithTransitionContext:completion:]
26  FrontBoardServices              0x34191756c         __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke.108
27  FrontBoardServices              0x341916198         -[FBSWorkspace _calloutQueue_executeCalloutFromSource:withBlock:]
28  FrontBoardServices              0x341921f88         __92-[FBSWorkspaceScenesClient createSceneWithIdentity:parameters:transitionContext:completion:]_block_invoke
29  libdispatch.dylib               0x320fac2fc         _dispatch_client_callout
30  libdispatch.dylib               0x320fafd44         _dispatch_block_invoke_direct
31  FrontBoardServices              0x34191251c         __FBSSERIALQUEUE_IS_CALLING_OUT_TO_A_BLOCK__
32  FrontBoardServices              0x34191249c         -[FBSMainRunLoopSerialQueue _targetQueue_performNextIfPossible]
33  FrontBoardServices              0x341912374         -[FBSMainRunLoopSerialQueue _performNextFromRunLoopSource]
34  CoreFoundation                  0x310fef0a8         __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
35  CoreFoundation                  0x310fee324         __CFRunLoopDoSource0
36  CoreFoundation                  0x310fecad8         __CFRunLoopDoSources0
37  CoreFoundation                  0x310feb814         __CFRunLoopRun
38  CoreFoundation                  0x310feb3f4         CFRunLoopRunSpecific
39  GraphicsServices                0x397b374f4         GSEventRunModal
40  UIKitCore                       0x31563e89c         -[UIApplication _run]
41  UIKitCore                       0x31563ded8         UIApplicationMain
42  MyXYTestApp                     0x20499034c         xamarin_UIApplicationMain (bindings.m:126)

I believe this is caused because of this event handler that changes the default marshalling behavior of exceptions:

sentry-dotnet/src/Sentry/Platforms/Cocoa/SentrySdk.cs

Lines 14 to 17 in bea3fcf

	ObjCRuntime.Runtime.MarshalManagedException += (_, args) =>
	{
	args.ExceptionMode = ObjCRuntime.MarshalManagedExceptionMode.UnwindNativeCode;
	};

The code was introduced after this discussion in the Xamarin repo: dotnet/macios#15252

According to @rolfbjarne from the .NET iOS/macOS team, CoreCLR (which is what is used on iOS when using Native AOT as well as on macOS) doesn't support MarshalManagedExceptionMode.UnwindNativeCode and therefore, this assertion is crashing the process:
https://github.com/xamarin/xamarin-macios/blob/907081f787315704a01c940cf28b46b47db23df0/runtime/runtime.m#L2362-L2364
https://github.com/xamarin/xamarin-macios/blob/907081f787315704a01c940cf28b46b47db23df0/runtime/runtime.m#L2452-L2455

References

• https://github.com/xamarin/xamarin-macios/wiki/.NET-release-notes-Xcode-13.3#exception-marshalling-is-enabled-by-default

[jamescrosswell]

According to @rolfbjarne from the .NET iOS/macOS team, CoreCLR (which is what is used on iOS when using Native AOT as well as on macOS) doesn't support MarshalManagedExceptionMode.UnwindNativeCode and therefore, this assertion is crashing the process:

Hm, I'm not sure I follow everything that's going on here but it seems like that assertion is also dependent on xamarin_is_gc_coop, which is only true when TARGET_OS_WATCH:

#if TARGET_OS_WATCH
bool xamarin_is_gc_coop = true;
#else
bool xamarin_is_gc_coop = false;
#endif

I can't see anything from Sentry in that stack trace though. Do you know what kind of exception was originally thrown (that resulted in this stack trace)?

[tipa]

I don' think it is dependent on xamarin_is_gc_coop. I might have posted links from the wrong commit, this one shows how the xamarin_assertion_message method is called in line 2441, like in the stack trace:
https://github.com/xamarin/xamarin-macios/blob/ed26faa94fe2734d9c1014d2e6ef7173d4d77690/runtime/runtime.m#L2441
and then abort() in line 1461:
https://github.com/xamarin/xamarin-macios/blob/ed26faa94fe2734d9c1014d2e6ef7173d4d77690/runtime/runtime.m#L1461

I don't know what exception was originally thrown (that's what I'm trying to find out), @rolfbjarne recommended that I should also subscribe to the ObjCRuntime.Runtime.MarshalManagedException event and then log the exception manually (more on that is documented here. UnwindNativeCode is documented as This isn't available when [...] when using CoreCLR).

[jamescrosswell]

OK thanks, that makes more sense. If we can find a way to throw an exception that reproduces this behaviour then we should be able to work out a solution 👍🏻

[tipa]

@jamescrosswell you can reproduce this behavior / stack trace by simply throwing an exception in the AppDelegate.FinishedLaunching method in an NativeAot-compiled app. You might have to upload a build to TestFlight as (to my knowledge) it is not possible to run NativeAOT compiled iOS apps locally.
Since there will be more adoption of NativeAOT in the future (e.g. with the launch of .NET 9), would it be possible to optionally disable this part of the Sentry initialization, so that the unsupported marshalling behavior isn't used?

sentry-dotnet/src/Sentry/Platforms/Cocoa/SentrySdk.cs

Lines 14 to 17 in bea3fcf

	ObjCRuntime.Runtime.MarshalManagedException += (_, args) =>
	{
	args.ExceptionMode = ObjCRuntime.MarshalManagedExceptionMode.UnwindNativeCode;
	};

[bitsandfoxes]

would it be possible to optionally disable this part of the Sentry initialization

We could definitely put these on the Cocoa specific native options - like we do with the LogCat options for Android.

[bitsandfoxes]

Upping the priority on this as it crashes the process.

[jamescrosswell]

Upping the priority on this as it crashes the process.

@bitsandfoxes what do we want to do about this?

you might have to upload a build to TestFlight as (to my knowledge) it is not possible to run NativeAOT compiled iOS apps locally.

Realistically, we can't test for things like this until/unless we create a developer profile and an app in the Apple store that we start down the QA route. Having done that once personally for a Flutter app, I suspect it's going to take about 3-5 days to get this all setup via CI so that we can easily deploy new versions of our app (e.g. Sentry.Samples.MAUI) to the iOS store... there are various encryption/deployment keys to manage that we'll need to sync between CI and our local machines etc.

I don't think we've got the bandwidth to do this right now. Perhaps once we've sorted out what we're going to do about net9.0 (and the device tests) we could circle back to this.

[tipa]

@bitsandfoxes what do we want to do about this?

Would it not be possible to allow an optional/temporary way to disable the changing of marshalling bahavior, as I suggested above?
I don't know if it would really work, but at the moment, the stack traces are just barely understandable...

[tipa]

I was able to run a Native AOT compiled app locally on a physical device using dotnet publish /t:Run /p:_DeviceName=DEVICEID where DEVICEID can be queried using xcrun xctrace list devices

[jamescrosswell]

I was able to run a Native AOT compiled app locally on a physical device using dotnet publish /t:Run /p:_DeviceName=DEVICEID where DEVICEID can be queried using xcrun xctrace list devices

That helps... it means we don't have to push it all the way through to test flight. Just a bit of fluffing around setting up encryption keys, developer profiles and configuring an app in the apple dev portal.

So it sounds a little less painful, when we do have the bandwidth to look into this.

[tipa]

When I create a blank iOS app (dotnet new ios) and throw an exception in FinishedLaunching while running with NativeAot, I do not get this part in the native stack trace (gathered by NSThread.NativeCallStack), which would indicate the process crashing here.

Thread 0 Crashed:
0   libsystem_kernel.dylib          0x39fe95fbc         __pthread_kill
1   libsystem_pthread.dylib         0x3e519567c         pthread_kill
2   libsystem_c.dylib               0x3210abb8c         abort
3   MyXYTestApp                     0x20499ce8c         xamarin_assertion_message (runtime.m:1461)

In your screenshot, these lines still seem to be present:

I also cannot reproduce the stack trace at the moment when using my own logging, the top of the stack trace always looks like the one below, even when setting the ExceptionMode to UnwindNativeCode - maybe it's because I have a different setup (e.g. no native library catching and sending out the crash report):

0   test                                0x00000001050f9ef0 xamarin_log + 6740
1   test                                0x00000001049ca660 test + 206432
2   test                                0x0000000104e28d80 test + 4787584
3   test                                0x0000000104e10b00 test + 4688640
4   test                                0x00000001050f0f94 xamarin_get_block_descriptor + 8052

Wish I could provide more help, but I'm also no expert in the matter, unfortunately.

[jamescrosswell]

When I create a blank iOS app (dotnet new ios) and throw an exception in FinishedLaunching while running with NativeAot, I do not get this part in the native stack trace (gathered by NSThread.NativeCallStack), which would indicate the process crashing here.

Thanks @tipa,

I'm running against 9.0.1 runtime so I think my stack trace corresponds to this version of the runtime.m (line 1478), which is ultimately triggered here (line 2484).

That makes sense since Bryce was setting the ExceptionMode to MarshalManagedExceptionModeDisable (identical to MarshalManagedExceptionModeUnwindNativeCode in AOT applications).

Using MarshalManagedExceptionModeDefault would yield different results - I think it ultimately sets the ExceptionMode to MarshalManagedExceptionModeThrowObjectiveCException... which presumably gives the stack traces that you're seeing.

After much fiddling around and digging into the internals of Xamarin, I think I might have a PR that fixes this:

• Only use MarshalManagedExceptionMode.UnwindNativeCode when building for Mono on net8.0 and earlier #3912

However, I haven't managed to test this on the CoreCLR. No matter what I try, on my physical iOS device, it detects that it's running on MonoVM 😕

I'm running dotnet publish ./samples/Sentry.Samples.Maui/Sentry.Samples.Maui.csproj -t:Run -f net9.0-ios18.0 -r ios-arm64 /p:_DeviceName=00008030-00120D190A38802E

That could indicate that the logic I'm using to detect whether CoreCLR is being used is incorrect... but I'm reasonably confident this should work:

sentry-dotnet/src/Sentry.Bindings.Cocoa/buildTransitive/Sentry.Bindings.Cocoa.targets

Lines 58 to 61 in 0087cf3

	The condition is a bit hacky - it relies on:
	https://github.com/xamarin/xamarin-macios/blob/0ab074390689880fe9269804a0156341d30251f6/dotnet/targets/Xamarin.Shared.Sdk.targets#L1004-L1006
	-->
	<PropertyGroup Condition="'$(_XamarinRuntime)' != 'CoreCLR'">

If I put together a pre-release including this fix, would you be able to test it? If this doesn't work, then I think we could just add an option to let you override the MarshalManagedExceptionMode manually. However I'd rather set the most appropriate option automatically if possible (the purpose of the PR above).

[tipa]

I'm running dotnet publish ./samples/Sentry.Samples.Maui/Sentry.Samples.Maui.csproj -t:Run -f net9.0-ios18.0 -r ios-arm64 /p:_DeviceName=00008030-00120D190A38802E

I used a similar command when testing, I think I didn't specify the -f and -r options, but that shouldn't make much of a difference.

However, I haven't managed to test this on the CoreCLR. No matter what I try, on my physical iOS device, it detects that it's running on MonoVM 😕

Based on the command above, I assume you were running this project? I don't know if you made any adjustments to the file prior to running it, but maybe the condition here isn't met? But that's also just a wild guess, because it looks right...

sentry-dotnet/samples/Sentry.Samples.Maui/Sentry.Samples.Maui.csproj

Lines 48 to 50 in 3e4af30

	<PropertyGroup Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios' or $([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'maccatalyst'">
	<PublishAot>true</PublishAot>
	</PropertyGroup>

If I put together a pre-release including this fix, would you be able to test it?

I can try as long as it doesn't take too long.

[jamescrosswell]

However, I haven't managed to test this on the CoreCLR. No matter what I try, on my physical iOS device, it detects that it's running on MonoVM 😕

OK, I think I've finally got it working. The MSBuild condition I had configured wasn't appropriate. I've tweaked this and now when I run on a physical device with this command, I get AOT Compilation and MarshalManagedExceptionMode.Default:

dotnet publish ./samples/Sentry.Samples.Maui/Sentry.Samples.Maui.csproj -f net9.0-ios18.0 -r ios-arm64 -t:Run /bl

And when running on a simulator with this command, I get MarshalManagedExceptionMode.UnwindManagedCode:

dotnet build ./samples/Sentry.Samples.Maui/Sentry.Samples.Maui.csproj -t:Run -c Debug -f net9.0-ios18.0 /p:_DeviceName=:v2:udid=129CDB6C-D5B7-40F6-BBE3-6DD3D397171D /bl

I believe this is finally what we wanted... @tipa I'll put together a pre-release and let you know when it's available so that you can double check my work. Thank you so much for your patience with this.

[jamescrosswell]

@tipa this should finally have been resolved in PR#3909 that was included in the 5.1.0 release.

That PR removes all/any code modifying the MarshalManagedExceptionMode and uses an alternate workaround to the problem that code was initially introduced to fix.

If you have a chance to test with 5.1.0 that'd be great (and I can leave this issue open for a few more days). If not, I understand and I'll close this issue immediately.

[tipa]

I did a quick test with both 5.0.1 and 5.1.0 and throwing an exception in FinishedLaunching

5.0.1

5.1.0

So with 5.1.0 I seem to get two crash reports per crash

• The correct one:

• But still also this one:

It looks like the second one is reported by the Cocoa SDK - is this expected behavior?

[jamescrosswell]

It looks like the second one is reported by the Cocoa SDK - is this expected behavior?

Hm, it is and it isn't. The fix in the 5.1.0 release suppresses EXC_BAD_ACCESS exceptions from the Cocoa SDK (those would typically be translated to and captured as managed NullReferenceExceptions).

Your code is creating EXC_CRASH errors rather than EXC_BAD_ACCESS errors, so the Sentry SDK doesn't suppress these.

It looks like you just did a throw new Exception(). I suspect the reason it's being caught by the Native SDK is because it's a terminal exception that's crashing the app, which results in the SIGABRT. I don't think we want to suppress all/any EXC_CRASH and/or SIGABRT errors in the NDK as that could prevent genuine native exceptions from being reported.

I think we just have to accept duplicate exceptions in this particular scenario then (until we can find a long term / root cause solution to the problem described in #3902.

Although you're getting two events for exceptions that crash the app, at least one of them should have a stack trace that helps find the code that it's coming from.

[tipa]

Yes, the original issue indeed appears to be fixed

[jamescrosswell]

OK thanks @tipa - I'll close this issue then. The remainer of the work will be tracked in #3902:

• Managed NullReferenceException causes EXC_BAD_ACCESS and SIGSEGV errors on iOS and Android  #3902