Intermittent test_evaluate_javascript failure in Cocoa CI

[mhsmith]

This has happened twice in the last two days:

• https://github.com/beeware/toga/actions/runs/8738532889
• https://github.com/beeware/toga/actions/runs/8713338180

tests/widgets/test_webview.py::test_load_empty_url PASSED                [ 98%]
=== Messages dropped during live streaming (use `log show` to see what they were)
=== Messages dropped during live streaming (use `log show` to see what they were)
=== Messages dropped during live streaming (use `log show` to see what they were)
tests/widgets/test_webview.py::test_static_content PASSED                [ 98%]
=== Messages dropped during live streaming (use `log show` to see what they were)
tests/widgets/test_webview.py::test_evaluate_javascript FAILED           [ 99%]
=== Messages dropped during live streaming (use `log show` to see what they were)
tests/widgets/test_webview.py::test_evaluate_javascript_error_without_handler PASSED [100%]

=================================== FAILURES ===================================
___________________________ test_evaluate_javascript ___________________________
Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 155, in _multicall
    teardown[0].send(outcome)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/unraisableexception.py", line 88, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
pytest.PytestUnraisableExceptionWarning: Exception ignored on calling ctypes callback function: <rubicon.objc.api.objc_method object at 0x10f0978b0>

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/rubicon/objc/api.py", line 335, in __call__
    result = self.py_method(py_self, *args)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/toga_cocoa/widgets/webview.py", line 28, in webView_didFinishNavigation_
    self.interface.on_webview_load()
AttributeError: 'NoneType' object has no attribute 'on_webview_load'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/runner.py", line 339, in from_call
    result: Optional[TResult] = func()
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/runner.py", line 260, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 159, in _multicall
    _warn_teardown_exception(hook_name, teardown[1], e)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 49, in _warn_teardown_exception
    warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: unraisableexception, Hook: pytest_runtest_call
PytestUnraisableExceptionWarning: Exception ignored on calling ctypes callback function: <rubicon.objc.api.objc_method object at 0x10f0978b0>

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/rubicon/objc/api.py", line 335, in __call__
    result = self.py_method(py_self, *args)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/toga_cocoa/widgets/webview.py", line 28, in webView_didFinishNavigation_
    self.interface.on_webview_load()
AttributeError: 'NoneType' object has no attribute 'on_webview_load'

For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning

[freakboy3742]

This is a weird error - it shouldn't be possible for self.interface to be None. I can only assume that it happens as some sort of garbage collection edge case - the test case is erroring out, causing the cleanup of the interface widget, but the native delegate is persisting just long enough to handle one last signal.

That suggests the actual problem is a failure of the javascript execution, but that an intermittent failure that shouldn't be occurring at all, so it's not clear how to reproduce the problem (or even instrument the problem for reporting purposes).

[mhsmith]

https://github.com/beeware/toga/actions/runs/8861280348 may be related, but in this case it involves the MapView rather than the WebView:

=================================== FAILURES ===================================
________________________________ test_add_pins _________________________________
Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 156, in _multicall
    teardown[0].send(outcome)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/unraisableexception.py", line 88, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
pytest.PytestUnraisableExceptionWarning: Exception ignored on calling ctypes callback function: <rubicon.objc.api.objc_method object at 0x10682b580>

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/rubicon/objc/api.py", line 335, in __call__
    result = self.py_method(py_self, *args)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/toga_cocoa/widgets/mapview.py", line 35, in mapView_regionDidChangeAnimated_
    region = self.impl.native.region
AttributeError: 'NoneType' object has no attribute 'native'

[freakboy3742]

Another variation on theme from this run

_______________________ ERROR at setup of test_load_url ________________________
Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 156, in _multicall
    teardown[0].send(outcome)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/unraisableexception.py", line 83, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
pytest.PytestUnraisableExceptionWarning: Exception ignored on calling ctypes callback function: <rubicon.objc.api.objc_method object at 0x107cf11b0>

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/rubicon/objc/api.py", line 335, in __call__
    result = self.py_method(py_self, *args)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/toga_cocoa/widgets/webview.py", line 30, in webView_didFinishNavigation_
    if self.impl.loaded_future:
AttributeError: 'NoneType' object has no attribute 'loaded_future'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/runner.py", line 339, in from_call
    result: Optional[TResult] = func()
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/_pytest/runner.py", line 260, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 160, in _multicall
    _warn_teardown_exception(hook_name, teardown[1], e)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/pluggy/_callers.py", line 50, in _warn_teardown_exception
    warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: unraisableexception, Hook: pytest_runtest_setup
PytestUnraisableExceptionWarning: Exception ignored on calling ctypes callback function: <rubicon.objc.api.objc_method object at 0x107cf11b0>

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/rubicon/objc/api.py", line 335, in __call__
    result = self.py_method(py_self, *args)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/toga_cocoa/widgets/webview.py", line 30, in webView_didFinishNavigation_
    if self.impl.loaded_future:
AttributeError: 'NoneType' object has no attribute 'loaded_future'

For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
=========================== short test summary info ============================
ERROR tests/widgets/test_webview.py::test_load_url - pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: unraisableexception, Hook: pytest_runtest_setup
PytestUnraisableExceptionWarning: Exception ignored on calling ctypes callback function: <rubicon.objc.api.objc_method object at 0x107cf11b0>

Traceback (most recent call last):
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/rubicon/objc/api.py", line 335, in __call__
    result = self.py_method(py_self, *args)
  File "/Users/runner/work/toga/toga/testbed/build/testbed/macos/app/Toga Testbed.app/Contents/Resources/app_packages/toga_cocoa/widgets/webview.py", line 30, in webView_didFinishNavigation_
    if self.impl.loaded_future:
AttributeError: 'NoneType' object has no attribute 'loaded_future'

The common theme appears to be that methods on the delegate are being invoked at a point where the widget shouldn't exist.

A workaround for these cases would be to add an if self.impl check anywhere that self.impl is being referenced... but that will result in branches that can't be reliably reached. That's not a huge issue as we can work around it with no-covers, but it's not a very satisfying response.

In the Webview case it's interesting to note that the error is manifesting when the impl is referenced - even though 2 lines earlier, there is a reference to the interface.

I have a vague hunch that the manifestation of this bug might be a side effect of beeware/rubicon-objc#256 - a weakly held object property (impl) is being disposed of, but the delegate is still firing.