[3.12] Crash of repeated Programs/_testembed.c on Windows DEBUG build

[picnixz]

Bug report

Bug description:

This is an issue that only affects Windows DEBUG builds and Python 3.12. I discovered it while writing #127780 and @Yhg1s (#127780 (comment)) investigated it more in details.

The reproducer is as follows:

PS D:\Python\cpython> D:\\Python\\cpython\\PCbuild\\amd64\\_testembed_d.exe test_repeated_init_exec 'import dis
>> import importlib._bootstrap
>> import opcode
>> import test.test_dis
>>
>> def is_specialized(f):
>>     for instruction in dis.get_instructions(f, adaptive=True):
>>         opname = instruction.opname
>>         if (
>>             opname in opcode._specialized_instructions
>>             # Exclude superinstructions:
>>             and \"__\" not in opname
>>         ):
>>             return True
>>     return False
>>
>> func = importlib._bootstrap._handle_fromlist
>>
>> # "copy" the code to un-specialize it:
>> func.__code__ = func.__code__.replace()
>>
>> assert not is_specialized(func), \"specialized instructions found\"
>>
>> for i in range(test.test_dis.ADAPTIVE_WARMUP_DELAY):
>>     func(importlib._bootstrap, [\"x\"], lambda *args: None)
>>
>> assert is_specialized(func), \"no specialized instructions found\"
>>
>> print(\"Tests passed\")'
--- Loop #1 ---
Tests passed
--- Loop #2 ---
Assertion failed: PyUnicode_CheckExact(ep->me_key), file D:\Python\cpython\Objects\dictobject.c, line 939

Paraphrasing Thomas, the assertion failure happens only the second time, namely after finalizing and reinitializing, and the ep the assertion fails on is a deallocated object. We suspect that something isn't refcounted correctly, isn't marked as immortal when it should be, or isn't recreated on interpreter starutp when it should be. However, we don't why the issue does not appear on Linux.

According to Thomas, this is likely an issue with repeated calls to Py_Finalize and Py_Initialize. However, since 3.12 and 3.13 manage objects lifetime quite differently, it might be easier to just patch the test currently failing as follows: 68a33f3.

CPython versions tested on:

3.12

Operating systems tested on:

Windows, Linux

Linked PRs

• [3.12] gh-128833: Patch test_embed.test_specialized_static_code_gets_unspecialized_at_Py_FINALIZE on Windows #128860

[ZeroIntensity]

Triaging: is everything done here?

[picnixz]

No, the root cause is still not explained perfectly and we still have no perfect fix. My patch seems to work but we don't really know why it works. I'm keeping it open unless we decide to just ignore that issue =/

[vstinner]

Operating systems tested on: Windows, Linux

I cannot reproduce the bug on the current 3.12 branch on Linux with Python built in debug mode.

[picnixz]

On Linux there was no issue IIRC but we tested on both and observed that this failed on Windows 3.12 but maybe it works now?