python3.12 introduces numerous memory leaks (as reported by ASAN)

[Mekk]

Bug report

Bug description:

Let example illustrate:

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.10 -c 'print("Hello")'
Hello

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
  PYTHONMALLOC=malloc \
  /usr/bin/python3.11 -c 'print("Hello")'
Hello

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.12 -c 'print("Hello")'
Hello

=================================================================
==3317995==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 55877 byte(s) in 1063 object(s) allocated from:
    #0 0x7f5d145847ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x539ddf  (/usr/bin/python3.12+0x539ddf) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 30732 byte(s) in 614 object(s) allocated from:
    #0 0x7f5d145847ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5573b9  (/usr/bin/python3.12+0x5573b9) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

SUMMARY: AddressSanitizer: 86609 byte(s) leaked in 1677 allocation(s).

CPython versions tested on:

3.12.1

Operating systems tested on:

Linux

Linked PRs

• gh-113190: Reenable non-debug interned string cleanup #113601

[Mekk]

I assume those are various „global” objects and to a degree the report is spurious, nevertheless using sanitizer while maintaining binary python extensions and code embedding python turned out to be invaluable for me many times (esp. considering new/borrow reference semantics is often poorly documented and related code error-prone).

On py3.10 I had nice, clear state, on py3.12 the same code reports plenty of leaks.

Example above is only a start, slightly more complicated code (clear and sane on 3.10) starts reporting much more leaks, mostly mentioning _PyUnicode_New, _PyUnicode_JoinArray , _PyUnicode_FromASCII (all rooted somewhere inside python3.12 binary)

Tested on python 3.12.1 from deadsnakes.

[Mekk]

Maybe one more example:

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
  PYTHONMALLOC=malloc \
  /usr/bin/python3.10 -c 'import pytest'

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.12 -c 'import pytest'

=================================================================
==3331693==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 591154 byte(s) in 11373 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5573b9  (/usr/bin/python3.12+0x5573b9) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 111806 byte(s) in 2108 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x539ddf  (/usr/bin/python3.12+0x539ddf) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 815 byte(s) in 16 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x549a29 in _PyUnicode_FromASCII (/usr/bin/python3.12+0x549a29) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 390 byte(s) in 6 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5a0cfd in _PyUnicode_JoinArray (/usr/bin/python3.12+0x5a0cfd) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 104 byte(s) in 2 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x584ce0 in PyUnicode_Concat (/usr/bin/python3.12+0x584ce0) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

SUMMARY: AddressSanitizer: 704269 byte(s) leaked in 13505 allocation(s).

                            

[Mekk]

Staying in stdlib:

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.10 -c 'import re'

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.12 -c 'import re'

=================================================================
==3333908==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 74271 byte(s) in 1472 object(s) allocated from:
    #0 0x7f34a15807ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5573b9  (/usr/bin/python3.12+0x5573b9) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 57512 byte(s) in 1095 object(s) allocated from:
    #0 0x7f34a15807ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x539ddf  (/usr/bin/python3.12+0x539ddf) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 260 byte(s) in 4 object(s) allocated from:
    #0 0x7f34a15807ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5a0cfd in _PyUnicode_JoinArray (/usr/bin/python3.12+0x5a0cfd) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

SUMMARY: AddressSanitizer: 132043 byte(s) leaked in 2571 allocation(s).

[neonene]

If your cases come from the commit 67807cf (3.12.0 alpha 1), I'm waiting for the memory release to be implemented at the finalization phase, and #113055 may be partially related.

[Mekk]

I can't comment about reasons, but the bug you mention claims that some leak existed, and 3.12 only increased it's scale in case multiple interpreters are in use ( #113055 (comment) ).

In my case (a) there is single interpreter in use and (b) 3.10 and 3.11 don't report any leaks (neither in simple examples above, nor in more complicated real code).

[vstinner]

Valgrind doesn't see any leak in the current 3.12 branch:

$ PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./python -c 'print("Hello")'
Hello

valgrind.log:

(...)
==1662842== HEAP SUMMARY:
==1662842==     in use at exit: 0 bytes in 0 blocks
==1662842==   total heap usage: 31,703 allocs, 31,703 frees, 4,419,701 bytes allocated
==1662842== 
==1662842== All heap blocks were freed -- no leaks are possible
(...)

---

Direct leak of 390 byte(s) in 6 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5a0cfd in _PyUnicode_JoinArray (/usr/bin/python3.12+0x5a0cfd) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

I'm not sure why this traceback is truncated, but without a more complete traceback, I don't see how to debug this issue.

[vstinner]

It's possible to build Python with clang -fsanitize=memory:

./configure CC=clang LD=clang --with-memory-sanitizer && make

The problem is that all dependencies of Python, such as OpenSSL, should be built with the same option. Otherwise, clang can emit false alarms. Example in OpenSSL:

  Uninitialized value was created by a heap allocation
    #0 0x484596 in malloc (/home/vstinner/python/3.12/python+0x484596) (BuildId: 0f3caceddd1581f15aab325a45ae12048629d15a)
    #1 0x7f28271d064a in OBJ_txt2obj (/lib64/libcrypto.so.3+0x1d064a) (BuildId: 5d012dcc6f62f35dabb8a129d641e3df7b731f11)
    #2 0x7f2827790c1c in _ssl_txt2obj_impl /home/vstinner/python/3.12/./Modules/_ssl.c:5285:11
    #3 0x7f2827790c1c in _ssl_txt2obj /home/vstinner/python/3.12/./Modules/clinic/_ssl.c.h:1351:20
    #4 0x72f535 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/vstinner/python/3.12/Objects/methodobject.c:438:24

Right now on my Fedora 39, even with detect_leaks=0, I fail to build Python successfully :-(

ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0" make

[WillAyd]

I'm not sure why this traceback is truncated, but without a more complete traceback, I don't see how to debug this issue.

In case it is helpful at all I can replicate the issue with a slightly more complete traceback from 3.12.4:

Direct leak of 111239 byte(s) in 2183 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f90dc618 in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f90dc618 in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f90dc618 in _PyUnicode_FromUCS1 /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:2036
    #4 0x5f50f90dc618 in _PyUnicode_FromUCS1 /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:2022

Direct leak of 58060 byte(s) in 1111 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f90c419b in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f90c419b in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f90c419b in unicode_decode_utf8 /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:4693

Direct leak of 459 byte(s) in 7 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f911318a in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f911318a in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f911318a in _PyUnicode_JoinArray /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:9661

Direct leak of 50 byte(s) in 1 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f90d1da1 in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f90d1da1 in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f90d1da1 in _PyUnicode_FromASCII /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1998