Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_cprofile crashed on main #95045

Closed
kumaraditya303 opened this issue Jul 20, 2022 · 9 comments
Closed

test_cprofile crashed on main #95045

kumaraditya303 opened this issue Jul 20, 2022 · 9 comments
Assignees
@kumaraditya303
Labels
3.10 only security fixes 3.11 only security fixes 3.12 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@kumaraditya303
Copy link
Contributor

Commit tested: 1834133

Reproducer:

./python -m test
Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7fe79c68cbb0
object refcount : 0
object type     : 0x55d1aa0d6f10
object type name: _lsprof.Profiler
object repr     : <refcnt 0 at 0x7fe79c68cbb0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007fe7a2694280 (most recent call first):
  Garbage-collecting
  File "/workspaces/cpython/Lib/test/test_cprofile.py", line 29 in test_bad_counter_during_dealloc
  File "/workspaces/cpython/Lib/unittest/case.py", line 579 in _callTestMethod
  File "/workspaces/cpython/Lib/unittest/case.py", line 623 in run
  File "/workspaces/cpython/Lib/unittest/case.py", line 678 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/test/support/testresult.py", line 140 in run
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1090 in _run_suite
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1216 in run_unittest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 281 in _test_module
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 742 in _main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 701 in main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 763 in main
  File "/workspaces/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/workspaces/cpython/Lib/runpy.py", line 88 in _run_code
  File "/workspaces/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase (total: 5)
Aborted (core dumped)

Interestingly it does not crashes if run with multiple processes.

cc @pablogsal
@kumaraditya303 kumaraditya303 added type-crash A hard crash of the interpreter, possibly with a core dump 3.12 bugs and security fixes labels Jul 20, 2022
@pablogsal
Copy link
Member

Does this happen on 3.11?

@kumaraditya303
Copy link
Contributor Author

3.11 does not seem affected, I tried running tests but haven't got any crashes yet.

@pablogsal
Copy link
Member

I am not able to reproduce this on my laptop. Will try again on my Linux machine later today

@kumaraditya303
Copy link
Contributor Author

I am able to reproduce on Linux with the current HEAD main.

@pablogsal
Copy link
Member

Can you bisect it?

@pablogsal
Copy link
Member

Cannot reproduce in my Linux box:

./python -m test
== CPython 3.12.0a0 (heads/main:23f6944c37, Jul 24 2022, 15:14:06) [GCC 12.1.0]
== Linux-5.18.13-arch1-1-x86_64-with-glibc2.35 little-endian
== cwd: /home/pablogsal/github/python/main/build/test_python_2216269æ
== CPU count: 36
== encodings: locale=UTF-8, FS=utf-8

 ........

== Tests result: SUCCESS ==

417 tests OK.

19 tests skipped:
    test_curses test_devpoll test_kqueue test_launcher test_msilib
    test_ossaudiodev test_smtpnet test_socketserver test_startfile
    test_tix test_tkinter test_ttk test_urllib2net test_urllibnet
    test_winconsoleio test_winreg test_winsound test_xmlrpc_net
    test_zipfile64

Total duration: 40 min 15 sec
Tests result: SUCCESS

@kumaraditya303
Copy link
Contributor Author

I can reproduce on main HEAD 78eb3f7

Crash:

Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7ff199c0b8c0
object refcount : 0
object type     : 0x556cad2891e0
object type name: _lsprof.Profiler
object repr     : <refcnt 0 at 0x7ff199c0b8c0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007ff19d4e6280 (most recent call first):
  Garbage-collecting
  File "/workspaces/cpython/Lib/test/test_cprofile.py", line 29 in test_bad_counter_during_dealloc
  File "/workspaces/cpython/Lib/unittest/case.py", line 579 in _callTestMethod
  File "/workspaces/cpython/Lib/unittest/case.py", line 623 in run
  File "/workspaces/cpython/Lib/unittest/case.py", line 678 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/unittest/suite.py", line 122 in run
  File "/workspaces/cpython/Lib/unittest/suite.py", line 84 in __call__
  File "/workspaces/cpython/Lib/test/support/testresult.py", line 140 in run
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1090 in _run_suite
  File "/workspaces/cpython/Lib/test/support/__init__.py", line 1216 in run_unittest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 281 in _test_module
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 317 in _runtest_inner2
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 360 in _runtest_inner
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 235 in _runtest
  File "/workspaces/cpython/Lib/test/libregrtest/runtest.py", line 265 in runtest
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 450 in run_tests_sequential
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 564 in run_tests
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 742 in _main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 701 in main
  File "/workspaces/cpython/Lib/test/libregrtest/main.py", line 763 in main
  File "/workspaces/cpython/Lib/test/__main__.py", line 2 in <module>
  File "/workspaces/cpython/Lib/runpy.py", line 88 in _run_code
  File "/workspaces/cpython/Lib/runpy.py", line 198 in _run_module_as_main

Extension modules: _testcapi, _xxsubinterpreters, _testinternalcapi, _testbuffer, _testmultiphase (total: 5)
Aborted (core dumped)

Environment:

== CPython 3.12.0a0 (heads/main:78eb3f79ea, Jul 27 2022, 10:43:00) [GCC 9.4.0]
== Linux-5.4.0-1074-azure-x86_64-with-glibc2.31 little-endian
== cwd: /workspaces/cpython/build/test_python_4447æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8

Another things is that if you remove test_asyncio then it passes, my guess is that it has something to do with the code specialization triggered by test_asyncio as that test executes a lot of code.

cc @markshannon @brandtbucher

I'll bisect later.

@bedevere-bot bedevere-bot mentioned this issue Jul 27, 2022
Merged
@kumaraditya303
Copy link
Contributor Author

kumaraditya303 commented Jul 27, 2022 •
edited
Loading

Turns out the profiler's deallocator did not untracked the object leading to the gc crash.

See #95315

@kumaraditya303 kumaraditya303 self-assigned this Jul 27, 2022
@kumaraditya303 kumaraditya303 added 3.11 only security fixes release-blocker labels Jul 27, 2022
@kumaraditya303 kumaraditya303 added the 3.10 only security fixes label Jul 27, 2022
miss-islington pushed a commit that referenced this issue Jul 27, 2022
@kumaraditya303
GH-95045: gc untrack _lsprof.Profiler before deallocating it (GH-95315)
deacf39 
Automerge-Triggered-By: GH:pablogsal
@bedevere-bot bedevere-bot mentioned this issue Jul 27, 2022
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 27, 2022
@kumaraditya303 @miss-islington
pythonGH-95045: gc untrack _lsprof.Profiler before deallocating it (p…
f499138 
…ythonGH-95315)

Automerge-Triggered-By: GH:pablogsal
(cherry picked from commit deacf39)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
@bedevere-bot bedevere-bot mentioned this issue Jul 27, 2022
Merged
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 27, 2022
@kumaraditya303 @miss-islington
pythonGH-95045: gc untrack _lsprof.Profiler before deallocating it (p…
a9a1434 
…ythonGH-95315)

Automerge-Triggered-By: GH:pablogsal
(cherry picked from commit deacf39)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
miss-islington added a commit that referenced this issue Jul 27, 2022
@miss-islington @kumaraditya303
GH-95045: gc untrack _lsprof.Profiler before deallocating it (GH-95315)
657efca 
Automerge-Triggered-By: GH:pablogsal
(cherry picked from commit deacf39)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
miss-islington added a commit that referenced this issue Jul 27, 2022
@miss-islington @kumaraditya303
GH-95045: gc untrack _lsprof.Profiler before deallocating it (GH-95315)
6b01fc7 
Automerge-Triggered-By: GH:pablogsal
(cherry picked from commit deacf39)

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
@kumaraditya303
Copy link
Contributor Author

Fixed by #95315

Repository owner moved this from Todo to Done in Release and Deferred blockers 🚫 Jul 27, 2022
@pablogsal pablogsal mentioned this issue Jul 27, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kumaraditya303 kumaraditya303

Labels
3.10 only security fixes 3.11 only security fixes 3.12 bugs and security fixes type-crash A hard crash of the interpreter, possibly with a core dump
Projects
Release and Deferred blockers 🚫
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pablogsal @kumaraditya303