Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-104909: Split some more insts into ops #109943

Merged
merged 10 commits into from
Sep 27, 2023

Conversation

gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Sep 27, 2023

I started by adding some code that shows the most deserving non-viable opcodes. Also doubling max trace size again.

I could do this all day, but the stats show there are diminishing returns. Output from python3 Tools/cases_generator/generate_cases.py -v before splitting the top 5:

Read 186 instructions, 46 ops, 207 macros, 12 pseudos, and 14 families from Python/bytecodes.c
The following ops are not viable uops:
      1,906,418,185 LOAD_ATTR_METHOD_WITH_VALUES        (specialization of LOAD_ATTR)
      1,408,928,326 LOAD_ATTR_METHOD_NO_DICT            (specialization of LOAD_ATTR)
      1,359,753,529 LOAD_ATTR_SLOT                      (specialization of LOAD_ATTR)
      1,064,113,118 STORE_ATTR_SLOT                     (specialization of STORE_ATTR)
        981,677,187 STORE_ATTR_INSTANCE_VALUE           (specialization of STORE_ATTR)
        958,283,266 YIELD_VALUE                        
        777,829,833 CALL                                     (unspecialized)
        490,635,610 SEND_GEN                            (specialization of SEND)
        436,706,438 JUMP_FORWARD                       
        430,588,960 EXTENDED_ARG                       
        391,017,977 JUMP_BACKWARD_NO_INTERRUPT         
        351,940,012 LOAD_ATTR_MODULE                    (specialization of LOAD_ATTR)
        347,159,058 LOAD_ATTR_WITH_HINT                 (specialization of LOAD_ATTR)
        308,391,927 FOR_ITER                                 (unspecialized)
        251,204,370 RETURN_GENERATOR                   
        241,313,761 CALL_LIST_APPEND                    (specialization of CALL)
        177,791,632 CALL_PY_WITH_DEFAULTS               (specialization of CALL)
        175,204,354 CALL_KW                            
        163,262,604 FOR_ITER_GEN                        (specialization of FOR_ITER)
        146,454,299 BINARY_SUBSCR_GETITEM               (specialization of BINARY_SUBSCR)
        143,423,491 LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES (specialization of LOAD_ATTR)
        136,661,968 STORE_FAST_LOAD_FAST               
        136,511,856 LOAD_ATTR_CLASS                     (specialization of LOAD_ATTR)
        112,730,831 SEND                                     (unspecialized)
         92,715,546 MAKE_CELL                          
         83,410,408 CALL_FUNCTION_EX                   
         69,188,245 CALL_ALLOC_AND_ENTER_INIT           (specialization of CALL)
         56,944,041 LOAD_ATTR_METHOD_LAZY_DICT          (specialization of LOAD_ATTR)
         52,280,673 LOAD_ATTR_PROPERTY                  (specialization of LOAD_ATTR)
         50,964,787 STORE_ATTR_WITH_HINT                (specialization of STORE_ATTR)
         22,352,640 LOAD_ATTR_NONDESCRIPTOR_NO_DICT     (specialization of LOAD_ATTR)
          8,497,626 RERAISE                            
          6,000,000 END_ASYNC_FOR                      
          5,253,976 BEFORE_WITH                        
          2,897,006 RAISE_VARARGS                      
          2,005,368 IMPORT_FROM                        
          1,968,468 IMPORT_NAME                        
              8,160 BEFORE_ASYNC_WITH                  
              5,907 RESUME                                   (unspecialized)
              1,540 LOAD_SUPER_ATTR                          (unspecialized)
                801 CLEANUP_THROW                      
                    ENTER_EXECUTOR                     
                    LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN   (specialization of LOAD_ATTR)

I took the counts from here (follow the pystats raw link).

  • Curious what's up with LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN? It was the same on the weekly run a week before; the week before that (Sept 10) the file format was different. @brandtbucher any intuition on this?

@gvanrossum
Copy link
Member Author

Another couple of questions for @brandtbucher (or @markshannon).

  • Sometimes there's duplication between two uops, e.g. _GUARD_DORV_VALUES and _STORE_ATTR_INSTANCE_VALUE both set dorv = *_PyObject_DictOrValuesPointer(owner);. In Tier 1 the assumption has always been that the C compiler's optimizer will eliminate the duplication (it's a static inline function). But in the Tier 2 interpreter, we end up executing that code twice (once in each uop), and in the copy-and-patch machine code it also takes up twice the space. It doesn't look too bad, because after removing asserts it's just obj - 3, and a modern CPU should do this very fast, but there might be other occurrences of this kind of duplication that are slightly less minimal.
  • There's an assert(type_version != 0); in the _GUARD_TYPE_VERSION macro. This existed in most of the instructions I split up, except in LOAD_ATTR_METHOD_NO_DICT. Was that an oversight, or can it be zero for that opcode? (If so, then my PR is incorrect.)
  • In _LOAD_ATTR_SLOT, which is an "action" uop, I left a DEOPT_IF call. This is unprincipled, but it seems to work, and splitting this off into its own guard would duplicate much of the work this uop does (in particular the indirect load). Thoughts?

@gvanrossum gvanrossum changed the title Split some more insts into ops gh-104909: Split some more insts into ops Sep 27, 2023
@Fidget-Spinner
Copy link
Member

  • In _LOAD_ATTR_SLOT, which is an "action" uop, I left a DEOPT_IF call. This is unprincipled, but it seems to work, and splitting this off into its own guard would duplicate much of the work this uop does (in particular the indirect load). Thoughts?

Not Mark or Brandt, but on the optimization side, we would have to treat this as an impure op, which makes most of the optimization opportunities go away. However, we're likely converting some of these to LOAD_CONST anyways so that might mitigate it?

@gvanrossum
Copy link
Member Author

Not Mark or Brandt, but on the optimization side, we would have to treat this as an impure op, which makes most of the optimization opportunities go away.

We already have other actions using DEOPT_IF (e.g. _LOAD_GLOBAL_BUILTINS). We can split all these up later. My first goal is just to be able to translate more bytecodes to Tier 2, I'm slowly working my way through the most frequently executed instructions (per the table shown above).

However, we're likely converting some of these to LOAD_CONST anyways so that might mitigate it?

That's still a bit in the future though. And I'm not sure how often LOAD_ATTR_SLOT in particular would become LOAD_CONST (since it specifically fetches an instance attribute). It seems more likely for some other variants like LOAD_ATTR_MODULE, and that would probably load the result from the cache, not be transformed in (some guard +) LOAD_CONST.

@gvanrossum
Copy link
Member Author

Going to merge this soon, basically as-is. Perf is same as base (1.00 faster).

@gvanrossum gvanrossum marked this pull request as ready for review September 27, 2023 17:44
@gvanrossum
Copy link
Member Author

I can't find a persistent pattern in the failing tests. Maybe it's a timeout in multiprocessing? I believe @vstinner recently made flaky tests more likely to cause CI failure, maybe it's that?

@gvanrossum
Copy link
Member Author

Well, okay, some tests fail consistently, so I have to bisect. Sorry for the accusations.

@gvanrossum
Copy link
Member Author

Looks like LOAD_ATTR_METHOD_WITH_VALUES is the culprit.

I had dropped one of the DEOPT_IF() calls! :-(
@gvanrossum gvanrossum merged commit 5bb6f0f into python:main Sep 27, 2023
@gvanrossum gvanrossum deleted the load-attr-uops branch September 27, 2023 22:27
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable LTO + PGO 3.x has failed when building commit 5bb6f0f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/29/builds/4838) and take a look at the build logs.
  4. Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/29/builds/4838

Failed tests:

  • test.test_concurrent_futures.test_shutdown

Failed subtests:

  • test_interpreter_shutdown - test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown
    self.assertFalse(err)
AssertionError: b'Exception in thread Thread-1:\nTraceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/threading.py", line 1066, in _bootstrap_inner\n    self.run()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/concurrent/futures/process.py", line 344, in run\n    self.add_call_item_to_queue()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/concurrent/futures/process.py", line 399, in add_call_item_to_queue\n    self.call_queue.put(_CallItem(work_id,\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/queues.py", line 94, in put\n    self._start_thread()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/queues.py", line 177, in _start_thread\n    self._thread.start()\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/threading.py", line 985, in start\n    _start_new_thread(self._bootstrap, ())\nRuntimeError: can\'t create new thread at interpreter shutdown\nTraceback (most recent call last):\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 274, in main\n    code = _serve_one(child_r, fds,\n           ^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one\n    code = spawn._main(child_r, parent_sentinel)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/spawn.py", line 132, in _main\n    self = reduction.pickle.load(from_parent)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__\n    self._semlock = _multiprocessing.SemLock._rebuild(*state)\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFileNotFoundError: [Errno 2] No such file or directory\n' is not false

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian PGO 3.x has failed when building commit 5bb6f0f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/249/builds/6464) and take a look at the build logs.
  4. Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/249/builds/6464

Failed tests:

  • test.test_multiprocessing_fork.test_processes

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 29, done.        
remote: Counting objects:   3% (1/29)        
remote: Counting objects:   6% (2/29)        
remote: Counting objects:  10% (3/29)        
remote: Counting objects:  13% (4/29)        
remote: Counting objects:  17% (5/29)        
remote: Counting objects:  20% (6/29)        
remote: Counting objects:  24% (7/29)        
remote: Counting objects:  27% (8/29)        
remote: Counting objects:  31% (9/29)        
remote: Counting objects:  34% (10/29)        
remote: Counting objects:  37% (11/29)        
remote: Counting objects:  41% (12/29)        
remote: Counting objects:  44% (13/29)        
remote: Counting objects:  48% (14/29)        
remote: Counting objects:  51% (15/29)        
remote: Counting objects:  55% (16/29)        
remote: Counting objects:  58% (17/29)        
remote: Counting objects:  62% (18/29)        
remote: Counting objects:  65% (19/29)        
remote: Counting objects:  68% (20/29)        
remote: Counting objects:  72% (21/29)        
remote: Counting objects:  75% (22/29)        
remote: Counting objects:  79% (23/29)        
remote: Counting objects:  82% (24/29)        
remote: Counting objects:  86% (25/29)        
remote: Counting objects:  89% (26/29)        
remote: Counting objects:  93% (27/29)        
remote: Counting objects:  96% (28/29)        
remote: Counting objects: 100% (29/29)        
remote: Counting objects: 100% (29/29), done.        
remote: Compressing objects:   6% (1/15)        
remote: Compressing objects:  13% (2/15)        
remote: Compressing objects:  20% (3/15)        
remote: Compressing objects:  26% (4/15)        
remote: Compressing objects:  33% (5/15)        
remote: Compressing objects:  40% (6/15)        
remote: Compressing objects:  46% (7/15)        
remote: Compressing objects:  53% (8/15)        
remote: Compressing objects:  60% (9/15)        
remote: Compressing objects:  66% (10/15)        
remote: Compressing objects:  73% (11/15)        
remote: Compressing objects:  80% (12/15)        
remote: Compressing objects:  86% (13/15)        
remote: Compressing objects:  93% (14/15)        
remote: Compressing objects: 100% (15/15)        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 15 (delta 14), reused 1 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '5bb6f0fcba663e1006f9063d1027ce8bd9f8effb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 5bb6f0fcba gh-104909: Split some more insts into ops (#109943)
Switched to and reset branch 'main'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:2821: clean-retain-profile] Error 1 (ignored)
In file included from Python/ceval.c:766:
Python/generated_cases.c.h: In function ‘_PyEval_EvalFrameDefault’:
Python/generated_cases.c.h:2634:31: warning: unused variable ‘tp’ [-Wunused-variable]
 2634 |                 PyTypeObject *tp = Py_TYPE(owner);
      |                               ^~
Python/generated_cases.c.h:3593:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3593 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
Python/generated_cases.c.h:3642:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3642 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
In file included from Python/executor.c:87:
Python/executor_cases.c.h: In function ‘_PyUopExecute’:
Python/executor_cases.c.h:1739:27: warning: unused variable ‘tp’ [-Wunused-variable]
 1739 |             PyTypeObject *tp = Py_TYPE(owner);
      |                           ^~
Python/executor_cases.c.h:2302:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2302 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
Python/executor_cases.c.h:2348:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2348 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
./Modules/readline.c: In function ‘setup_readline’:
./Modules/readline.c:1256:21: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1256 |     rl_startup_hook = on_startup_hook;
      |                     ^
./Modules/readline.c:1258:23: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1258 |     rl_pre_input_hook = on_pre_input_hook;
      |                       ^
./Modules/socketmodule.c: In function ‘getsockaddrarg’:
./Modules/socketmodule.c:2468:9: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 2468 |         strncpy((char *)sa->salg_name, name, sizeof(sa->salg_name));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/_decimal/libmpdec/context.c:56: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time

./Modules/_decimal/libmpdec/context.c:56: warning: mpd_setminalloc: ignoring request to set MPD_MINALLOC a second time

In file included from Python/ceval.c:766:
Python/generated_cases.c.h: In function ‘_PyEval_EvalFrameDefault’:
Python/generated_cases.c.h:2634:31: warning: unused variable ‘tp’ [-Wunused-variable]
 2634 |                 PyTypeObject *tp = Py_TYPE(owner);
      |                               ^~
Python/generated_cases.c.h:3593:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3593 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
Python/generated_cases.c.h:3642:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 3642 |                 PyTypeObject *owner_cls = Py_TYPE(owner);
      |                               ^~~~~~~~~
In file included from Python/executor.c:87:
Python/executor_cases.c.h: In function ‘_PyUopExecute’:
Python/executor_cases.c.h:1739:27: warning: unused variable ‘tp’ [-Wunused-variable]
 1739 |             PyTypeObject *tp = Py_TYPE(owner);
      |                           ^~
Python/executor_cases.c.h:2302:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2302 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
Python/executor_cases.c.h:2348:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
 2348 |             PyTypeObject *owner_cls = Py_TYPE(owner);
      |                           ^~~~~~~~~
./Modules/readline.c: In function ‘setup_readline’:
./Modules/readline.c:1256:21: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1256 |     rl_startup_hook = on_startup_hook;
      |                     ^
./Modules/readline.c:1258:23: warning: assignment to ‘int (*)(const char *, int)’ from incompatible pointer type ‘int (*)(void)’ [-Wincompatible-pointer-types]
 1258 |     rl_pre_input_hook = on_pre_input_hook;
      |                       ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘doParseXmlDecl’ at ./Modules/expat/xmltok.c:1197:13:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1075:9:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1081:12:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1088:9:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1115:7:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1043:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1128:9:
./Modules/expat/xmltok.c:390:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  390 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

make: *** [Makefile:2040: buildbottest] Error 5

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 LTO 3.x has failed when building commit 5bb6f0f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/567/builds/4991) and take a look at the build logs.
  4. Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/567/builds/4991

Failed tests:

  • test.test_multiprocessing_fork.test_processes

Summary of the results of the build (if available):

==

Click to see traceback logs
Note: switching to '5bb6f0fcba663e1006f9063d1027ce8bd9f8effb'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 5bb6f0fcba gh-104909: Split some more insts into ops (#109943)
Switched to and reset branch 'main'

In file included from Python/ceval.c:766:
Python/generated_cases.c.h: In function ‘_PyEval_EvalFrameDefault’:
Python/generated_cases.c.h:2634:31: warning: unused variable ‘tp’ [-Wunused-variable]
                 PyTypeObject *tp = Py_TYPE(owner);
                               ^~
Python/generated_cases.c.h:3593:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
                 PyTypeObject *owner_cls = Py_TYPE(owner);
                               ^~~~~~~~~
Python/generated_cases.c.h:3642:31: warning: unused variable ‘owner_cls’ [-Wunused-variable]
                 PyTypeObject *owner_cls = Py_TYPE(owner);
                               ^~~~~~~~~
In file included from Python/executor.c:87:
Python/executor_cases.c.h: In function ‘_PyUopExecute’:
Python/executor_cases.c.h:1739:27: warning: unused variable ‘tp’ [-Wunused-variable]
             PyTypeObject *tp = Py_TYPE(owner);
                           ^~
Python/executor_cases.c.h:2302:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
             PyTypeObject *owner_cls = Py_TYPE(owner);
                           ^~~~~~~~~
Python/executor_cases.c.h:2348:27: warning: unused variable ‘owner_cls’ [-Wunused-variable]
             PyTypeObject *owner_cls = Py_TYPE(owner);
                           ^~~~~~~~~
./Include/internal/pycore_semaphore.h:54:1: warning: type of ‘_PySemaphore_Wakeup’ does not match original declaration [-Wlto-type-mismatch]
 _PySemaphore_Wakeup(_PySemaphore *sema);
 ^
Python/parking_lot.c:197:1: note: ‘_PySemaphore_Wakeup’ was previously declared here
 _PySemaphore_Wakeup(_PySemaphore *sema)
 ^
Python/parking_lot.c:197:1: note: code may be misoptimized unless -fno-strict-aliasing is used
./Include/internal/pycore_semaphore.h:57:18: warning: type of ‘_PySemaphore_Init’ does not match original declaration [-Wlto-type-mismatch]
 PyAPI_FUNC(void) _PySemaphore_Init(_PySemaphore *sema);
                  ^
Python/parking_lot.c:53:1: note: ‘_PySemaphore_Init’ was previously declared here
 _PySemaphore_Init(_PySemaphore *sema)
 ^
Python/parking_lot.c:53:1: note: code may be misoptimized unless -fno-strict-aliasing is used
./Include/internal/pycore_semaphore.h:49:1: warning: type of ‘_PySemaphore_Wait’ does not match original declaration [-Wlto-type-mismatch]
 _PySemaphore_Wait(_PySemaphore *sema, _PyTime_t timeout_ns, int detach);
 ^
Python/parking_lot.c:178:1: note: ‘_PySemaphore_Wait’ was previously declared here
 _PySemaphore_Wait(_PySemaphore *sema, _PyTime_t timeout, int detach)
 ^
Python/parking_lot.c:178:1: note: code may be misoptimized unless -fno-strict-aliasing is used
./Include/internal/pycore_semaphore.h:58:18: warning: type of ‘_PySemaphore_Destroy’ does not match original declaration [-Wlto-type-mismatch]
 PyAPI_FUNC(void) _PySemaphore_Destroy(_PySemaphore *sema);
                  ^
Python/parking_lot.c:81:1: note: ‘_PySemaphore_Destroy’ was previously declared here
 _PySemaphore_Destroy(_PySemaphore *sema)
 ^
Python/parking_lot.c:81:1: note: code may be misoptimized unless -fno-strict-aliasing is used

make: *** [Makefile:2040: buildbottest] Error 5

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian 3.x has failed when building commit 5bb6f0f.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/424/builds/5033) and take a look at the build logs.
  4. Check if the failure is related to this commit (5bb6f0f) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/424/builds/5033

Failed tests:

  • test.test_multiprocessing_spawn.test_misc

Failed subtests:

  • test_dict - test.test_multiprocessing_spawn.test_misc.TestSyncManagerTypes.test_dict
  • test_bounded_semaphore - test.test_multiprocessing_spawn.test_misc.TestSyncManagerTypes.test_bounded_semaphore
  • test_condition - test.test_multiprocessing_spawn.test_misc.TestSyncManagerTypes.test_condition

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5931, in test_dict
    self.run_worker(self._test_dict, o)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5799, in run_worker
    self.wait_proc_exit()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5781, in wait_proc_exit
    join_process(self.proc)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 100, in join_process
    threading_helper.join_thread(process)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/threading_helper.py", line 113, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 52.5 seconds


Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5854, in test_condition
    self.run_worker(self._test_condition, o)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5799, in run_worker
    self.wait_proc_exit()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5781, in wait_proc_exit
    join_process(self.proc)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 100, in join_process
    threading_helper.join_thread(process)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/threading_helper.py", line 113, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 52.5 seconds


Traceback (most recent call last):
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5845, in test_bounded_semaphore
    self.test_semaphore(sname="BoundedSemaphore")
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5841, in test_semaphore
    self.run_worker(self._test_semaphore, o)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5799, in run_worker
    self.wait_proc_exit()
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 5781, in wait_proc_exit
    join_process(self.proc)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/_test_multiprocessing.py", line 100, in join_process
    threading_helper.join_thread(process)
  File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/support/threading_helper.py", line 113, in join_thread
    raise AssertionError(msg)
AssertionError: failed to join the thread in 52.5 seconds

csm10495 pushed a commit to csm10495/cpython that referenced this pull request Sep 28, 2023
These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR`
that weren't already viable uops:

* Split LOAD_ATTR_METHOD_WITH_VALUES
* Split LOAD_ATTR_METHOD_NO_DICT
* Split LOAD_ATTR_SLOT
* Split STORE_ATTR_SLOT
* Split STORE_ATTR_INSTANCE_VALUE

Also:

* Add `-v` flag to code generator which prints a list of non-viable uops
  (easter-egg: it can print execution counts -- see source)
* Double _Py_UOP_MAX_TRACE_LENGTH to 128



I had dropped one of the DEOPT_IF() calls! :-(
@vstinner
Copy link
Member

I believe @vstinner recently made flaky tests more likely to cause CI failure, maybe it's that?

Right. Tests are now run with --rerun-fail on GitHub Actions and buildbots: any test marks the whole build as a failure.

Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
These are the most popular specializations of `LOAD_ATTR` and `STORE_ATTR`
that weren't already viable uops:

* Split LOAD_ATTR_METHOD_WITH_VALUES
* Split LOAD_ATTR_METHOD_NO_DICT
* Split LOAD_ATTR_SLOT
* Split STORE_ATTR_SLOT
* Split STORE_ATTR_INSTANCE_VALUE

Also:

* Add `-v` flag to code generator which prints a list of non-viable uops
  (easter-egg: it can print execution counts -- see source)
* Double _Py_UOP_MAX_TRACE_LENGTH to 128



I had dropped one of the DEOPT_IF() calls! :-(
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants