Upgrade macOS to Python 3.11 to match Homebrew

[jwnimmer-tri]

See Homebrew/homebrew-core#114154.

• Drake: [workspace] Update macOS to python@3.11 #18262
• Drake External Examples: Switch to Python 3.11 on Mac drake-external-examples#248
• Rebuild CI provisioned images:
  • x86 vsphere
  • arm64 orka

[jwnimmer-tri]

The goal here would be to land a PR for this "immediately" (i.e., within ~1 day) after Homebrew switches their default version, and then do a Drake stable release immediately after that (\CC @SeanCurtis-TRI).

We'll need to update our macOS VM images concurrently with the PR.

I don't know if it's possible, but maybe we can test a Draft PR using Unprovisioned CI, by specifying the python3.11 paths more explicitly, even before the default changes? That wouldn't work in case numpy or stuff is missing though. I guess we'll see.

[svenevs]

Would just temporarily adding python 3.11 to the images (in addition to 3.10) be sufficient?

It's difficult to have CI choose a "variant image" (unless we create additional Jenkins jobs).

[jwnimmer-tri]

My guess was that a PR with this (below) would be easy to test in Unprovisioned?

--- a/setup/mac/binary_distribution/Brewfile
+++ b/setup/mac/binary_distribution/Brewfile
@@ -23,7 +23,7 @@ brew 'numpy'
 brew 'msgpack-cxx'
 brew 'openblas'
 brew 'pkg-config'
-brew 'python@3.10'
+brew 'python@3.11'
 brew 'spdlog'
 brew 'suite-sparse'
 brew 'tinyxml'

[svenevs]

=> #18262, added some jobs to start testing but will need more tomorrow.

We'll need to update our macOS VM images concurrently with the PR.

Sorry this part got me a little turned around. We can have a snapshot prepared with the python switch and just not switch Jenkins over until it lands.

[SeanCurtis-TRI]

and then do a Drake stable release immediately after that.

Looking at the PR, it looks like they're getting closer to merging. So, that means we need the release up and running very soon. Good to know.

[RussTedrake]

The initial goal says ~1 day after homebrew pulls the trigger. We're way past that now. Can we put an update here to help users (like me?) that are fighting against the upgrade?

On my local machines, I've just been avoiding at all costs any homebrew calls that would update python. But the mac CI for my course repositories keeps breaking.

[jwnimmer-tri]

Python 3.11 isn't the default yet: Homebrew/homebrew-core#114154, and https://github.com/RobotLocomotion/drake-external-examples is passing CI with 3.10.

Could you provide links to failures (and your CI setup scripts) so we can investigate?

[RussTedrake]

I see. Looking through the CI console, it seems that the default scipy uses python@3.11, which screws it up for everything else.

russt@TRI-FVFG13URQ6LT l4dc % brew info scipy
==> scipy: stable 1.9.3 (bottled), HEAD
Software for mathematics, science, and engineering
https://www.scipy.org
/opt/homebrew/Cellar/scipy/1.9.2 (1,344 files, 63.9MB)
  Poured from bottle on 2022-10-15 at 17:51:00
/opt/homebrew/Cellar/scipy/1.9.3_1 (1,344 files, 63.7MB) *
  Poured from bottle on 2022-11-13 at 06:46:44
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/scipy.rb
License: BSD-3-Clause
==> Dependencies
Build: libcython ✘, pythran ✘, swig ✘
Required: gcc ✔, numpy ✔, openblas ✔, pybind11 ✔, python@3.11 ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 7,181 (30 days), 19,253 (90 days), 62,199 (365 days)
install-on-request: 6,687 (30 days), 18,026 (90 days), 57,088 (365 days)
build-error: 0 (30 days)

Today's failure is

Installing scipy
==> Downloading ...
[...]
==> Installing dependencies for scipy: numpy, pybind11, openssl@1.1 and python@3.11
==> Installing scipy dependency: numpy
==> Pouring numpy--1.23.4_1.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/numpy/1.23.4_1: 1,732 files, 44.9MB
==> Installing scipy dependency: pybind11
==> Pouring pybind11--2.10.1.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/pybind11/2.10.1: 240 files, 3.8MB
==> Installing scipy dependency: openssl@1.1
==> Pouring openssl@1.1--1.1.1s.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/openssl@1.1/1.1.1s: 8,101 files, 18.5MB
==> Installing scipy dependency: python@3.11
==> Pouring python@3.11--3.11.0.monterey.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3-3.11
Target /usr/local/bin/2to3-3.11
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3-3.11'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.11

To list all files that would be deleted:
  brew link --overwrite --dry-run python@3.11

Possible conflicting files are:
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config
==> /usr/local/Cellar/python@3.11/3.11.0/bin/python3.11 -m ensurepip
==> /usr/local/Cellar/python@3.11/3.11.0/bin/python3.11 -m pip install -v --no-d
==> Summary
🍺  /usr/local/Cellar/python@3.11/3.11.0: 3,[163](https://github.com/RussTedrake/manipulation/actions/runs/3454903908/jobs/5766546518#step:4:164) files, 61.3MB
==> Installing scipy
==> Pouring scipy--1.9.3_1.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/scipy/1.9.3_1: 1,344 files, 70.4MB
Installing scipy has failed!
Homebrew Bundle failed! 1 Brewfile dependency failed to install.
Error: Process completed with exit code 1.

The original error happened on Nov 3, because without any changes, the rules_python was trying to pull torch through python3.11 and torch didn't support python3.11

 (ERROR: Could not find a version that satisfies the requirement torch==1.12.1 (from versions: none)
ERROR: No matching distribution found for torch==1.12.1
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/external/rules_python/python/pip_install/extract_wheels/__main__.py", line 5, in <module>
    main()
  File "/private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/external/rules_python/python/pip_install/extract_wheels/__init__.py", line 81, in main
    subprocess.run(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 569, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/python3', '-m', 'pip', '--isolated', 'wheel', '-r', '/Users/runner/work/manipulation/manipulation/requirements.txt', '--wheel-dir', '/private/var/tmp/_bazel_runner/7b4dc6001ecdbeb1b5ef1e14edba8c37/external/pip']' returned non-zero exit status 1.
)
Analyzing: 24 targets (54 packages loaded, 271 targets configured)
WARNING: errors encountered while analyzing target '//htmlbook/tools/python:defs': it will not be built

I managed to fix that by adding

# Try to force rules_python to find python@3.10
python3 --version
brew unlink python@3.10 && brew link --overwrite python@3.10
python3 --version

to my workflows/ci/monterey/setup

[jwnimmer-tri]

From a quick skim, it looks like Homebrew/homebrew-core#115544 just went in yesterday, which means that all users of Homebrew scipy must switch their code to use a non-default python3 Homebrew formula (@3.11). Homebrew doesn't make it easy to get CI setup recipes correct during its drawn-out Python version transition, but it seems like you have it under control now?

My inclination is still to defer Drake's switch to 3.11 until Homebrew/homebrew-core#114154 lands, so that the python3 on the $PATH is compatible with Drake.

[RussTedrake]

i don't have it under control, but rather than spend more time, i'll just wait out the transition i guess.

[jwnimmer-tri]

For reference, label:python-3.11-migration is a report about Python 3.11 transition status in Homebrew. It looks like scipy, pytorch, virtualenv, etc. will all do a flag-day transition from 3.10 to 3.11 on their own timescales, so users will only be able to use ~half of the available Python ecosystem for weeks at a time.

[RussTedrake]

Just FYI -- I've finally fixed my mac CI on my two course repos, which have been broken by the homebrew transition for over a month. As recommended by @jwnimmer-tri, I've decided to take scipy from pip instead of brew, with the expectation that I will probably have to take numpy from pip instead of brew, too. At the moment, I'm able to be successful with

numpy==1.24.1 ; sys_platform == "darwin"
scipy==1.10.0 ; sys_platform == "darwin" 

in my requirements, where the numpy version was pegged to match the version installed by homebrew.

[svenevs]

All of the ducks appear to be in a row here, preliminary CI results for x86 and arm64 seem to indicate no bad weather on the horizon. Marking as complete, if things go well with nightlies we expect to have a new release with py311 artifacts for mac 🥳 (edit: in the near future)