Python 3.11

[Bo98]

I'm opening this to discuss changes that I've proposed to be made to Python 3.11. These don't have to happen if people disagree. I do not plan to backport these.

There are two areas of concern that I'm hoping to address:

1. Parity with how Apple and Python themselves ship Python. Even MacPorts.
2. Legality of the effect of GPL dependencies on dependents, given how widespread our Python is now being used. Notably we can't exactly legally ship anything that might have been using readline or gdbm with something that uses GPL-2.0-only or OpenSSL 1.1.

To achieve this, this PR has the following changes:

• gdbm is now available as a separate formula python-gdbm@3.11
  • Compatibility concerns:
    • dbm.gnu is not available until you install that separate formula.
    • dbm.ndbm no longer uses gdbm_compat. It uses the system ndbm implementation (on macOS) and Berkeley DB 5 on Linux.
      • Users who need to keep a database created by dbm.ndbm in an older version of Python may experience compatibility issues. You'll need to read your database using the older version of Python and convert to another format.
    • dbm defaulted to dbm.gnu, and will still do when python-gdbm@3.11 is installed.
    • For those who can't migrate yet, Python 3.10 will continue to be available for at least 4 years.
• readline implementation has been switched to libedit
  • Compatibility concerns:
    • API wise, the two are largely compatible with one exception: the initialization file may have a different format, as documented at https://docs.python.org/3/library/readline.html.
    • For the Python Shell/REPL, .python_history may have compatibility issues between libedit and readline.
      • This incompatbility already existed if you switched between Homebrew Python and Apple Python.
      • If your previous history file was never written by libedit before (i.e. you have never used another Python shell other than Homebrew) then it may reset upon using Homebrew Python 3.11's shell. Otherwise it will typically convert ok (at least from my limited testing).
      • Spaces in history items will appear "corrupt" when going back to a previous version of Python that still uses GNU readline.
      • We could patch to change the history filename if absolutely necessary (though arguably the previous GNU readline implementation should have done that instead).
  • Note that SQLite, another Python dependency, also uses readline. Ideally that would change too but it isn't actually used in libsqlite3 so can probably get away with it if there's strong opposition to changing that.

We don't have to do these changes, but if we don't we'll need better auditing of where the readline and dbm modules are used and the licensing compatibility.

[yurikoles]

3.11 is released.

[cclauss]

Do we need to include gdbm and tkinter in this pull request? It can easily take six months to get a new version of Python to be the homebrew default. The more complex this PR is, the longer it will take for it to be ratified. Small moves.

[Bo98]

Do we need to include gdbm and tkinter in this pull request? It can easily take six months to get a new version of Python to be the homebrew default. The more complex this PR is, the longer it will take for it to be ratified. Small moves.

We will not be making breaking changes in a patch release. Breaking changes in a new formula is ok because it's a separate formula, with no forced upgrade path.

tkinter shouldn't even be a problem given we've done the same for 2 years now?

If there's strong opposition to gdbm I can of course drop that. Though no one has voiced either way yet, probably given it was rc2 I tested and we don't ship non-stable releases. Will push the final version in a few minutes, and will make sure this is merged in some form this week.

[iMichka]

Looks good to me.

Just one question, should we add --disable-dbm to python@3.11 to avoid opportunistic building/linking?

[iMichka]

Might also want one more review from someone else, in case I missed something during my review.

[Bo98]

Just one question, should we add --disable-dbm to python@3.11 to avoid opportunistic building/linking?

--with-dbmliborder controls that. I made sure it does not contain gdbm.

[MikeMcQuaid]

Makes sense to me! All questions are non-blocking except if we can remove the revision (if we can't: don't and ship regardless).

[MikeMcQuaid]

Not seen this DSL before but: I really like it. To attempt a nerd-snipe: if anyone else agrees with me it might be cool to allow a similar DSL for install or post_install or similar for versioned formulae to share more code. If anyone else agrees: I'll write up an issue. CC @Homebrew/maintainers for thoughts.

[EricFromCanada]

It's documented but rarely used for versioned formulae; it's instead found in those that are derived from / entangled with another formula, e.g. lemon.

[MikeMcQuaid]

@EricFromCanada Yeh, I was more thinking a differing DSL something like

def install
  formula "python@3.11"
end

or something

[Bo98]

No objections after 24 hours of green CI. We're good to go.

[BrewTestBot]

@Bo98 has triggered a merge.

[Bo98]

It can easily take six months to get a new version of Python to be the homebrew default.

Just to touch on this. I do not think we should be taking that long. Half of the point of moving to a versioned formulae approach was to detach the default from dependents.

If this is proven stable enough with no issues reported after a few weeks, and we've got basics supporting 3.11 like six (i.e. the few formulae which support multiple versions of Python), I don't see why we can't change the default by the end of the year.

If we can't, then I consider the process to have failed somewhere given everything we ship should be able to work regardless of what python3 points to.

[cclauss]

I don't see why we can't change the default by the end of the year.

This pull request went really smoothly! Thanks for making this happen. Brew rules!

[carlocab]

I'm unclear as to why we didn't change the default now, actually. Not sure why support in six, etc. should be a prerequisite?

[Bo98]

I'm unclear as to why we didn't change the default now, actually

Less moving parts and avoids having to do a long CI run, so users can get Python 3.11 in < 2 days after release.

We could definitely try change the default now though. Open a separate PR so we can do a CI run to check.

[ahankinson]

Thank you so much for this, and so quickly! Is there any way to get this new formula to install itself as python3 on the command-line? I noticed this with brew info python@3.11:

If you do not need a specific version of Python, and always want Homebrew's `python3` in your PATH:
  brew install python3

but this seems to prevent the case where a user needs a specific version of Python (3.11) but also always wants Homebrew's python3 on their PATH to point to that version. Otherwise python3 points to the Apple-installed version, which is definitely not what is wanted.

In the past brew link python@3.11 worked to "install" a specific version of Python as python3, but it seems like this no longer works...

[Bo98]

Unversioned and major-versioned symlinks `python`, `python3`, `python-config`, `python3-config`, `pip`, `pip3`, etc. pointing to
`python3.11`, `python3.11-config`, `pip3.11` etc., respectively, have been installed into
  /usr/local/opt/python@3.11/libexec/bin

[carlocab]

We could definitely try change the default now though. Open a separate PR so we can do a CI run to check.

Yep, we can do that when we catch up with the PR backlog.

[cclauss]

@ahankinson You can brew install python-launcher and then use py instead of python3.

brew home python-launcher to find out more.

py --list to see the versions of Python installed.