Add support for Python 3.11

[joshuataylor]

resolves #6147

Description

To support Python 3.11, various changes need to be made. Hopefully it won't be too much. I know that some BC Mashumaro changes will need to be made (see here).

The other adapters will need to rely on this. I believe the Snowflake adapter (the only one I have experience with) might be a bit painful, due to Snowflake not planning on releasing wheels soon. Pyarrow 10.0.1 does support 3.11, so we'll see. 🤷

There are a few changes to be made with the dataclasses - see here: https://docs.python.org/3.11/whatsnew/3.11.html#dataclasses

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
• I have run changie new to create a changelog entry

[jtcohen6]

@joshuataylor Thanks for taking the initiative here!! We have every intent of including 3.11 compatibility for dbt-core v1.4 release (January).

It's possible that, due to varying dependencies, some adapters may not support py311 by then — that's ok, it's why we opened separate issues to track the upgrade for each.

It looks like there are some failing tests to sort out; I have a hunch that these are related to the changes to field defaults in dataclasses, which you flagged, and that they might be tricky to sort out. I think I was running into the same error (AttributeError: '_MISSING_TYPE' object has no attribute 'replace_dict') when I made an attempt at this last month (48e6f96). My hacky approach was to make some changes to how we initialize dataclasses that are themselves attributes of other dataclasses. While it worked enough to get a dbt parse off the ground, mypy wasn't too thrilled.

cc @leahwicz - Let's plan to devote some engineering capacity to this soon. We can decide whether it makes sense to build off Josh's excellent start here.

[joshuataylor]

Maybe for connectors which don't support Python 3.11 (Snowflake for example says "Q1 2023") yet we can document how to get that connector to work with 3.11.

I'm also happy to continue working on this and hammering through it, what sort of timeline do y'all have to look into this? Week(s)? Otherwise I'll continue looking into it this week. I believe the main blockers are the dataclass changes in 3.11.

I'd also be happy to get the dbt-snowflake stuff to work, I have tested Snowflake with 3.11 on both Linux/MacOS with Apple Silicon with native wheels and it works nicely.

edit: Also to future people working on this -- please modify anything in this PR at your whim :)

[jtcohen6]

Timeline of next few weeks sounds right to me. We don't have this slated for our team's current sprint; we likely want to slot it in for the next one. If you're able to make progress in the meantime, amazing (+ much appreciated)!

[joshuataylor]

I'm shaving the hairiest yak with this one :-).

I've found a reference to python/mypy#1362 , as I'm getting this issue with mashumaro and defaults:

ValueError: mutable default <enum 'NVEnum'> for field novalue is not allowed: use default_factory

I believe bumping mypy to the newest version might help, will keep digging and let you know. I'm playing with dbt-snowflake as well, and I believe it's just a matter of updating the quote_policy to the new 3.11 way of doing things and referencing the new library and that should be good, which is nice. So far it looks like pretty minimal changes are needed for libraries, with a few changes for core around dataclasses 🎉 .

[joshuataylor]

@jtcohen6 I had to end up doing 0172c4e#diff-1f46b05ee5e8edb06d10f6a5104795ccda0d1c2c458a203db4914d68115978b5 , or mashumaro would complain that the default NVEnum had changed.

With this branch, and a tweak to dbt-snowflake to add the wheels for 3.11, dbt can compile and run our dbt snowflake project :-)

edit: Nevermind, looks like 3.1.1 fixes this. 🎉

[joshuataylor]

I've made a Snowflake PR here - dbt-labs/dbt-snowflake#336

Once the connector is released, we can bump setup.py and in theory it'll work and we won't be blocked waiting for this PR, a release can just be made with support for the new connector version 🎉

[jtcohen6]

@joshuataylor I'm going to close this in favor of #6326, which looks just about good to go. I cherry-picked one of your commits, and included you in the changelog entry :)