SNOW-507597: Not supported error on Apple M1 #986
Comments
|
Hey @richiejthomas, we are starting to look into what will be needed to support M1 and will provide an update once we have completed our analysis. |
|
@sfc-gh-hchaturvedi I successfully just installed and ran dbt-snowflake on an M1 Macbook Pro using latest snowflake-connector-python master branch. There the pyarrow dependency is no longer a problem (current published latest version has a dependency issue with pyarrow 5). Could I ask the snowflake team to please publish a patch version from latest master (remember to bump the versio number :) )? I think that will solve everyone's problem. On a M1 Macbook: # Create a python 3.9 environment (3.10 did *not* work with dbt, I think there aren't arm wheel for 3.10?)
# I did `conda create --name dbt python=3.9`
# conda activate dbt
pip install https://github.com/snowflakedb/snowflake-connector-python/archive/refs/heads/master.zip # install snowflake-connector-python master
pip install dbt-core
# check `which dbt`, ensure it is the right one, otherwise reactivate environment
# call `dbt`, this should work
pip install dbt-snowflake
dbt deps # fix new dependencies
dbt run -m your_modelHere's the python process during |
|
@sfc-gh-hchaturvedi I'm also facing this issue - just wondering if there is any progress on this? |
|
It seems that version 2.7.4 supports M1 |
|
I upgraded to 2.7.4 ( When I tried to force the use of the To me it still looks like the Is there anything I need to change/install/reinstall to get this to work? |
|
I don't see pyarrow==6.0.1 anywhere in the requirements/dependencies of snowflake-connector-python nor in the |
|
So I tried uninstalling and installing the snowflake connector via pip (without cache) but that didn't work. Instead I decided to build from source. I followed the instructions in the readme to build the wheel from the repo and it works perfectly now. My guess is that the PyPI distribution is not built for ARM? I'm not entirely sure what exactly is uploaded to PyPI. |
|
Also wanted to add that installing via: also works for M1 instead of the PyPI install. |
|
I first installed via pip from the main branch. Then I uninstalled and installed 2.7.3 and 2.7.4 and it worked just fine for me. Unfortunately I am not too familiar with python toolchain to be sure if I by chance have different dependencies after I installed from main. |
|
@sfc-gh-dseweryn are you installing via PyPI? Or building the wheel from source? |
|
A couple of clarifications: If you're trying to install in Docker on an M1 Mac, then you have different requirements than if you were installing on your M1 Mac. We've had trouble with a lot of packages after we began running our docker containers natively on Mac rather than emulating them via Rosetta. Many of those problems are being worked out. The snowflake connector still has some issues: Installing on Mac, the following fails: $ pip install snowflake-connector-python[pandas,secure-local-storage]
zsh: no matches found: snowflake-connector-python[pandas,secure-local-storage]
I do not know how to solve this. Installing in Docker (Debian linux) on an M1 Mac: File "/root/venv/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 978, in fetch_pandas_all
raise NotSupportedError
snowflake.connector.errors.NotSupportedError: Unknown errorTry reinstalling snowflake-connector-python at version 2.7.2 by Starting from an empty Python 3.9 environment, you can get it running like this: pip install snowflake-connector-python[pandas,secure-local-storage]==2.7.4 --no-use-pep517
pip install -U snowflake-connector-python==2.7.2 |
|
The first time the connector installed successfully for me I used: then ran then I uninstalled snowflake-connector-python, updated I hope this clarifies everything :) |
|
@thomasaarholt So you have to put quotes around the package to install via zsh: When I do this I have the pyarrow dependencies, but they don't appear to be built for ARM when you download from PyPI. Also I have no problem running debian docker containers with |
|
@sfc-gh-dseweryn thanks for the clarification. From a new environment, If you try to install from PyPI with no cache I'm wondering if the first install from source is cached, and then the other subsequent installs are working from the locally cached wheel instead of going out to PyPI for the wheel. Because for my M1 mac it seems to only work when I build the wheel locally. When I download from PyPI the wheel doesn't work. I just tested with a new environment both building from source and getting the wheel from PyPI. Building from source (this works - note the wheel name has arm64 in it): This wheel doesn't work on M1 from PyPI: So you can see that when building from source we have: Maybe it is an OS issue? MacOS 11.0 vs MacOS 10.14? But I am not sure if the universal wheel is actually universal for M1. I think that this is the underlying issue, and there needs to be a wheel in PyPI which is compatible with M1. |
|
@kzecchini god, that's an unfortunate need of quotes! Thanks for pointing that out! I've noticed the M1 arm vs universal issue too. I know that pip will always prefer specific wheels over universal wheels if available. I notice the arm ones are not available on pypi. If you force building the wheels, then it will work on M1 mac: |
|
I successfully run sql queries into pandas using the following dockerfile: Where from snowflake.connector import connect
con = connect(
user=my_username,
password=my_password,
account=sql_account,
)
df = con.cursor().execute("select * from database.schema.my_table limit 10").fetch_pandas_all()
print("It worked!")
print(df)Run as
|
|
@thomasaarholt Thanks for the info! I just verified that adding the So it just seems like the solution for PyPI would be to get the ARM distribution uploaded so folks could easily install a prebuilt wheel that works on M1. But the workaround for now is to either install from source, or to use the |
|
@thomasaarholt Thanks for the workaround using the |
|
Annoyingly, the above will not build using google clouds docker using buildx to build multi-arch images (specifically building an arm image on x86_64). It fails with a qemu fatal error. |
|
Hi, I was facing the same error (M1 mac), and this thread really helped me out. The suggestion of using However, during this, I discovered another issue, although after spending some time debugging it. I kinda find a solution for it, but I'm still not sure why that error was happening. I thought it would be a good idea to put it here, in case of some face the same issue as me, it might help them. So, someone smarter and well-versed, in this topic, than me (like @thomasaarholt ) might be able to explain to me why it was happening. So, after you use the
This is the same error I was getting while trying to run Just to give the EXACT steps I followed, in case, someone wants to recreate it: At this point, running Everything here installs perfectly but running After debugging it for some time, I figure out the cause of it. Basically, it is not jupyterlab (or other similar packages) that is causing the issue, but one of its dependencies - CFFI (https://cffi.readthedocs.io/en/latest/). CFFI is installed with snowflake-connector-python as one of it's dependencies and the package works perfectly, but when you install jupyterlab conda installs cffi again (as we know conda doesn't check for packages installed by PyPI). You can recreate the same error just by doing Work around I found, just install jupyterlab or any other package which has cffi as it's dependency through PYPI in the conda virtual environment. However, I'm still confused why does cffi installation from PyPI works, but from Conda doesn't, even though conda-forge has osx-arm compatible installer for cffi. My guess is - this dependency is the same reason why I hope it helps anyone who might struggle with this issue, or if someone can tell me why this issue is happening in the first place! |
|
So the problem that this issue is discussing: means that this package's c-extension is not available. I can see this being a wide spread issue before I think Anaconda environments could have more issues, but I'm not familiar with them. |
|
^^ This is the right way to solve this problem! Thanks @sfc-gh-mkeller 🙌 |
|
@sfc-gh-mkeller this is still an issue because the snowflake connector wheels (as of 2.7.7) aren't built for arm: see also #1089 (comment) could the snowflake connector wheels be built for arm? |
|
We're currently working on releasing ARM builds for our Python connector |
|
In my case, the solution we found is before running Pipfile install
Any update soon? cheers |
|
running python 3.9 solved my issue |
|
So we have been releasing MacOS ARM wheels since release 2.7.8. Please be aware that this does not mean that you won't see: This error could mean 2 things:
Hopefully most of you are running into this because the C-extension is missing and this should have been helped by us releasing MacOS ARM wheels. |
|
@thomasaarholt solution worked for me. I don't need dbt, but whatever it worked on a 2022 MacBook Pro M1 running Venture. The standard install method game me a dependency error related to PyArrow |
|
Thanks for this info everyone. I was having trouble installing the connector ( https://pypi.org/project/snowflake-connector-python/3.0.0/#files The connector installed fine on Python 3.10.6, 3.10.10, 3.11.2. We can see that a wheel isn't fetched for 3.7 (Makes sense), but it does for |
|
These are also some follow up steps I needed to do to get it to successfully execute a query and avoid the |
|
@fearedbliss our limitation comes from |
|
@sfc-gh-mkeller Thanks Mark. We'll give that a shot. Worse case scenario, we'll need to start working on a Python 3.7 -> 3.11 migration. Given 3.7 has an EOL of 2023-06-27, it wouldn't be the worse thing to do haha. Plus 3.11 has some very nice speed improvements, error handling and reporting improvements as well. Thanks LLVM and Rust for pushing the ecosystem forward! |
|
@sfc-gh-mkeller It doesn't seem like it's really possible to easily get a Python 3.7 M1 build given what I looked at from PyArrow's codebase and some comments in the code. I'm starting the process of migrating our stuff to Python 3.11 (from Python 3.7) across our infra. I guess one of the benefits is that we'll be able to get Py 3.11 support until 2027! But yea, I'll need to do the work to get this to happen though. |
Please answer these questions before submitting your issue. Thanks!
What version of Python are you using?
Python 3.9.1
What operating system and processor architecture are you using?
macOS-12.0.1-arm64-arm-64bit
pip freeze)?asn1crypto==1.4.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.7
cryptography==3.4.8
idna==3.3
numpy==1.21.4
oscrypto==1.2.1
pandas==1.3.4
pyarrow==5.0.0
pycparser==2.21
pycryptodomex==3.11.0
PyJWT==2.3.0
pyOpenSSL==20.0.1
python-dateutil==2.8.2
pytz==2021.3
requests==2.26.0
six==1.16.0
snowflake-connector-python==2.7.1
urllib3==1.26.7
import pandas as pd
import snowflake.connector
conn = snowflake.connector.connect(
user='',
password='',
account='',
warehouse='',
database='',
schema=''
)
cur = conn.cursor()
sql = "select id from <TABLE_NAME>"
cur.execute(sql)
df = cur.fetch_pandas_all()
print(df)
Expected to see dataframe values printed although saw this error
File "lib/python3.9/site-packages/snowflake/connector/cursor.py", line 968, in fetch_pandas_all
raise NotSupportedError
snowflake.connector.errors.NotSupportedError: Unknown error
Process finished with exit code 1
Cannot attach full log since it included confidential info although the error might be happening here
2021-11-22 23:01:56,853 - MainThread cursor.py:616 - execute() - DEBUG - executing SQL/command
2021-11-22 23:01:56,853 - MainThread cursor.py:649 - execute() - DEBUG - binding: [select id from <TABLE_NAME>] with input=[None], processed=[{}]
2021-11-22 23:01:56,854 - MainThread cursor.py:696 - execute() - INFO - query: [select id from <TABLE_NAME>]
2021-11-22 23:01:56,854 - MainThread cursor.py:426 - _execute_helper() - DEBUG - Cannot use arrow result format, fallback to json format
The text was updated successfully, but these errors were encountered: