Symbol not found: _deflate from grpc 1.12.0 #15510

ssarangi · 2018-05-22T19:44:19Z

What version of gRPC and what language are you using?

gRPC Version: 1.12.0 Python: 3.6.4 (From Anaconda)

What operating system (Linux, Windows, …) and version?

OSX (High Sierra)

What runtime / compiler are you using (e.g. python version or version of gcc)

Python: 3.6.4 (From Anaconda)

What did you do?

If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.
A simple grpc tutorial fails to run and fails on import.

What did you expect to see?

I expected the server to run.

What did you see instead?

python server.py
Traceback (most recent call last):
File "server.py", line 1, in
import grpc
File "/Users/satyajits/anaconda3/lib/python3.6/site-packages/grpc/init.py", line 22, in
from grpc._cython import cygrpc as _cygrpc
ImportError: dlopen(/Users/satyajits/anaconda3/lib/python3.6/site-packages/grpc/_cython/cygrpc.cpython-36m-darwin.so, 2): Symbol not found: _deflate
Referenced from: /Users/satyajits/anaconda3/lib/python3.6/site-packages/grpc/_cython/cygrpc.cpython-36m-darwin.so
Expected in: flat namespace
in /Users/satyajits/anaconda3/lib/python3.6/site-packages/grpc/_cython/cygrpc.cpython-36m-darwin.so

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment?

None.

ssarangi · 2018-05-22T20:05:32Z

I tried this with virtualenv as well and the same issue exists as well. So its not Anaconda specific.

Traceback (most recent call last):
File "server.py", line 1, in
import grpc
File "/Users/satyajits/dev/tmp/venv/lib/python3.6/site-packages/grpc/init.py", line 22, in
from grpc._cython import cygrpc as _cygrpc
ImportError: dlopen(/Users/satyajits/dev/tmp**/venv/lib/python3.6/**site-packages/grpc/_cython/cygrpc.cpython-36m-darwin.so, 2): Symbol not found: _deflate
Referenced from: /Users/satyajits/dev/tmp/venv/lib/python3.6/site-packages/grpc/_cython/cygrpc.cpython-36m-darwin.so
Expected in: flat namespace
in /Users/satyajits/dev/tmp/venv/lib/python3.6/site-packages/grpc/_cython/cygrpc.cpython-36m-darwin.so
(venv)

nathanielmanistaatgoogle · 2018-05-23T00:46:23Z

@mehrdada: would you be able to take a look at this in the next day or two?

mehrdada · 2018-05-23T01:13:18Z

Cannot reproduce on Python3 installed via brew:

Last login: Tue May 22 10:00:24 on ttys006
Carbon:~ m$ python3
-bash: python3: command not found
Carbon:~ m$ brew install python3
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core, caskroom/cask).
==> New Formulae
brew-php-switcher                libbitcoin-protocol              nifi-registry                    wireguard-go
==> Updated Formulae
cmake ✔               dash                  gnome-builder         libbitcoin-explorer   pam-u2f               sops
agedu                 dbxml                 goenv                 libbitcoin-server     pari                  source-to-image
ammonite-repl         dcd                   grafana               libidn2               passenger             spades
angular-cli           debianutils           grpc                  libphonenumber        pdftoedn              spdlog
annie                 dependency-check      gst-editing-services  libqalculate          pdftoipe              spigot
ansible               dhall-json            gst-libav             librealsense          pegtl                 spring-roo
apibuilder-cli        diff-pdf              gst-plugins-bad       libu2f-host           petsc                 srtp
aptly                 diffoscope            gst-plugins-base      libxlsxwriter         pgloader              stella
archey                dlib                  gst-plugins-good      libxo                 pilosa                stunnel
aria2                 dnscrypt-proxy        gst-plugins-ugly      logtalk               plantuml              swiftformat
armadillo             docfx                 gst-python            lxc                   plowshare             telegraf
artifactory           dscanner              gst-rtsp-server       mariadb               poppler               tepl
aws-sdk-cpp           e2fsprogs             gst-validate          maxwell               postgrest             texmath
awscli                emscripten            gstreamer             mercurial             pound                 tippecanoe
babl                  faas-cli              gtksourceview@4       meson                 pre-commit            traefik
bcal                  fabio                 heroku                mill                  proguard              twarc
binaryen              fabric                icarus-verilog        mitmproxy             psqlodbc              txr
bind                  fdroidserver          imagemagick           monero                pumba                 uhd
bitrise               feh                   imagemagick@6         mongo-c-driver        purescript            uncrustify
bmake                 firebase-cli          infer                 mpc                   qjackctl              vala
cake                  fluent-bit            ispc                  mupdf                 qpm                   vegeta
calicoctl             fn                    jenkins               mupdf-tools           qt                    vim
cglm                  folly                 jfrog-cli-go          mutt                  raylib                vte3
cimg                  fork-cleaner          jhiccup               n                     redshift              vtk
citus                 fq                    jsonpp                nano                  restic                watch
clang-format          fribidi               keepassc              neofetch              restview              wireguard-tools
cockroach             gauge                 kitchen-sync          node-build            rocksdb               x265
coffeescript          gegl                  knot                  node@8                root                  xonsh
convox                geoserver             kompose               nsd                   sagittarius-scheme    xqilla
coreos-ct             git-ftp               kubernetes-helm       nss                   shfmt                 you-get
cppcms                git-lfs               lastpass-cli          opensc                skaffold              youtube-dl
cromwell              gitlab-runner         launch4j              osm2pgrouting         snakemake             zopfli
curl                  glib-networking       libbitcoin-client     osquery               solr                  zsh-autosuggestions
==> Renamed Formulae
latexila -> gnome-latex

==> Installing dependencies for python: sphinx-doc, gdbm, openssl, readline, sqlite, xz
==> Installing python dependency: sphinx-doc
==> Downloading https://homebrew.bintray.com/bottles/sphinx-doc-1.7.4.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring sphinx-doc-1.7.4.high_sierra.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /Users/m/Library/Homebrew,
because this formula is mainly used internally by other formulae.
Users are advised to use `pip` to install sphinx-doc.

If you need to have this software first in your PATH run:
  echo 'export PATH="/Users/m/Library/Homebrew/opt/sphinx-doc/bin:$PATH"' >> ~/.bash_profile

==> Summary
🍺  /Users/m/Library/Homebrew/Cellar/sphinx-doc/1.7.4: 2,642 files, 36.8MB
==> Installing python dependency: gdbm
==> Downloading https://homebrew.bintray.com/bottles/gdbm-1.14.1_1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring gdbm-1.14.1_1.high_sierra.bottle.tar.gz
🍺  /Users/m/Library/Homebrew/Cellar/gdbm/1.14.1_1: 20 files, 555.7KB
==> Installing python dependency: openssl
==> Downloading https://www.openssl.org/source/openssl-1.0.2o.tar.gz
######################################################################## 100.0%
==> perl ./Configure --prefix=/Users/m/Library/Homebrew/Cellar/openssl/1.0.2o_1 --openssldir=/Users/m/Library/Homebrew/etc/openssl
==> make depend
==> make
==> make test
==> make install MANDIR=/Users/m/Library/Homebrew/Cellar/openssl/1.0.2o_1/share/man MANSUFFIX=ssl
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
  /Users/m/Library/Homebrew/etc/openssl/certs

and run
  /Users/m/Library/Homebrew/opt/openssl/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /Users/m/Library/Homebrew,
because Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries.

If you need to have this software first in your PATH run:
  echo 'export PATH="/Users/m/Library/Homebrew/opt/openssl/bin:$PATH"' >> ~/.bash_profile

For compilers to find this software you may need to set:
    LDFLAGS:  -L/Users/m/Library/Homebrew/opt/openssl/lib
    CPPFLAGS: -I/Users/m/Library/Homebrew/opt/openssl/include
For pkg-config to find this software you may need to set:
    PKG_CONFIG_PATH: /Users/m/Library/Homebrew/opt/openssl/lib/pkgconfig

==> Summary
🍺  /Users/m/Library/Homebrew/Cellar/openssl/1.0.2o_1: 1,784 files, 12.2MB, built in 5 minutes 40 seconds
==> Installing python dependency: readline
==> Downloading https://homebrew.bintray.com/bottles/readline-7.0.3_1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring readline-7.0.3_1.high_sierra.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /Users/m/Library/Homebrew,
because macOS provides the BSD libedit library, which shadows libreadline.
In order to prevent conflicts when programs look for libreadline we are
defaulting this GNU Readline installation to keg-only..

For compilers to find this software you may need to set:
    LDFLAGS:  -L/Users/m/Library/Homebrew/opt/readline/lib
    CPPFLAGS: -I/Users/m/Library/Homebrew/opt/readline/include

==> Summary
🍺  /Users/m/Library/Homebrew/Cellar/readline/7.0.3_1: 46 files, 1.5MB
==> Installing python dependency: sqlite
==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.23.1.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring sqlite-3.23.1.high_sierra.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /Users/m/Library/Homebrew,
because macOS provides an older sqlite3.

If you need to have this software first in your PATH run:
  echo 'export PATH="/Users/m/Library/Homebrew/opt/sqlite/bin:$PATH"' >> ~/.bash_profile

For compilers to find this software you may need to set:
    LDFLAGS:  -L/Users/m/Library/Homebrew/opt/sqlite/lib
    CPPFLAGS: -I/Users/m/Library/Homebrew/opt/sqlite/include
For pkg-config to find this software you may need to set:
    PKG_CONFIG_PATH: /Users/m/Library/Homebrew/opt/sqlite/lib/pkgconfig

==> Summary
🍺  /Users/m/Library/Homebrew/Cellar/sqlite/3.23.1: 11 files, 3MB
==> Installing python dependency: xz
==> Downloading https://homebrew.bintray.com/bottles/xz-5.2.4.high_sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring xz-5.2.4.high_sierra.bottle.tar.gz
🍺  /Users/m/Library/Homebrew/Cellar/xz/5.2.4: 92 files, 1MB
==> Installing python
==> Downloading https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tar.xz
######################################################################## 100.0%
==> ./configure --prefix=/Users/m/Library/Homebrew/Cellar/python/3.6.5 --enable-ipv6 --datarootdir=/Users/m/Library/Homebrew/Cella
==> make
==> make install PYTHONAPPSDIR=/Users/m/Library/Homebrew/Cellar/python/3.6.5
==> make frameworkinstallextras PYTHONAPPSDIR=/Users/m/Library/Homebrew/Cellar/python/3.6.5/share/python
==> Downloading https://files.pythonhosted.org/packages/72/c2/c09362ab29338413ab687b47dab03bab4a792e2bbb727a1eb5e0a88e3b86/setupto
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.
######################################################################## 100.0%
==> Downloading https://files.pythonhosted.org/packages/5d/c1/45947333669b31bc6b4933308dd07c2aa2fedcec0a95b14eedae993bd449/wheel-0
######################################################################## 100.0%
==> make html
==> /Users/m/Library/Homebrew/Cellar/python/3.6.5/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-script
==> /Users/m/Library/Homebrew/Cellar/python/3.6.5/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-script
==> /Users/m/Library/Homebrew/Cellar/python/3.6.5/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-script
==> Caveats
Python has been installed as
  /Users/m/Library/Homebrew/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /Users/m/Library/Homebrew/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run
  brew install python@2

Pip, setuptools, and wheel have been installed. To update them run
  pip3 install --upgrade pip setuptools wheel

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /Users/m/Library/Homebrew/lib/python3.6/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /Users/m/Library/Homebrew/Cellar/python/3.6.5: 9,139 files, 154.7MB, built in 4 minutes 7 seconds
Carbon:~ m$ python3
Python 3.6.5 (default, May 22 2018, 18:03:28)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Carbon:~ m$ cd `mktemp -d`
Carbon:tmp.2cTF23AS m$ python3
Python 3.6.5 (default, May 22 2018, 18:03:28)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
Carbon:tmp.2cTF23AS m$ python3 -m pip install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/b6/30/96a02b2287098b23b875bc8c2f58071c35d2efe84f747b64d523721dc2b5/virtualenv-16.0.0-py2.py3-none-any.whl (1.9MB)
    100% |████████████████████████████████| 1.9MB 9.4MB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-16.0.0
Carbon:tmp.2cTF23AS m$ python3 -m virtualenv .
Using base prefix '/Users/m/Library/Homebrew/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /private/var/folders/qx/zz09fcl11v9c1z9w3rh5g_jc0000gn/T/tmp.2cTF23AS/bin/python3.6
Also creating executable in /private/var/folders/qx/zz09fcl11v9c1z9w3rh5g_jc0000gn/T/tmp.2cTF23AS/bin/python
Installing setuptools, pip, wheel...done.
Carbon:tmp.2cTF23AS m$ bin/pip install grpcio
Collecting grpcio
  Downloading https://files.pythonhosted.org/packages/e3/c4/b984569ed8ccca0da7e4798632344243d41240908c338fb4c60d26d0af06/grpcio-1.12.0-cp36-cp36m-macosx_10_7_intel.whl (1.9MB)
    100% |████████████████████████████████| 1.9MB 3.5MB/s
Collecting six>=1.5.2 (from grpcio)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, grpcio
Successfully installed grpcio-1.12.0 six-1.11.0
Carbon:tmp.2cTF23AS m$ bin/python
Python 3.6.5 (default, May 22 2018, 18:03:28)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import grpc
>>> grpc.insecure_channel('localhost:13000')
<grpc._channel.Channel object at 0x100cffd30>
>>>

nicolasnoble · 2018-05-23T03:01:07Z

@mehrdada the reporter was talking about anaconda as a distribution method. https://anaconda.org/anaconda/python - I don't think it's fair to use this label if you don't try to reproduce exactly :-)

ssarangi · 2018-05-23T03:33:48Z

Hi, as I mentioned I was able to reproduce this with both Anaconda as well as virtualenv... I haven't tried python via brew but I use Anaconda as the primary distribution. However, I see the same issue whether i use conda or pip to install grpc.

mehrdada · 2018-05-23T03:56:40Z

@nicolasnoble @ssarangi I assume the virtualenv was using "plain" Python; that's why I asked for reporter help with the reproduction? Is the virtualenv using Anaconda as well?
I don't believe we actively test for Anaconda or other variants of Python (I'm not familiar with Anaconda at all). If you can only reproduce with Anaconda, I suggest filing the bug with the Python distribution. They can probably either help fix the issue on their side or at least point us to what the delta is between theirs and the "regular" (e.g. brew) Python install.

ssarangi · 2018-05-23T04:08:24Z

virtualenv was indeed plain python. Also this doesn't look like a distribution bug since dlopen for cgrpc shared library can't find the _deflate method so this seems more like an issue with grpc rather.

mehrdada · 2018-05-23T04:08:50Z

@ssarangi As a potential workaround, can you ask pip to force install grpcio from source and see if it works?

mehrdada · 2018-05-23T04:09:55Z

Where do I get the Python you used for virtualenv to try repro?

ssarangi · 2018-05-23T04:52:19Z

@mehrdada I just used regular pip once I activated the virtualenv. Let me try to build grpcio from source.

ssarangi · 2018-05-23T04:56:27Z

@mehrdada I just tried building and installing from source and still see the same issue. I had uninstalled the pip version and verified that and then built from source.

nicolasnoble · 2018-05-23T05:16:11Z

So I guess it means it's all from anaconda. Meaning that the virtualenv is provided by it too.

mehrdada · 2018-05-23T05:40:58Z

We had a similar internal bug about Anaconda specifically (b/76345760) and supporting Anaconda was deemed Infeasible by @nathanielmanistaatgoogle and closed. But if a Python binary/install fetched from brew or https://www.python.org/ reproduces this behavior, we should certainly look at it.

ssarangi · 2018-05-23T06:03:05Z

I did look at that bug. I was surprised that virtualenv did work though. I will try it out on a non-anaconda distro on osx and report back.

im-sure · 2018-05-24T08:44:04Z

Have you solved it? I have the same problem.

ssarangi · 2018-05-24T16:00:24Z

It did work on brew based python so it could be a difference in python binary. Would have been great to have it work with anaconda but as long as it works with brew as well I can manage.

mehrdada · 2018-05-24T16:04:29Z

@ssarangi If you have the bandwidth, I strongly encourage you to please follow-up on this issue with Anaconda. They will either hopefully resolve this or provide us with the rationale on why this fails and hints on what the best way to resolve this is (ideally with a convincing argument), as none of us working on gRPC Python happen to be Anaconda experts at the moment.

im-sure · 2018-05-25T10:32:57Z

I solved it by using this command：
$ conda install grpcio
because I found that my anaconda hasn’t installed grpc ;)

wandering007 · 2018-07-12T10:00:44Z

Try adding from concurrent import futures before import grpc, works for me.

iZhangZheng · 2018-07-13T12:21:45Z

@im-sure also worked for me, thank you!

maccam912 · 2018-07-13T15:07:31Z

I was importing grpc first, but after moving "from concurrent import futures" above it, it seems to work fine. Thanks for the fix @wandering007

bolerap · 2018-08-08T08:02:42Z

thank you @wandering007 It works

ruanchong · 2018-08-16T12:56:14Z

@wandering007 This is amazing.
I'm running TF Serving MNIST example. Put import grpc after

from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc
import mnist_input_data

resolves this error.

nicolasnoble added lang/Python priority/P1 kind/customer issue platform/macOS labels May 23, 2018

nicolasnoble assigned nathanielmanistaatgoogle May 23, 2018

nathanielmanistaatgoogle assigned mehrdada and unassigned nathanielmanistaatgoogle May 23, 2018

mehrdada added the disposition/requires reporter action label May 23, 2018

nicolasnoble removed the disposition/requires reporter action label May 23, 2018

mehrdada added priority/P2 and removed priority/P1 labels May 23, 2018

ssarangi closed this as completed May 24, 2018

lidalei mentioned this issue Sep 24, 2018

Cannot start server and client in virtual environment #16559

Closed

lock bot locked as resolved and limited conversation to collaborators Nov 14, 2018

lock bot unassigned mehrdada Nov 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Symbol not found: _deflate from grpc 1.12.0 #15510

Symbol not found: _deflate from grpc 1.12.0 #15510

ssarangi commented May 22, 2018

ssarangi commented May 22, 2018 •

edited

Loading

nathanielmanistaatgoogle commented May 23, 2018

mehrdada commented May 23, 2018

nicolasnoble commented May 23, 2018

ssarangi commented May 23, 2018

mehrdada commented May 23, 2018 •

edited

Loading

ssarangi commented May 23, 2018

mehrdada commented May 23, 2018

mehrdada commented May 23, 2018

ssarangi commented May 23, 2018

ssarangi commented May 23, 2018

nicolasnoble commented May 23, 2018

mehrdada commented May 23, 2018 •

edited

Loading

ssarangi commented May 23, 2018

im-sure commented May 24, 2018 •

edited

Loading

ssarangi commented May 24, 2018

mehrdada commented May 24, 2018

im-sure commented May 25, 2018

wandering007 commented Jul 12, 2018 •

edited

Loading

iZhangZheng commented Jul 13, 2018

maccam912 commented Jul 13, 2018

bolerap commented Aug 8, 2018

ruanchong commented Aug 16, 2018

Symbol not found: _deflate from grpc 1.12.0 #15510

Symbol not found: _deflate from grpc 1.12.0 #15510

Comments

ssarangi commented May 22, 2018

What version of gRPC and what language are you using?

What operating system (Linux, Windows, …) and version?

What runtime / compiler are you using (e.g. python version or version of gcc)

What did you do?

What did you expect to see?

What did you see instead?

Anything else we should know about your project / environment?

ssarangi commented May 22, 2018 • edited Loading

nathanielmanistaatgoogle commented May 23, 2018

mehrdada commented May 23, 2018

nicolasnoble commented May 23, 2018

ssarangi commented May 23, 2018

mehrdada commented May 23, 2018 • edited Loading

ssarangi commented May 23, 2018

mehrdada commented May 23, 2018

mehrdada commented May 23, 2018

ssarangi commented May 23, 2018

ssarangi commented May 23, 2018

nicolasnoble commented May 23, 2018

mehrdada commented May 23, 2018 • edited Loading

ssarangi commented May 23, 2018

im-sure commented May 24, 2018 • edited Loading

ssarangi commented May 24, 2018

mehrdada commented May 24, 2018

im-sure commented May 25, 2018

wandering007 commented Jul 12, 2018 • edited Loading

iZhangZheng commented Jul 13, 2018

maccam912 commented Jul 13, 2018

bolerap commented Aug 8, 2018

ruanchong commented Aug 16, 2018

ssarangi commented May 22, 2018 •

edited

Loading

mehrdada commented May 23, 2018 •

edited

Loading

mehrdada commented May 23, 2018 •

edited

Loading

im-sure commented May 24, 2018 •

edited

Loading

wandering007 commented Jul 12, 2018 •

edited

Loading