Segfault Connecting After Importing Asana (2.7.X)

[theaeolianmachine]

Hey psycopg2 maintainers,

Recently we encountered an issue where by running the following program, you get a segfault when psycopg2 tries to connect to a database:

import asana
import psycopg2
config = {'database': u'my_database', 'host': u'127.0.0.1', 'password': 'my_password', 'port': 2201, 'user': u'root'}
psycopg2.connect(**config)

asana is a 3rd-party library, in particular it's this one: Github | PyPI

There are two important characteristics of the repro:

• It only repros on psycopg2 2.7.* (in particular, tested 2.7.3 and 2.7); downgrading to 2.6.2 no longer creates the issue.
• It's import order dependent - if you import psycopg2 first, the issue no longer occurs, such as:

import psycopg2  # Note psycopg2 is imported first, and this issue no longer occurs
import asana
config = {'database': u'my_database', 'host': u'127.0.0.1', 'password': 'my_password', 'port': 2201, 'user': u'root'}
psycopg2.connect(**config)

The backtrace we got looks like it's in making an SSL connection and using libcrypto. Of note is that asana is all in python and doesn't explicitly do any crypto work, but it uses requests (requests >=2.14.2, == 2.14.*) and requests_oauthlib (requests_oauthlib >= 0.6.1, == 0.6.*) which to my knowledge should also use lower level libraries when making SSL connections.

The backtrace itself:

(gdb) run test.py
Starting program: /usr/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff634abe2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(gdb) backtrace
#0  0x00007ffff634abe2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#1  0x00007ffff634ad69 in ENGINE_finish () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#2  0x00007ffff635df05 in EVP_DigestInit_ex () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#3  0x00007ffff3cae8a4 in ssl23_connect () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/./libssl-a65b360f.so.0.9.8e
#4  0x00007ffff3f13a9c in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#5  0x00007ffff3f13fbd in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#6  0x00007ffff3f0175e in PQconnectPoll () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#7  0x00007ffff3f020ce in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#8  0x00007ffff3f029bf in PQconnectdb () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#9  0x00007ffff413c241 in _conn_sync_connect (self=0x7ffff4368188) at psycopg/connection_int.c:716
#10 conn_connect (self=self@entry=0x7ffff4368188, async=<optimized out>) at psycopg/connection_int.c:812
#11 0x00007ffff413cf84 in connection_setup (async=<optimized out>, dsn=<optimized out>, self=0x7ffff4368188) at psycopg/connection_type.c:1277
#12 connection_init (obj=0x7ffff4368188, args=<optimized out>, kwds=<optimized out>) at psycopg/connection_type.c:1362
#13 0x000000000055f6db in ?? ()
#14 0x00000000004c8a7d in ?? ()
#15 0x00000000005ab155 in _PyObject_CallFunction_SizeT ()
#16 0x00007ffff41360b8 in psyco_connect (self=<optimized out>, args=<optimized out>, keywds=<optimized out>) at psycopg/psycopgmodule.c:111
#17 0x000000000052714b in PyEval_EvalFrameEx ()
#18 0x0000000000568b3a in ?? ()
#19 0x0000000000525cb7 in PyEval_EvalFrameEx ()
#20 0x0000000000567d14 in ?? ()
#21 0x0000000000465bf4 in PyRun_FileExFlags ()
#22 0x000000000046612d in PyRun_SimpleFileExFlags ()
#23 0x0000000000466d92 in Py_Main ()
#24 0x00007ffff7814f45 in __libc_start_main (main=0x466e50 <main>, argc=2, argv=0x7fffffffdee8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
    stack_end=0x7fffffffded8) at libc-start.c:287
#25 0x0000000000577c2e in _start ()

Can you all see if you can reproduce this issue? We unfortunately had to downgrade psycopg2 as a result, and we'd love to have some of the great 2.7 features back.

Thank you!

[theaeolianmachine]

One more note - this is connecting to Amazon Redshift. I actually just had trouble reproducing this in Postgres proper :(.

[fogzot]

I've seen this happen on other platforms/languages too. Two lbraries, libpq and whatever used by asana are linked against different versions of libcrypto. A solution is to make sure there is only 1 version of libcrypto onbthe system: you may need to build from source to make sure all packages use it.

[dvarrazzo]

The problem is caused by the psycopg2 wheel package and will not present compiling psycopg from source. You can find details here: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi

[theaeolianmachine]

Thank you for the info!

[dvarrazzo]

@theaeolianmachine I'm trying to fix this problem but I have problems reproducing it.

Would you be able to provide a Dockerfile recreate it? Thank you very much.

[theaeolianmachine]

Hey @dvarrazzo - I appreciate you looking into it further, but we did fix this problem by installing via source (appreciate the context). Given the lack of deep knowledge I personally have around our systems and what else is installed on them, it might take a bit. - just let me know if you'd like to keep hunting down this issue.