-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gunicorn 19.x causing problems with django-1.7.1 and gevent #927
Comments
Just a question first: does taking out the preload option help? |
I think we import the django wsgi application in the master process, which at least on django 1.7, initializes all of the django models on import. |
Hello @tilgovi, thank you for your time. I cant confirm that using Do you have any idea what gunicorn cloud be doing differently about this in versions I have now a second problem. I use Raven[1] in my django app. And accorging to the Raven docs, when using a WSGU middleware[1] it is possible to use Raven as a WSGI object (wrapping the original django WSGI app), and when using gunicorn (as an external command, not the Raven is For now I will get back to gunicorn [1] http://raven.readthedocs.org/en/latest/config/django.html#wsgi-middleware |
@daltonmatos the way databases are setup in django 1.7.1 has probably changed are apparently not threadsafe. I am not sure what to do at that point. |
One more thing, just found out today, Had to switch back to 18 to make it work. FYI. |
This seems to be the same issue as #879, which was closed but not fixed. It makes gunicorn 19.x. unusable with Django. Perhaps only for the gevent worker though? I haven't tested for other worker types. |
@tilgovi i wonder if it's not due to the change done in the environ. in 19.x we set https://github.com/benoitc/gunicorn/blob/master/gunicorn/workers/ggevent.py#L99 This change may be handledd differently in django. And a greenlet is not really a thread. Thoughts? At least maybe we could try to revert it and check if it solves the issue. Do we have a test or a reproducible way for this issue? |
Oh, I bet that is the issue. Good memory, @benoitc. I still don't know why yet. As I said at the time to justify the change, I can't imagine that it would make anything break because it would mean that the framework would only take more precautions about sharing data. Evidently, I may be wrong. |
rever change introduced in R19 to solve issues with django. fix #927
I pushed the branch |
I'm still able to reproduce this error using fix/927 with django 1.8.1 (and 1.7.1) and gevent 1.0.1 |
That's reassuring to me, even if we still don't know the cause. |
So I tried using the example application and didn't reproduce the issue running the following command line:
Can anyone provides me a reproducible example? |
I can still reproduce this issue invoking gunicorn like so:
Is there any relevant details you need? |
@brockhaywood should be Looking at the trace above, it could also be the monkey patching. Maybe threads shouldn't be monkey patched. Can you try the following patch?
|
Using the above patch with the following wsgi no longer produces the specified error:
I'm now seeing a different and new error but have no idea if it's related:
This is occurring, 1in 10/15 requests is doing a count on a model. I suspect this is an error in my application which is now uncovered. |
@brockhaywood cool! btw ahat's your version of gevent? For your latest error it dies not seem related to gunicorn indeed. |
Yah, good news! On version 1.0.1 of gevent. |
@tilgovi so I wonder if we shouldn't patch gunicorn removing the thread monkey patching. Maybe we should also set |
@benoitc I'm able to reproduce the new error that I referenced above with a very simple Django application that has just one view and does a simple model query and count if I run multiple concurrent requests. Do you have any suggestions as to which component is the most likely culprit? Is this likely a gevent problem or psycogreen? |
More likely psycogreen. Are you using such hook to setup it?
|
Oh, I'm actually using
I'll try your suggestion. |
Your code is more accurate if I read the code of psycogreen: |
Ok, I'll try opening a ticket with psycogreen |
can you try to use the hook |
Still occurs for me when patching psycogreen in post_worker_init. I've submitted a ticket to psycogreen as well: |
@brockhaywood OK :/ Also found this link: https://bitbucket.org/dvarrazzo/psycogreen-hg/issue/1/databaseerror-execute-used-with . I am not sure if it could help you though. |
If it works to patch early, maybe that should be considered as a change. |
I can't remember if that was ever the case and if that caused problems, but it's always tricky with monkey patching. |
Our own production gunicorn application requires preloading and so also requires monkey-patching in the master (we use Pyramid's/paste's I'm a current gevent maintainer. I'm happy to look at any issues in gevent required to make this scenario work better. We're expecting to drop 1.1 soon-ish and it would be great if this was a supported option. |
I tried applying the same monkey patch logic at the top of our wsgi implementation, and we started seeing the I'll followup if I have a eureka or get lucky with regards to the app code, but so far for us, patching earlier in the gunicorn master code is the only fix that solves all our issues. |
@jamadden thanks so much. @jzupko it's usually important that the patching happens before any imports that might be affected. That's why I suggested to do it in a server hook, like by putting a |
@benoitc do we have any mechanism for worker-specific config? |
I'm adding this to the release 20 milestone and suggesting that we consider monkey patching the master, maybe, if it can be done without re-introducing problems like #478. |
@tilgovi you have the |
Does anyone have a temporary fix for this until 20.0 is out? Or is the only solution to choose between disabling |
@fletom temporary fix for? Did you try the latest master? Did it works for you? |
@benoitc I mean a fix for the "DatabaseWrapper objects" error. Current gunicorn master doesn't work at all for me actually. On regular 19.6.0 I get this (normal):
And on latest master with the 100% exact same configuration I get this, and nothing listening on any port as far as I can tell:
Seems bugged? |
@fletom how do you launch gunicorn? |
@benoitc In this case, it's My
|
I am stumbling upon this issue with Putting these lines on top of the gunicorn_conf.py file seems to solve the issue for me: import gevent.monkey
gevent.monkey.patch_thread() Update:
Because of them django initiated the connection before gunicorn's monkey-patching. |
@gukoff proposed fix didn't work for me, I still get the same error 😞 |
I'm going to close this and open a discussion with the mailing list label on the R20 milestone for monkey-patching the arbiter in gevent. If you are having problems related to anything in this discussion and you think it might be a bug in Gunicorn, please open an issue. You can mention this one by number, or link to comments, if it helps add context, but at this point this is an old ticket with conversations in parallel about several versions of Gunicorn and different symptoms. |
I would like to note that as of 2024, this problem persists. I encountered a similar issue at work, and the problem was resolved by removing the |
Hello,
I'm seeing this error with gunicorn
19.x
(tested with19.0
and19.1.1
) using thegevent
worker. I'm usingdjango-1.7.1
.this is the exception:
Here is my
gunicorn.conf.py
:The error occurs every time, just accessing any URL of the django app (that depends on any model, of course) triggers this error. Any ideas?
Since changing to
gunicorn-18.0
solves the problem, I'm assuming thatgunicorn-19.x
is doing something different.If you need any additional information, let me know and I will post it here.
For now I'm using
18.0
.Thanks a lot.
The text was updated successfully, but these errors were encountered: