Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When restarting gearmand it will block redis if using it as the queue backend #59

Open
nickpeirson opened this issue Dec 7, 2016 · 2 comments

Comments

@nickpeirson
Copy link

When replaying the jobs back into gearmand from redis it uses the KEYS command, which is a blocking operation in redis that the redis documentation recommend not be used in production:

Don't use KEYS in your regular application code.

If gearmand is hooked up to it's own redis instance this may not be an issue, but if other parts of your application share redis this is likely to be painful if gearmand restarts.

The quick 'fix' to this is to move from using KEYS to using SCAN, however this still has to scan over all keys in the redis instance which will be potentially slow if there are other keys being stored in redis from other applications.

A more complete fix is to track all of the keys added in a set, e.g. when calling SET key data, also do SADD gearman_jobs key, and the complementary DEL and SREM. This then allows for jobs to be reloaded using SMEMBERS which only looks through relevant keys contained within the set, instead of all keys.

@nickpeirson
Copy link
Author

This could be further expanded to have a set for each function_name and then a set for all function_names. This would add an extra lookup, e.g. SMEMBERS gearman_functions then looping through the results to do SMEMBERS function_name, but would make inspecting jobs in redis more user friendly.

@SpamapS
Copy link
Member

SpamapS commented Dec 7, 2016

See my other comments on your other redis queue suggestions. This one is indeed something that would improve the redis plugin. If that's something you still want to do, patches will be welcomed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants