-
-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
_PushStatus stays stuck as 'running' #4315
Comments
Have there been issues dequeuing? For handling a failure it makes sense that we would just indicate it as failed like it is. As for relying on |
That would be utterly inefficient, having to do n additional writes on the server befor sending the push’s. The count is only used to know how many batches we need to send in a distributed manner (keep in mind we have 20+ parse-server instances with as many push workers). Count also used as décrémented after the worker completed the job, main issue is that count in unlikely to reach 0 and PushStatus stays running. |
We haven’t seen any issues dequeuing, and it doesn’t matter, we don’t guarantee delivery yet, and this should be at the queue level that it should be solved, ie: the message should be put back in the working queue if processing didn’t start |
It would be super slow, but it wouldn't miss! But that would be too slow most likely. If I'm understanding this right the query used to get the original count is later used to continue to provide I think I need to take a look at this code to see if something more obvious comes to mind, I haven't looked it over really. The one thing that I was thinking of, for preventing hanging pushes when installations are removed, would be to determine if the, although the count is not 0, the Installations available to be sent is 0. Basically pick either or, but this is just speculation. |
In any case, counts are unreliable, and documented as such on mongodb. So, for all that matters, it’s useful to estimate the count but not more. Also, for the sake of consistency, if an object enters the scope of a push, it should be included, even if there’s some latency. Otherwise, that’s a constraint on updatedAt we should add, which is unacceptable also, as may conflict with user defined constraints. |
Yeah :/, modifying the query could mess with a user defined constraint. I'm assuming you're talking about sharded cluster counts, considering that it would make sense we can't really trust it. In lieu of anything from my end could you elaborate on some of those fixes you proposed initially? |
Closing as done! |
When sending large number pushes, the push pipeline originally evaluates the amount of push to send by running a count on the _Installations table. This count is then stored in the
_PushStatus
object and decremented each times the PushWorker completes the push send.Now, it creates multiple issues.
Counts are knowingly unreliable with mongodb. So we should not rely on those.
One fix would be to
Thoughts? @montymxb @acinader
The text was updated successfully, but these errors were encountered: