Jobs are en-queued but it's not processing #1218
Comments
|
same problem |
|
same problem. any help appreciated |
|
Also having the same issue. Restarting the windows service (which runs the Hangfire service) fixes the issue for a while, even a couple days. Then all of a sudden jobs are en-queued and not picked up for processing. Dashboard (on separate web app) says the Hangfire server is live and connected and good heartbeat, etc. |
|
Please use the STDump utility to obtain stack traces of your threads, when everything is blocked, and post them here. |
|
I think its relates to azure servicebus. After job is processed if you re queue using dashboard it stays stuck |
|
Sure thing, thank you for the quick response. I currently have queued jobs that are not getting picked up. I also notice that now my server heartbeat is 7 hours. Here's the output of running stdump on the server process: |
|
@sethsteenken, thanks for the stack trace. I don't see any worker threads, and without the logs it's hard to say what happened. This may be caused by user code that throws the |
|
Sergey, I currently have the same problem, although Hangfire user to work succressfully for 2 years and without any problems. |
|
In our case the problem was that we updated Hangfire.SqlServer.dll to version 1.6.21 from version 1.6.6.0, but Hangfire.Core.dll remained of version 1.6.6.0. This caused the jobs being queued but not processed, and the Hangfire portal showed no error messages whatsoever, which made quite difficult to understand the cause of this problem. |
|
Is there any update on this issue? |
|
@NeenuSunil, please provide any logs with exception stack traces, dashboard screenshots, configuration logic and output of the STDump to diagnose what happens. |
|
We are having the same issue with Hangfire. The following versions are installed: Hangfire.Core 1.7.6 Would you like me try and collect any additional logs by some means? We had almost 3200 jobs enqueued overnight. |
|
We don't have any issues now. We resolved the problem by |
|
@NeenuSunil Can you point me the documentation which says there is a bug with hangfire if prefix names include hypen. |
|
Submitted what is hopefully a fix for issues with dashes in schema name: #1531 |
|
Any update on this issue? We are having the same issue with Hangfire. The following versions are installed: The problem is only when I deploy it on azure app service linux, not on my local machine |
|
I'm having this issue, some jobs are not processing and have days in the queue, and these jobs last at least 15 seconds to complete.
|
|
Which Hangfire Version are you using?
Are you redis Or sql??
If you are using redis, is it cluster mode enabled??
…On Fri, 29 May 2020, 22:22 George Universe, ***@***.***> wrote:
I'm having this issue, some jobs are not processing and have days in the
queue, and these jobs last at least 15 seconds to complete.
[image: image]
<https://user-images.githubusercontent.com/3822009/83284813-7b117a80-a19a-11ea-92f5-b4ab35b5fe81.png>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1218 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AES6XSMQEWVO7BWFFF63GNLRT7R6JANCNFSM4FITZ2UQ>
.
|
|
@kunaldhande we are having the same issues. We used to be on Windows App Service and no issues. We recently migrated to a linux VM in azure and since then sporadically it freezes, even after a job successfully finished, the queue does not continue. We have to restart the service.. Version: Using PostgreSQL, but before Memory and both have the same issues, I wouldn't say its DB related |
Currently using SQL Server on Windows Server 2019. |
|
Same error, using Hangfire version 1.7.11 on Linux, .NET Core 2.2.403. It happens randomly. |
|
We got the same problem using Hangfire 1.7.11 and Hangfire Pro 2.2.2 on Ubuntu 1804 with .NET Core 3.1.301. Storage is Azure SQL (S1 Tier) Most jobs are stuck in the enqueued state or fail to transition to successful state upon work completion. Any support highly is appreciated |
|
The only workaround for it that I've found is to completely restart the dotnet process every 8 hours or so, which is a real PITA. I quite like Hangfire and don't want to replace it, but having to restart the whole stack just for that won't be feasible in the long-term. I wonder if anyone else has found a better workaround, or even a fix? |
|
It's almost impossible to find out what's happened without additional information, please see the following link and describe everything, including "ProTips" section. There are a lot of reasons for this to happen, including different deadlocks in background job methods themselves. https://github.com/HangfireIO/Hangfire/blob/master/CONTRIBUTING.md |
|
I had saved the logs back then, when I knew for sure that the process had stalled in the same hour. The UI was still working, the jobs were just stuck in the enqueued state. There's only one line that's odd there, about the "DelayedJobScheduler recovered from the Faulted state". I'll try to post another set of logs as soon as the problem reappears, and maybe get the stack dump too. Hangfire v1.7.11 |
|
I see you are using MySql as a job storage which is a community-based extension. Also try to include the DEBUG log level - I see there's "Execution DelayedJobScheduler recovered from the Faulted state" message, and there also should be prior messages with DEBUG level with exact exception. |
|
After some digging in the official doc, I saw there were multiple MySql connectors. This one in particular caught my eye because it's apparently fixing some deadlocks. https://github.com/MiloszKrajewski/Hangfire.Storage.MySql I'll try it whenever I'll have time to. I think it's worth a shot. Thanks and sorry in advance for wasting your time if that was the cause, @odinserj ! |
|
Changing the connector to this one https://github.com/MiloszKrajewski/Hangfire.Storage.MySql seems to have fixed my issue. Haven't had to restart the server in a week. |
|
I am also having same problem, Jobs are en-queued but it's not processing Using following version
Any help appreciated |
|
It seems like it is fetching the job and enqueuing the job but Why then the job is moved to Failed queue if processing of job is failed. If its production environment and clients are waiting for enqueued job, you can try restart server and it might start processing the jobs but issue still has to be fixed. |
@Gheri Thanks for the replay, I already tried this and even after restarting server the same issue occurs. |
|
I am also having the same problem, I already change to hangfire.storage.mysql. The problem still exist. |
|
It's still happening for us, with Hangfire version 1.7.25 using redis storage with Hangfire Pro 2.8.10. Jobs got enqueued but never picked up to be processed. Here's the output of running stdump on the server process:
EIDT: it's strange, if I click on the requeue button, the job will be processed, but all future enqueue jobs are pending again in this state. |
|
Any support would be highly appreciated @odinserj |
|
Hm, looks very strange, especially when everything is restored after Requeue button is pushed. I need to understand what happens – it's either publish event is not recognised or queue doesn't contain the background job identifier. Is it possible to run the following command (where If it returns an empty list then it's either key name is misspelled or background job identifier is not in the queue. If not, then something strange happens with event publication. Please also send me your configuration code related to Hangfire and the output of the |
|
Thanks for your reply! Here are for example two enqueued jobs in the dashboard:
and I can see them in the Redis storage with the proper *queue key:
and here is an example, that the jobs get processed as soon as I click on the Requeue button (at this point both pending jobs were processed, although I clicked Requeue on just one of the pending one):
Will send you next, during the day, the configuration code related to Hangfire. Please share any input you have so far. |
|
Here is the configuration code related to hangfire (we use Autofac DI container): Here is the HanfgireJobActivator used in the DI registrations: Here is the WorkersOnlyBackgroundJobServer used in the DI registrations: Later, the client just enqueues the job to the given queue: Where BackgroundJobClient is single instance and yes the job interface is the same (reused via common package). We use single Redis instance (no cluster). Used packages in the processing worker: |
|
@odinserj safe yourself some time reading all this :) I checked the client that enqueues the jobs now and figured it out where the problem is:
just updated to
and the issue is gone :) Thanks for your time and the great package! And sorry for the confusion. |
|
Oh, |
|
Just in case some still facing this, I had a similar issue but my problem was that I didn't defined my queues names in the startup. Adding this solved the problem
|
|
I'm closing and locking this issue now, because almost any kind of problem in background processing will lead to the symptom "Jobs are enqueued but not processing", and more specified details required. |
Hi Team,
I've the job which is reading some data from sql db and adding that in console.
Seems like heartbeat is fine and jobs are enqueued but the processing part is not working.
Have attached screen shots of server and jobs.
Would you please let me know what am I missing.
Best Regards,
Damodar
The text was updated successfully, but these errors were encountered: