Improve StateMachineQueue

[theoilie]

Description

• Add jobId to decision tree logs and log each step separately so we can see where it fails
• Add logs for Bull queue events
• Increase Bull lockDuration (time after which job is marked as stalled)
• Make consumers return an empty object (docs do this and this comment says it helped)
• Update Bull to the latest version. Possibly relevant issues that this resolves:
  • Missing lock for job failed when closing queue and job takes more than 30 seconds to complete [BUG] OptimalBits/bull#1415
  • completed and failed jobs should be removed from the stalled set OptimalBits/bull#1600
  • Errors thrown inside of queue "completed" handler result in "Missing lock" errors OptimalBits/bull#2213

Tests

Searched local logs to make sure Bull events are being logged, jobIds are being logged, and processStateMachineOperation is running as expected.

How will this change be monitored? Are there sufficient logs?

Search logs for "processStateMachineOperation() Decision Tree" and see if there are any gaps in any node. Debug gaps by looking for stateMachineQueue Job [Error|Waiting|Active|Lock Extension Failed|Completed|Stalled|Failed] logs.

[SidSethi]

lgtm!
if you've confirmed this works locally, prob good to take out to single staging node from branch before merging, esp with the bull major version bump. that one is scary

side note - i noticed you didn't bring over the Queue#obliterate() step. totally fine, just flagging that may be another lever we can pull
see docs for info - https://www.notion.so/audiusproject/Bull-Queue-6bef860a5e784d19a099df8c32003fdf#b0bc0f63fb754a33a08607a22e628e21

[theoilie]

lgtm! if you've confirmed this works locally, prob good to take out to single staging node from branch before merging, esp with the bull major version bump. that one is scary

side note - i noticed you didn't bring over the Queue#obliterate() step. totally fine, just flagging that may be another lever we can pull see docs for info - https://www.notion.so/audiusproject/Bull-Queue-6bef860a5e784d19a099df8c32003fdf#b0bc0f63fb754a33a08607a22e628e21

great! I'll see how everything looks on a single node.

I left out the obliterate because it seemed like something that might do better in an experiment instead of on all nodes. I also figured if we already removeOnComplete and removeOnFail then I'm not sure how much it would help to obliterate once when the CN initializes

[theoilie]

I noticed during one test on staging that the delayed job (with id 2217) ran, but the repeatable job never ran. this could be a state issue from restarting a few times, so I added back Queue#obliterate to see if that helps. I looked into it more and it doesn't seem like it'll hurt anything (also Vicky said it helped with a Bull issue she had before)

[theoilie]

soaking these changes on staging over the weekend didn't fix the Bull issues, but the increased logging helped and there were no new regressions. re-requesting a review

[SidSethi]

with major version bump, lets both go through logs for other CN queues and ensure proper function

[SidSethi]

lgtm pending version bump confirmation