Emit metrics for record_progress endpoint

[Mark-Simulacrum]

Previously we were only tracking the worker time, not the endpoint. We see that there is a direct correlation with the throughput of a job and the worker time. This seems wrong to me, because as long as the worker is keeping up with the input rate, the throughput shouldn't be affected.

Note that we believe that the worker should not affect the HTTP endpoint at all - we connect these with a bounded queue and pushing into the queue is done with try_send, which shouldn't block (https://docs.rs/crossbeam-channel/latest/crossbeam_channel/struct.Sender.html#method.try_send) and returns an error if the queue is full. We already emit a metric if the queue is full, and that's not happening here.

The hope is that the extra metric here gives us some clue for what the problem is.

Metric graphs:

[Mark-Simulacrum]

@bors r+

[bors]

📌 Commit bac1249 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

[bors]

⌛ Testing commit bac1249 with merge 4d35849...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 4d35849 to master...