ScheduledReporter queues up runnables when gauge is slow

[ChadiEM]

Problem Statement

The current implementation of ScheduledReporter relies on a ScheduledExecutorService scheduled at fixed rate:

this.scheduledFuture = executor.scheduleAtFixedRate(runnable, initialDelay, period, unit);

Every period, a thread will report, except when one of the Gauge implementations (getValue()) is slower than that period (for example, network latency). In that case, threads will start to accumulate. When the latency is resolved, there will be a spike of report requests.

Reproducible Scenario

To reproduce, have:

• a gauge that sleeps 10 seconds the first three times, then 0 the upcoming times before returning a value
• a ScheduledReporter with a delay of 5 seconds that prints all gauges.

You will get reports at 0, 10, 20, 30, 30, 30, 30, 35, and 40.

Possible solution

I suggest to replace executor.scheduleAtFixedRate with executor.scheduleWithFixedDelay, so that the next call to report is separated by period with respect to the previous call, instead of having an absolute period.

[joschi]

@ChadiEM Thanks for reporting this!

@dropwizard/metrics @dropwizard/committers Any opinions or reservations on changing the reporting strategy of ScheduledReporter as described by @ChadiEM?

[the-thing]

I think this is a good idea. If they gauge is slow, one should use CachedGauge, but sometimes this can happen unexpectedly on production. Those reporting burst are undesired.

[richwklein]

Should this issue be resolved by #1590?

[ChadiEM]

Yes! Thank you.