feat: new estimator variance check

[jakmeier]

The condition to check for high variance of measurements so far has been
that no single sample is further than 15% from the mean away. We often
missed that when running with 5 iterations and one warm-up iteration.
The counter-intuitive side of this check is that is becomes less likely
that variance is considered low enough, if more iterations are added.

The new condition computes actual variance and standard deviation.
Now we expect that standard deviation is lower than 10%.

Also add an outer loop to block overhead computation. Usually there
is a loop + a number of actions inside a block. But this estimation
has empty blocks and therefore each measurement so far was
just a single apply block call. Grouping 10 empty blocks together
for each measurement and adding an outer loop for --iters reduces
the overall variance.

[jakmeier]

Main motivation behind this: CE results have a lot of UNCERTAIN HIGH-VARIANCE results.

For time measurements, the block overhead in particular was marked uncertain. And now that it gets subtracted from many other measurements, this infects all those other estimations that then also get marked uncertain.

[matklad]

This is reasonable!

I wonder though, should we add an extra check for outliers (eg, check that no measurement is 2x larger than all other ones, or something)?

Sometimes we have a case where, eg, a first iteration is very slow (due to some caching), and it'd be useful to catch those specifically.

[matklad]

Yeah, I recall it being tricky to deal with the case where there are no gas costs, so you don't know the metric. If we rely on this being a newtral element(eg, even if the actual metric is icont), then perhaps let's add zero-arg (GasCost::zero())?

Actually, thinking about this more, the reason why GasCost tracks metric is historical:

originally, we measured only time or only icount, and had to thread metric throughout. Since then, we've switched to multidimentional metric (ie, we count IO costs separately).

So perhaps a bigger refactor is due, where we say that GasCost is a product of measurements fro several metrics, where each metric is an Option (so, each specific gas cost knows what metrics did it actually track). Then zero would naturlaly return all components as Some(0) (ie, zero includes all metrics, measured as zero) .

[jakmeier]

Yeah such a refactor makes sense, I'll do it after merging this.

[matklad]

if samples.len() <= 1 { return f64::Inf }
/ (samples.len() - 1)

[jakmeier]

I guessed you meant to say two things here

1. the method should be robust to all type-correct inputs, such as empty samples (=> I changed it to return true in such case.)
2. We should use sample standard deviation instead, i.e. divide by n-1 instead of n. (=> done)

If I misinterpreted your comment, please let me know :)

[jakmeier]

I wonder though, should we add an extra check for outliers (eg, check that no measurement is 2x larger than all other ones, or something)?

Sometimes we have a case where, eg, a first iteration is very slow (due to some caching), and it'd be useful to catch those specifically.

@matklad If I understand your concern correctly, then I believe variance is already good enough for that because it is quadratically sensitive to large deviations. Let me maybe put some numbers on the table as an example. CE runs with 1 warm-up iteration and 5 measured iterations. If the non-warm-up results were [200,100,100,100,100] then mean = 120, variance = 2000, and standard deviation = 44.72. So the percentage of stddev / mean = 37.27%, far above the 10% the new check is set at.

Or do you suggest to add an extra check, in the sense that we would mark it not as "HIGH-VARIANCE" in such a case but instead "SINGLE-OUTLIER"?

[matklad]

I believe variance is already good enough for that because it is quadratically sensitive to large deviations

Yeah, this makes sense, I guess just variance is enough then!