Finality Voting Threshold metrics for alerting

[bhazzard]

Observability and alerting will be important to allow production network operators to remediate when voting falls below a threshold that will allow finality to advance.

To this end, Node Operators need a way to configure alerts that fire when finality voting falls below a configurable threshold.

Suggested Metrics:

• number votes that should have been included in the block (using knowledge of schedule changes)
• number of votes that were actually included in the block

This issue is related to #227

[bhazzard]

Potentially also: per finalizer, total delta time between block time and vote time.

This would give a picture of how long it takes for each finalizer to vote after a block is produced.

[bhazzard]

Decision needed: should we add this to Prometheus or Debug logs, or both?

[arhag]

The logs already provide the above information and more.

Moreover, we now have a new endpoint (see #453) that provides useful information about the last votes by each finalizer which can more easily be used to build alerting tools to alert if some finalizers are failing to participate.

So I think we can close this issue.