Send an alert after 3 consecutive errors

[tomlebl]

If there are 3 consecutive errors IconNMR run stops. In that case, it would be useful to sent an e-mail alert to admins.

[tomlebl]

There could be more complicated logic for error handling that could make the system more resilient by avoiding unnecessary stops of IconNMR after 3 consecutive errors .

Number of consecutive error can be set by admin
If defined number of last entries in history table has "Error" status, NOMAD submits an experiment with priority for a standard sample that permanently sits in the last holder of sample changer. If tracker detects that the standard experiments gets error then NOMAD shuts the queue and sends error message to admins.
If there would be other samples waiting in the queue, we will get race condition here. Thus, the standard sample can go in just after the errors or after next experiment
That should not be a big problem as we can simply disable the feature in IconNMR or set the value of consecutive error to +2. In that case, if standard sample gets through successfully IconNMR error sequence gets interrupted and machine keeps running.