Add options to purge big workers

[TyeMcQueen]

Described in Issue #3251

[pajod]

looks like unrelated whitespace changes

[pajod]

Does not appear to match documentation: 0.95+0.10 = 1.05 > 100%. But the general idea of averaging out late firing timers seems much more reasonable, so I'd change documentation to clarify that its not the pause between invocations but the average interval.

[sylt]

I wonder if it would make sense to put all this logic into prune_worker? I mean, something like:

    def prune_worker(self):
        """\
        Kill the worker with highest prune score
        """
        if self.cfg.prune_seconds <= 0:
            return

        if self.next_prune is None:
            self.next_prune = time.monotonic() + self.cfg.prune_seconds

        if time.monotonic() < self.next_prune:
            return
        
        maxi = self.cfg.prune_floor
        victim = 0
        for pid in list(self.WORKERS.keys()):
            score = self.cfg.prune_function(pid)
            if maxi < score:
                maxi = score
                victim = pid
        if victim != 0:
            self.log.info(f"Pruning worker (pid: {victim}) with score {score}")
            self.kill_worker(victim, signal.SIGTERM)

        self.next_prune += self.cfg.prune_seconds * random.uniform(0.95, 1.05)

... and then call self.prune_worker() just after self.maybe_promote_master()? This has the benefit of keeping all worker prune logic at the same place.

I had two other minor suggestions as well (incorporated in the proposed method above):

• Would random.uniform(0.95, 1.05) work instead of 0.95 + 0.10 * random.random()? I recon it's probably the same, but the former is a little bit shorter.
• Since we're not interested in the workers themselves but only their PIDs, perhaps calling self.WORKERS.keys() would suffice?