Failed heartbeats will longer raise exceptions

[sbonebrake]

Addresses #559
A test can be added if this is agreed to be the correct approach.

[michaelklishin]

@sbonebrake I'm a bit concerned that we might leave HeartbeatSender threads around between recoveries. Will need to do some testing/profiling. Otherwise I think this solution is OK. Thank you!

[sbonebrake]

I think it's fine to have the HeartbeatSender thread still running during recovery as long as it fails silently. I think it's probably a good idea to add an open? check to transport#write_without_timeout.

def write_without_timeout(data, raise_exceptions = false)
  begin
    if open?
      @writes_mutex.synchronize { @socket.write(data) }
      @socket.flush
     end
    rescue SystemCallError, Bunny::ConnectionError, IOError => e
      close
    end
  end
end

Thoughts?

[michaelklishin]

I'd prefer stopping heartbeat senders since recovery won't be happening more than once every few seconds (worst case), so instantiating a new one and starting a new thread would not be a problem.

[sbonebrake]

I just realized that if the heartbeat threads stopped on failure then they would rely on automatically_recover, but that option may be false. Eg { threaded: false, automatically_recover: false }
Updated my PR with a toxiproxy test case

[michaelklishin]

Thank you!