Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not join @thread if that is the current Thread #590

Merged
merged 1 commit into from
Feb 5, 2020
Merged

Do not join @thread if that is the current Thread #590

merged 1 commit into from
Feb 5, 2020

Conversation

fuegas
Copy link
Contributor

@fuegas fuegas commented Feb 4, 2020

If you're running multiple Bunny connections in a pooled manner, for example using the gem connection_pool, the threading model can be altered by the pooling gem to disable handling of exceptions.

For example, the gem connection_pool disables the handling when it creates an instance, and enables the handling when you're using a connection:

  def with(options = {})
    Thread.handle_interrupt(Exception => :never) do
      conn = checkout(options)
      begin
        Thread.handle_interrupt(Exception => :immediate) do
          yield conn
        end
      ensure
        checkin
      end
    end
  end

When you've setup a connection and released it to the pool, it slumbers in a Thread of which the handling of exceptions is disabled. If you receive a (forced) disconnect or a NetworkError occurs, it triggers the recovery (when enabled) of Bunny. This recovery relies on a ShutdownSignal to be raised on the reader_loop thread which needs to be caught by the clean_up_on_shutdown in Session. Because the exception is not handled anymore, the code in maybe_shutdown_reader_loop in Session tries to join the reader_loop thread to force a re-raise of a thrown unhandled exception. That join is done on the reader_loop thread which is the same thread that started the recovery so we'll receive a ThreadError telling us "Target thread must not be current thread (ThreadError)".

To prevent the above from happening we can check if the thread we're trying to join is the same thread that is currently running and not join if it is the same thread.

This PR fixes #589.

If you're running multiple Bunny connections in a pooled manner, for example using the gem `connection_pool`, the threading model can be altered by the pooling gem to disable handling of exceptions.

For example, the gem `connection_pool` disables the handling when it creates an instance, and enables the handling when you're using a connection:

```ruby
  def with(options = {})
    Thread.handle_interrupt(Exception => :never) do
      conn = checkout(options)
      begin
        Thread.handle_interrupt(Exception => :immediate) do
          yield conn
        end
      ensure
        checkin
      end
    end
  end
```

When you've setup a connection and released it to the pool, it slumbers in a Thread of which the handling of exceptions is disabled. If you receive a (forced) disconnect or a NetworkError occurs, it triggers the recovery (when enabled) of Bunny. This recovery relies on a ShutdownSignal to be raised on the `reader_loop` thread which needs to be caught by the `clean_up_on_shutdown` in `Session`. Because the exception is not handled anymore, the code in `maybe_shutdown_reader_loop` in `Session` tries to join the `reader_loop` thread to force a re-raise of a thrown unhandled exception. That join is done on the `reader_loop` thread which is the same thread that started the recovery so we'll receive a ThreadError telling us "Target thread must not be current thread (ThreadError)".

To prevent the above from happening we can check if the thread we're trying to join is the same thread that is currently running and not join if it is the same thread.
@michaelklishin michaelklishin self-assigned this Feb 4, 2020
@fuegas fuegas requested a review from michaelklishin February 4, 2020 23:55
@michaelklishin michaelklishin merged commit e518b0d into ruby-amqp:master Feb 5, 2020
michaelklishin added a commit that referenced this pull request Feb 5, 2020
Do not join @thread if that is the current Thread

(cherry picked from commit e518b0d)
@michaelklishin
Copy link
Member

Backported to 2.14.x-stable. Please proceed with whatever you were doing before discovering #589, if no issues come up in a few days I will cut a release. Thanks again for contributing a really non-trivial usability improvement!

@michaelklishin michaelklishin added this to the 2.14.4 milestone Feb 5, 2020
michaelklishin added a commit that referenced this pull request Feb 5, 2020
michaelklishin added a commit that referenced this pull request Feb 5, 2020
References #589, #590.

(cherry picked from commit 784ebf2)
@fuegas
Copy link
Contributor Author

fuegas commented Feb 5, 2020

I'll patch our staging environment and run some rigorous tests on it. If anything pops up I'll leave a comment in #589 or #590.

@fuegas
Copy link
Contributor Author

fuegas commented Feb 11, 2020

@michaelklishin: just as an update, we've encountered no irregularities during testing. We did see correctly recovering connections.

@michaelklishin
Copy link
Member

OK, I'll cut a new release now. Thanks for following up.

@michaelklishin
Copy link
Member

2.14.4 is out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Forced connection close on Bunny with ConnectionPool can't recover
2 participants