Phased restart crash when upgrading Rack Gem despite using 'prune_bundler'

[TheTeaNerd]

Summary

I upgraded rack in my Gemfile and Puma/Bundler unexpectedly crashed on restart. As we use prune_bundler in our cluster and don't preload, we expected that the Puma cluster master would be isolated from Gem changes, but able to respawn new workers which use the new Gems.

It only seems to crash if Gems directly used by Puma are upgraded. The version of Bundler doesn't seem to matter (but not exhaustively tested.)

Have I understood the expected behaviour correctly? Any workarounds or suggestions welcomed.

How to replicate

The Production issue can be simply replicated in Development without needing to do a full deploy:

• My Gemfile.lock specifies rack (1.5.2) and puma (2.11.2)
• Load gems: bundle install
• Config file as per the deployment guide

 worker 2
 threads 1
 prune_bundler              <-- The important bit
 rackup DefaultRackup
 port 3000
 environment 'development'

• Let's start Puma: bundle exec puma -C ./config/puma.rb

* Pruning Bundler environment                   <-- Good
[9372] Puma starting in cluster mode...
[9372] * Version 2.11.2 (ruby 2.1.2-p95), codename: Intrepid Squirrel
[9372] * Min threads: 1, max threads: 4
[9372] * Environment: development
[9372] * Process workers: 2
[9372] * Phased restart available
[9372] * Listening on tcp://0.0.0.0:3000
[9372] Use Ctrl-C to stop
[9374] + Gemfile in context: /Users/<me>/Documents/<org>/<project>/Gemfile
[9375] + Gemfile in context: /Users/<me>/Documents/<org>/<project>/Gemfile
[9372] - Worker 0 (pid: 9374) booted, phase: 0
[9372] - Worker 1 (pid: 9375) booted, phase: 0

• Update the Gemfile.lock to use rack (1.5.3)
• bundle install
• Ask for a phased restart: kill -USR1 <pid of puma cluster master>
• Crash:

/Users/<me>/.rvm/gems/ruby-2.1.2@<project>/gems/bundler-<version>/lib/bundler/runtime.rb:34: in `block in setup':
You have already activated rack 1.5.2, but your Gemfile requires rack 1.5.3.
Prepending `bundle exec` to your command may solve this. (Gem::LoadError)

[ebeigarts]

Same issue here, locking puma and rack to specific versions as a workaround, so that bundle update doesn't try to update them.

gem 'puma', '2.11.3', require: false
gem 'rack', '1.5.3' # https://github.com/puma/puma/issues/705

[chewi]

Also happens with USR2.

I think I see how too. Puma fires up the new instance with a bunch of -I options pointing to the few dependencies it needs. One of these is rack. I need to look closer to understand why it doesn't pick up the newer version, although the newer version may not be the one your application is about to use. It's also not clear whether this is a known weakness in the mechanism. It's not documented, at least.

Wait, I've got an idea…

[chewi]

Hmm it's harder than I thought. I was trying to get it to do a bundle exec inside prune_bundler before re-execing yet again without Bundler like it currently does. I don't understand why it's so complicated though. If you allow the Bundler environment in Puma itself, why can't the restart simply be something like this?

Bundler.with_clean_env do
  Kernel.exec(Gem.ruby, '-S', 'bundle', 'exec', *argv)
end

I quickly hacked this in and it seems to work here but maybe I'm missing the bigger picture.

[evanphx]

I'm looking into this today.

[evanphx]

With a totally minimal app I'm not reproducing what you're seeing. I did the following steps:

1. Add rack = 1.5.2 and puma to a Gemfile
2. Add prune_bundler and 'workers 2` to conf.rb
3. Add run lambda { |env| [200, {}, ["Hello World"]] } to hello.ru
4. Run bundle exec puma -C conf.rb hello.ru
5. See that puma boots fine and check that it responds
6. Change the Gemfile to have rack = 1.5.3
7. Run bundle update rack
8. Run kill -USR1 $PUMA_MASTER_PID
9. See the workers restart phased properly

Can you try this with your Gemfile and the simple hello.ru and see what the results are?

[evanphx]

Also, everyone seeing this problem, can you dump in your configuration files? Maybe there is something that is cause this in there.

[chewi]

I'm back from work now so can't test this in a hurry but try modifying the rack version in Gemfile.lock by hand instead of locking the Gemfile to a specific version. That's more representative of what's happening in production.

[evanphx]

Editing the Gemfile.lock directly to change the version of rack doesn't
seem to cause a crash.

On Wed, Jun 10, 2015 at 12:01 PM, James Le Cuirot notifications@github.com
wrote:

I'm back from work now so can't test this in a hurry but try modifying the
rack version in Gemfile.lock by hand instead of locking the Gemfile to a
specific version. That's more representative of what's happening in
production.

—
Reply to this email directly or view it on GitHub
#705 (comment).

[chewi]

Indeed, I just tried to reproduce that way at home and failed. I tried with a pristine Rails app and that crashed like before. So Rails must be a factor.

Actually that makes sense. hello.ru does not load Bundler by itself so all that's happening is that Puma is stripping Bundler out and it subsequently never gets used. Rails fires up Bundler in boot.rb.

Bingo, if you add require 'bundler/setup' to the start of hello.ru then that triggers it.

[chewi]

I should add that the existing solution is broken for basic applications like hello.ru that do not explicitly load Bundler by themselves. Puma strips out Bundler and only loads dependencies required for itself so any additional dependencies required by the application would be missing. My simpler solution, suggested above, would fix this problem, as well as negating the need for any of the existing prune_bundler code, including puma-wild. Is there a downside?

[evanphx]

@chewi Oh yes, quite right. Let me dig into this and figure out what's going on.

[chewi]

Rack 1.5.5 is out now and I'm apprehensive about this tripping us (and others!) up in production again. I've made a suggestion but it's down to you to decide on the best solution.

[chewi]

@evanphx Well that was unexpected. Are you sure this is the right way to go? It does sidestep the issue as Puma has no other dependencies but it's not inconceivable that you may want to add one later. Removing Rack clearly also came at the cost of expanding the code base a little. Was there a problem with my suggestion?

[evanphx]

@chewi Your solution doesn't work because it leaves the process in a unreliable state because it's simply not possible to load the gems from a Gemfile into the process where another Gemfile is already loaded.

Removing the rack dependency is really the only way to make puma resilient against Gemfile changes.

[chewi]

@evanphx I thought that between Bundler.with_clean_env and Kernel.exec, you'd get a totally clean slate but I trust your judgement.

[evanphx]

@chewi Part of the reason that doesn't work (I had to think through the whole flow just now) is would blow up if the users Gemfile did not include rack, which is certainly possible.

[chewi]

@evanphx Shouldn't the Gemfile include puma and therefore rack?

[evanphx]

Mmmm, that is a good point. I started the experiment to remove the rack dependency on a plane with the idea that I wasn't actually using much of rack anyway. I was correct as well, I only used a tiny bit of utils, the builder, and the common logger. If I had used a ton of it, I doubt I'd have gone this direction, but since I was just a little bit, the dependency seemed more cumbersome.

[chewi]

That's fair enough and I really don't mind but I thought I should follow this up because my approach would ultimately be less code (e.g. no more puma-wild) and would keep the door open for other dependencies in future. This could have happened with any gem, it's just that rack was the only dependency.

[evanphx]

@chewi You're right, it is simpler. I think now that I've removed the only gem dependency that puma has I should be able to remove a whole bunch of code from puma-wild (perhaps even all of it).

In the future, if gem dependencies are added, I'll likely implement your idea.

[dcrec1]

Sorry, how do I resolve this problem? Im using a Rails application and when restarting I get the follow message:

/usr/local/lib/ruby/gems/2.3.0/gems/bundler-1.13.6/lib/bundler/runtime.rb:40:in block in setup': You have already activated rack 2.0.1, but your Gemfile requires rack 2.0.3. Prepending bundle exec to your command may solve this. (Gem::LoadError)`

[evmorov]

Sorry, how do I resolve this problem?

Add require 'bundler/setup' to your config/puma.rb

[gencer]

Sorry, how do I resolve this problem?

Add require 'bundler/setup' to your config/puma.rb

After adding this to top, puma server did not start at all. Is there any other solution for updated gems on puma workers?