It’s a lazy Sunday and I’m looking at the server logs thinking to myself: hm, no one is on the server, all is quiet, why not do an upgrade? I did some testing on my localhost to make sure I thought I knew what was happening and I then I logged onto our Production system. After doing a
gem update I restarted the servers and got the very very unwelcome message:
Ruby on Rails application could not be started These are the possible causes: There may be a syntax error in the application's code. Please check for such errors and fix them. A required library may not installed. Please install all libraries that this application requires. The application may not be properly configured. Please check whether all configuration files are written correctly, fix any incorrect configurations, and restart this application. A service that the application relies on (such as the database server or the Ferret search engine server) may not have been started. Please start that service. Further information about the error may have been written to the application's log file. Please check it in order to analyse the problem. Error message: can't activate activesupport (>= 2.3.2, runtime), already activated activesupport-2.2.2 Exception class: Gem::Exception Application root: /xxx/xxx/xxx/xxxxxxxxxxx/xxx/xx/x Backtrace: # File Line Location 0 /usr/lib64/ruby/site_ruby/1.8/rubygems.rb 149 in `activate' 1 /usr/lib64/ruby/site_ruby/1.8/rubygems.rb 165 in `activate' 2 /usr/lib64/ruby/site_ruby/1.8/rubygems.rb 164 in `each' 3 /usr/lib64/ruby/site_ruby/1.8/rubygems.rb 164 in `activate' 4 /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb 35 in `require' ...
Fortunately I had save the output of a
gem list prior to the update so I could compare all the things that got touched. After futzing around with things for a while I started to try to back out the upgrades by
gem uninstall GEM -v VER. This went on for a bit, and the situation wasn’t improving. Then after the
activemerchant gem was uninstalled that’s when the apps came back to life. Aha!
I looked at my localhost setup and my
environment.rb was actually different from production’s in one significant way:
gem ‘activemerchant’, ‘=1.4.1′
Oh. The clue was also that upon uninstalling activemerchant 1.4.2 there was a message about it depending on Rails 2.3. Well, after updating the production
environment.rb everything went back to working order.
- Thorough testing on localhost doesn’t guarantee it works on production
- Updating gems en masse can cause umpteen problems — always update specific gems
- Use a maintenance window in the middle of the night