Phusion Blog

Ruby Rogues 143: Phusion Passenger Enterprise with Hongli Lai and Tinco Andringa

We’ve been invited by Ruby Rogues to participate in a podcast about Phusion Passenger Enterprise. This podcast covers the following topics:

Hongli Lai and Tinco Andringa Introductions
Phusion Passenger introduction
Rack
Node.js, MeteorJS, Python Support
Processes and Threads
- Ruby Rogues Episode #58 – Book Club: Working with Unix Processes with Jesse Storimer
- Ruby Enterprise Edition
- Smart Spawning
Advantages of Phusion Passenger Enterprise
- Rolling Restarts
- Mass Deployment
Passenger vs Unicorn
Error Resistant Deploys
Hosting
- DreamHost
Apache, Nginx support
Stability Issues
Documentation and Support

Listen to the podcast at the Ruby Rogues website

Thanks Ruby Rogues for hosting us!

Ruby Enterprise Edition 1.8.7-2012.02 released; End of Life imminent

Update: Ruby Enterprise Edition has officially reached End-of-Life in 2012.

What is Ruby Enterprise Edition?

Ruby Enterprise Edition (REE) is a server-oriented distribution of the official Ruby interpreter, and includes various additional enhancements, such as:

A “copy-on-write friendly” garbage collector, capable of reducing Ruby on Rails applications’ memory usage by 33% on average.
The tcmalloc memory allocator, which lowers overall memory usage and boosts memory allocation speed.
The ability to performance tune the garbage collector.
The MBARI patch set, for improved garbage collection efficiency.
Various analysis and debugging features.

REE can be easily installed in parallel to your existing Ruby interpreter, allowing you switch to REE with minimal hassle or risk. REE has been out for several years now and is already used by many high-profile websites and organizations, such as New York Times, Twitter, Shopify and 37signals.

Ruby Enterprise Edition is 100% open source.

Changes

Ruby Enterprise Edition 2011.12 and 2012.01 have actually also been quietly released, but they didn’t have proper announcements on our blog. Here’s a cumulative summary:

Upgraded to Ruby 1.8.7-p358

REE 2011.01 was based on 1.8.7-p330. This latest version includes not only bug fixes but also various security fixes.

Upgraded to RubyGems 1.8.15

The previous REE release included RubyGems 1.5.2.

For a long time we’ve been reluctant to upgrade RubyGems past 1.5 because many libraries were incompatible with RubyGems > 1.5. But today, the situation has been reversed: many gemspecs in the RubyGems.org repository are incompatible with RubyGems < 1.8. As such, we’ve upgraded RubyGems to the latest version.

Upgraded to the MBARI 8 patch set

Brent Roman has released a new version of his MBARI patch set, which solves many mysterious stability problems that plagued previous versions. This REE release should be significantly more stable.

Compatible with Xcode >= 4 and OS X Lion

The REE installer now properly advises Xcode >= 4 users and OS X Lion users on how to install the proper compiler that’s necessary for installing REE.

Experimental zero-copy context switch patch removed

This experimental patch set was never production-ready, so as of this release it has been removed.

End of Life

Support for Ruby 1.8.x is slowly being dropped by the upstream Ruby core developers in favor of Ruby 1.9 and beyond. The Rails team has recently announced that they will be dropping Ruby 1.8 support in Rails 4. As such, we are also slowly End-of-Life’ing Ruby Enterprise Edition.

We have no plans to create a Ruby 1.9-based version of Ruby Enterprise Edition for the following reasons:

A copy-on-write patch has recently been checked into Ruby 2.0.
Many of the patches in Ruby Enterprise Edition are simply not necessary in 1.9.
We wish to focus our efforts on Phusion Passenger and other products. Instead of doing many things poorly, we want to do a few things, but do them very very well.

We plan on providing minor fixes and updates for the time being, but users are recommended to slowly migrate to Ruby 1.9. Phusion Passenger <= 3.1 users can use reverse proxy setups to run multiple Ruby versions on the same server. Starting from Phusion Passenger 3.2 it will support multiple Ruby versions natively without the need for reverse proxy setups.

Looking for new maintainers

We believe that Ruby Enterprise Edition has served its purpose. That said, we understand that many people still rely on Ruby 1.8 for the time being because of compatibility issues. Therefore we would like to ask for volunteers who want to take over maintenance of Ruby Enterprise Edition. Please contact us if you are interested!

Download & upgrade

To install Ruby Enterprise Edition, please visit the download page. To upgrade from a previous version, simply install into the same prefix that you installed to last time. Please also refer to the documentation for upgrade instructions.

XCode 4 ships a buggy compiler

You may have heard of LLVM, a compiler infrastructure library. You may have heard of GCC, the GNU C and C++ compiler. Those two are completely separate software products, but there exists llvm-gcc which is a GCC frontend that utilizes LLVM. All OS X versions <= Snow Leopard originally shipped with regular gcc, but as of Xcode 4 (which is also the default on OS X Lion) Apple has switched to llvm-gcc as their default compiler. “Hurray”, some people may think, as they heard that LLVM generates better code and/or is faster than default GCC. Unfortunately, the reality is not that great.

llvm-gcc is unmaintained and is considered deprecated by its developers. It has many bugs that will never be fixed. Unfortunately these are not obscure bugs that few people will notice: today, while working on Ruby Enterprise Edition, we encountered several! Calling alloca(0) should return a pointer to the top of the stack. On llvm-gcc however, that code returns NULL but only when compiled with -O2! llvm-gcc also generates subtly different code that causes the MBARI patch set to fail completely.

To make matters worse, consider this program:

#include <alloca.h>

int
main() {
    if (alloca(0) != NULL) {
        return 0;
    } else {
        return 1;
    }
}

Question: when compiled with “llvm-gcc -O2”, does this program exit with 0 or with 1?

That was a trick question. It always returns with 0, even when alloca(0) is broken and actually returns NULL. Turns out the optimizer in -O2 thinks that alloca(0) doesn’t return NULL (even though it does) and replaces if (alloca(0) != NULL) with if (true).

It is unfortunate that we have to declare OS X’s default compiler as being broken. We do not recommend its use. Instead, we recommend people to use /usr/bin/gcc-4.2 (which is the non-LLVM GCC compiler) instead of /usr/bin/gcc (which is a symlink to /usr/bin/llvm-gcc). You can enable this on most build systems by setting these two environment variables:

export CC=gcc-4.2
export CXX=g++-4.2

On a side note, Ruby Enterprise Edition 2011.12 is being released. The official announcement hasn’t been written yet, but we’ve made haste because of the recent Ruby security vulnerability. You can already get the files on the Ruby Enterprise Edition website. An announcement will follow soon.

Update

some people have pointed out that they consider alloca(0) undefined. I guess it depends on the way you interpret the documentation. It would seem pretty clear to me what alloca(0) means, just like what memcpy(x, y, 0) means, but it’s understandable when some people would interpret it as undefined.
Still, alloca(0) failing is only one of the problems that prevented Ruby from compiling properly.

Why alloca(0)?

alloca(0) is used in Ruby Enterprise Edition’s conservative garbage collector, as a way to detect the end of the stack. There’s a bunch of fallback #ifdefs in the code: on platforms that don’t have alloca, it detects the end of the stack by calling a forced-non-inline function which returns the address of its sole local variable, but that assumes that the compiler supports the ‘noinline’ keyword. In any case, all of versions depend on highly platform-specific behavior.

Installing Ruby Enterprise Edition on OS X Lion

We’ve received some reports about users being unable to install Ruby Enterprise Edition on OS X Lion. Apparently the compilation process segfaults. It turns out that OS X Lion has switched to llvm-gcc as the default compiler. We currently suspect that the segfaults are caused by incompatibility between llvm-gcc’s code generation and the MBARI patch set (similar problems have been seen in the past but with other compilers on other platforms).

To install Ruby Enterprise Edition on Lion one needs to force the compiler to plain GCC instead of llvm-gcc. This can be done by setting the environment variable CC to /usr/bin/gcc-4.2. For example:

$ sudo bash
# export CC=/usr/bin/gcc-4.2
# ./installer

Or, when using RVM (from Stack Overflow):

$ rvm remove ree
$ export CC=/usr/bin/gcc-4.2
$ rvm install --force ree

Ruby Enterprise Edition 1.8.7-2011.03 released

Yes folks, another REE release within 2 weeks. We won’t bother you with updates again for a while now, I swear. 😉 But this update is worth it. The changes are as follows.

Fixed threading bugs

As mentioned in the release notes for REE 2011.01, MRI 1.8 has threading bugs that still aren’t resolved in 1.8.7-p334. REE 2011.01 contains a fix for this problem, but after running it for a while in production we’ve discovered that the fix may cause crashes.

It turned out that MRI’s 1.8 branch (the one that is to become MRI 1.8.8) has an updated fix that doesn’t crash, and it has had this fix for more than a year now, but it isn’t included in MRI 1.8.7. We’ve notified the Ruby core team about this and Shyouhei Urabe says he’ll backport it to the next MRI 1.8.7 release. In the mean time we’ve taken the liberty to backport it ourselves.

Users with heavily multi-threaded Ruby apps are strongly advised to upgrade to REE 2011.03. Without this fix threading is effectively useless on both the upstream MRI 1.8.7 releases as well as previous REE releases.

Fixed compilation problems and potential crashes on OS X when MacPorts are used

OS X has this interesting ecosystem in which a lot of users install software through MacPorts. MacPorts software is typically installed to /opt/local. This causes two problems.

The first one is that the compiler does not look in /opt/local by default. Some Ruby gems with native extensions rely on third-party libraries not shipped with OS X. If the user installs those dependencies with MacPorts then gems may not detect them automatically.

The other problem is that people end up with different versions of libraries that are also shipped with the OS. For example OS X comes with OpenSSL 0.9.8 while MacPorts installs OpenSSL 1.0.0 to /opt/local. This may cause crashes when some libraries try to use OpenSSL 0.9.8 and others try to use 1.0.0! For example the we’ve installed libcurl via MacPorts. libcurl depends on OpenSSL so MacPorts installs that as well. The Curb gem depends on libcurl, and its extconf.rb is smart enough to look in /opt/local for libcurl, so the Curb C extension ends up being compiled against MacPorts’s OpenSSL 1.0.0. In the mean time, the Ruby OpenSSL extension’s extconf.rb does not look in /opt/local, so it ends up being compiled against OpenSSL 0.9.8. The net/https library uses the Ruby OpenSSL extension. So if your app uses both Curb and net/https (which may happen indirectly, e.g. via ActiveResource) then Ruby may attempt to load both OpenSSL libraries. These two conflict with each other and may cause crashes. So if you’re an OS X user and you’ve seen mysterious segmentation fault crashes in net/http.rb, then this is probably the cause. Obviously the problem is not limited to only OpenSSL: any situation involving two different versions of the same library to be loaded will cause problems as well.

To fix all this we’ve now modified REE to always look in /opt/local when compiling things.

Download & upgrade

Ruby Enterprise Edition 1.8.7-2011.02 released

What is Ruby Enterprise Edition?

Ruby Enterprise Edition (REE) is a server-oriented distribution of the official Ruby interpreter, and includes various additional enhancements, such as:

A “copy-on-write friendly” garbage collector, capable of reducing Ruby on Rails applications’ memory usage by 33% on average.
The tcmalloc memory allocator, which lowers overall memory usage and boosts memory allocation speed.
The ability to performance tune the garbage collector.
The MBARI patch set, for improved garbage collection efficiency.
The zero-copy context switching patch, included as an experimental feature.
Various analysis and debugging features.

“We switched to enterprise ruby to get the full benefit of the [copy-on-write] memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices.”
— Tobias Lütke (Shopify)

Ruby Enterprise Edition is 100% open source.

Changes

This REE release is merged with Ruby 1.8.7-p334 because of the recent security vulnerabilities. Users are advised to upgrade as soon as possible.

Download & upgrade

Ruby Enterprise Edition 1.8.7-2011.01 released

What is Ruby Enterprise Edition?

Ruby Enterprise Edition (REE) is a server-oriented distribution of the official Ruby interpreter, and includes various additional enhancements, such as:

A “copy-on-write friendly” garbage collector, capable of reducing Ruby on Rails applications’ memory usage by 33% on average.
The tcmalloc memory allocator, which lowers overall memory usage and boosts memory allocation speed.
The ability to performance tune the garbage collector.
The MBARI patch set, for improved garbage collection efficiency.
The zero-copy context switching patch, included as an experimental feature.
Various analysis and debugging features.

“We switched to enterprise ruby to get the full benefit of the [copy-on-write] memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices.”
— Tobias Lütke (Shopify)

Ruby Enterprise Edition is 100% open source.

Changes

Upgraded to Ruby 1.8.7-p330

The previous REE release was based on 1.8.7-p249.

Upgraded to RubyGems 1.5.2

The previous REE release included RubyGems 1.3.7.

Please note that Rails versions older than about 2.3.8 are not compatible with RubyGems 1.5. So if you find your Rails apps unable to start after upgrading REE, you know why. There are two things you can do:

Manually downgrade RubyGems.
Upgrade your apps to Rails 2.3.11 or 3.0.4. We recommend this choice because of the recent security fixes in Rails.

Upgraded tcmalloc to version 1.7

The previous REE release included tcmalloc 1.4. Version 1.7 also supports OS X Snow Leopard, so tcmalloc support in REE is now enabled by default on OS X.

Fix backport: zlib garbage collection problem (fixes ‘bundle install’ problems)

We’ve received a lot of reports about ‘bundle install’ sometimes failing for mysterious reasons. It reports something along the lines of:

tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error)

The reason behind this problem has long remained elusive because of the difficulty to reproduce it, until Aman Gupta found a lead. It turned out to be a garbage collection bug in the Ruby zlib extension. The bug is even present in the latest upstream Ruby 1.8.7 release (p330), though not in Ruby 1.9. We’ve backported the fix to 1.8 and included it REE. Issue #45.

Fix backport: mutex deadlock detection

Multithreaded programs can sometimes crash with a mysterious “deadlock detected – mutual join” error message even if the code is correct (example). This is apparently a bug in Ruby’s Mutex implementation, but the fix is not included in the latest upstream Ruby 1.8.7 release (p330). We’ve backported the fix and included it in REE. Issue #46.

Fixed –dont-install-useful-gems in the installer

The installer used to crash when this flag is passed. This has now been fixed. Issue #42.

Fixed memory leaks in the zero-copy context switching patch

Fix contributed by Kurt Stephens. Issue #57.

Download & upgrade

Ruby Enterprise Edition 1.8.7-2010.02 released

It has been a while since the last REE release. We apologize for not releasing earlier, it’s been very busy for us lately. Nonetheless, a number of important issues have motivated us to release again, including various Rails 3 compatibility issues. Read on for more information.

What is Ruby Enterprise Edition?

Ruby Enterprise Edition (REE) is a server-oriented distribution of the official Ruby interpreter, and includes various additional enhancements, such as:

A “copy-on-write friendly” garbage collector, capable of reducing Ruby on Rails applications’ memory usage by 33% on average.
The tcmalloc memory allocator, which lowers overall memory usage and boosts memory allocation speed.
The ability to performance tune the garbage collector.
The MBARI patch set, for improved garbage collection efficiency.
The zero-copy context switching patch, included as an experimental feature.
Various analysis and debugging features.

REE can be easily installed in parallel to your existing Ruby interpreter, allowing you switch to REE with minimal hassle or risk. REE has been out for about a year now and is already used by many high-profile websites and organizations, such as New York Times, Shopify and 37signals.

“We switched to enterprise ruby to get the full benefit of the [copy-on-write] memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices.”
— Tobias Lütke (Shopify)

Ruby Enterprise Edition is 100% open source.

Changes

Upgraded to Ruby 1.8.7-p249

The previous REE release was based on 1.8.7-p248. p249 hasn’t changed much: it only includes some WEBrick fixes.

Upgraded to RubyGems 1.3.7

The previous REE release included RubyGems 1.3.5. 1.3.7 is required by the latest version of Bundler as well as Rails 3.

Backported various bug fixes

The following bug fixes are fixed by upstream Ruby, but not yet released, i.e. these fixes are not part of the latest Ruby 1.8.7-p249 release. We’ve backported these fixes because they solve important compatibility issues.

Fixed a Marshal bug that was apparently caused by GCC optimizations. This is a major bug that appears to be responsible for all the REE crash bug reports of late. It is so severe that the Rails 3 documentation actually recommends not using 1.8.7-p248 and 1.8.7-p249:

“Note that Ruby 1.8.7 p248 and p249 has marshaling bugs that crash Rails 3.0.0. Ruby 1.9.1 outright segfaults on Rails 3.0.0, so if you want to use Rails 3 with 1.9.x, jump on 1.9.2 trunk for smooth sailing.”

Ruby bug #2557. Given that Ruby 1.8 is still so widely used, being forced to use Ruby 1.9.2 (-dev version even) is not such a good thing. With these backports Rails 3 should be once again usable on 1.8, at least until upstream releases a new version with the fix.
Fixed an “undefined method `closed?’ for nil:NilClass” Net::HTTP bug. Ruby issue #2708 and REE issue #35.
Fixed a bug where the ‘super’ keyword doesn’t behave correctly. Ruby issue #2537 and REE issue #40.

Fixed various FreeBSD issues

REE on FreeBSD would occasionally crash with a bizarre “Illegal Instruction” error. After some though investigations, it would appear that the problem is caused by the MBARI patch set in combination with some FreeBSD oddities. MBARI tries to reserve the upper part of the system stack for the garbage collector. In order to do this, it queries the OS for the size of the stack. FreeBSD reports a large size (on 64-bit FreeBSD it reports 512 MB by default), but in reality only about 4 MB could be used: if you go over it then the process will crash. We’ve fixed this issue by limiting the stack usage to 4 MB when on FreeBSD.
Fixed some long-standing iconv installation bugs. The iconv Ruby extension is used by various important parts of Ruby and Rails. FreeBSD installs the iconv .h headers files into /usr/local/include, but gcc doesn’t look in this location by default, and neither does the iconv extension’s extconf.rb. We’ve modified the REE installer to force the compiler to look in /usr/local/include while installing the iconv extension.
Added a bootstrap binary for x86_64 FreeBSD 8. This means that on this platform you don’t have to install Ruby first before you can run the REE installer (which is written in Ruby).

Rational and gcd performance improvement patches

Kurt Stephens has contributed a set of patches which dramatically improve the performance of the Rational class and the #gcd method. Rational performance has been improved by over 50%. Ruby issue #2561 and REE issue #23

Various other minor bug fixes

GEM_HOME, GEM_PATH and RUBYOPT are unset before running the installer so that those options can’t interfere with installation.
RUBY_HEAP_SLOTS_GROWTH_FACTOR wasn’t properly parsed as a floating point number. This has now been fixed.
Fixed OpenSSL compilation problems. Patch contributed by hso@nosneros.net. REE issue #39.

More Ubuntu packages

We now provide packages for:

Ubuntu 8.04 32-bit
Ubuntu 8.04 64-bit
Ubuntu 10.04 32-bit
Ubuntu 10.04 64-bit

Download & upgrade

Ruby Enterprise Edition 1.8.7-2010.01 released

What is Ruby Enterprise Edition?

Ruby Enterprise Edition (REE) is a server-oriented distribution of the official Ruby interpreter, and includes various additional enhancements, such as:

A “copy-on-write friendly” garbage collector, capable of reducing Ruby on Rails applications’ memory usage by 33% on average.
The tcmalloc memory allocator, which lowers overall memory usage and boosts memory allocation speed.
The ability to performance tune the garbage collector.
The MBARI patch set, for improved garbage collection efficiency.
The zero-copy context switching patch, included as an experimental feature.
Various analysis and debugging features.

“We switched to enterprise ruby to get the full benefit of the [copy-on-write] memory characteristics and we can absolutely confirm the memory savings of 30% some others have reported. This is many thousand dollars of savings even at today’s hardware prices.”
— Tobias Lütke (Shopify)

Ruby Enterprise Edition is 100% open source.

Changes

Upgraded to Ruby 1.8.7-p248

The previous REE release was based on 1.8.7-p174.

Improved compiler optimization options

The previous REE release was compiled with -Os -fno-strict-aliasing.

-fno-strict-aliasing was used to avoid improper code generation by GCC 4.4. This was actually caused by some aliasing bugs in Ruby’s util.c source file. The problems have been fixed in 1.8.7-p174 so we’ve now removed this compilation flag, allowing for better compiler optimizations.

It turned out that -O2 yields better performance than -Os in many production environments, though some microbenchmarks might indicate otherwise. Therefore we’ve now replaced -Os with -O2.

Fixed OpenSSL extension compilation problems on systems with OpenSSL 1.0

At this time, upstream Ruby cannot be compiled on systems with OpenSSL 1.0 because of compatibility problems in the Ruby OpenSSL extension. Fedora 12 includes OpenSSL 1.0. We’ve applied a patch by the Fedora guys and added some minor changes to fix some compilation warnings. These patches have been send upstream. Ruby issue #2022.

Backported an IO#write exception bug fix

Upstream Ruby 1.8.7-p248 has a bug in its IO#write method: it always raises Errno::EINVAL even when a different error occured. We found this problem while testing Phusion Passenger on this Ruby release.

We’ve submitted a patch upstream. This patch is also applied in this REE release.

Thread timer fix now merged upstream

Previous REE releases included Joe Damato’s and Aman Gupta’s thread timer fix. This fix has now found its way back upstream and is included by default in 1.8.7-p248, so we’ve removed the patch from our source tree.

Fix a crash bug in the zero-copy context switching patch set

This crash can be reproduced by running “god”, which will eventually cause a crash. Aman Gupta has fixed this problem.

Please note that the zero-copy context switching patch set is disabled by default, and must be explicitly enabled by passing –fast-threading to the installer. It is currently still marked as experimental because there are some known issues with the Kernel::fork method. Issue #9.

Ubuntu package now contains debugging symbols

Previous REE Ubuntu packages that we release had binaries with debugging symbols stripped, in order to minimize the package sizes. We no longer strip the debugging symbols now because Joe and Aman’s Memprof depends on the presence of debugging symbols. Memprof should work out-of-the-box with this release of REE.

Please note that although the binaries are larger, this does not affect performance in any way. The debugging symbols are only used for debugging and introspection purposes and do not affect the runtime behavior of Ruby at all.

Developer documentation is now installed by default

RDoc and RI documentation are now installed by default. You can avoid this by passing --no-dev-docs to the installer.

The Ubuntu packages include developer documentation.

Installer now checks for the existence of the ‘patch’ utility

This fixes bug #10.

Some documentation updates

Parts are contributed by Trevor Turk.

Download & upgrade

Google Tech Talk on Ruby Enterprise Edition

Last Friday we visited the awesome Googleplex and gave a tech talk there about Ruby Enterprise Edition. This talk elaborates a bit on how REE works under the hood. Many thanks to John Woodell for making this possible!

What is Ruby Enterprise Edition?

Changes

End of Life

Looking for new maintainers

Download & upgrade

Update

Why alloca(0)?

Download & upgrade

What is Ruby Enterprise Edition?

Changes

Download & upgrade

What is Ruby Enterprise Edition?

Changes

Download & upgrade

What is Ruby Enterprise Edition?

Changes

Download & upgrade

What is Ruby Enterprise Edition?

Changes

Download & upgrade

Featured articles