Phusion white papers Phusion overview

Phusion Blog

Phusion Passenger 3.0.0 final released

By Hongli Lai on October 18th, 2010

Phusion Passenger is an Apache and Nginx module for deploying Ruby web applications. It has a strong focus on ease of use, stability and performance. Phusion Passenger is built on top of tried-and-true, battle-hardened Unix technologies, yet at the same time introduces innovations not found in most traditional Unix servers. Since version 3.0 it can also run standalone without an external web server, making it not only easier for first-time users but also ideal on development environments.


Thanks for all the support on Twitter everyone!

What’s new compared to 2.2

Phusion Passenger 3 is a major new release and brings about many, many improvements in the areas of performance, features and stability. It is fully compatible with the latest technologies such as Rails 3.0, Ruby 1.9.2, RVM and Bundler.

You can read about the improvements in Phusion Passenger 3 through a set of articles that we had written on this subject:

What’s new compared to 3.0.0 RC 1

[Apache] `passenger-install-apache2-module –snippet` no longer prints ANSI color codes
This makes it possible to pipe its output directly to a config file. Fixes issue #548.
[Apache] Fixed thread stack size problems on FreeBSD
Previously it might crash for bizarre reasons.
[Nginx] Upgraded to Nginx 0.8.52 by default
This is because Nginx 0.8 has recently become the stable release series.
[Standalone] Fixed Unix domain socket support
There was a bug in Phusion Passenger Standalone which would cause the `-S` option to fail.

How do I upgrade to 3.0.0?

Via a gem

First install the gem with the following command:

gem install passenger

If you’re using Phusion Passenger for Apache or for Nginx, then re-run the Apache or Nginx module installer, whichever is appropriate:

passenger-install-apache2-module
passenger-install-nginx-module

At the end the installer will tell you to paste a configuration snippet into your web server config file. Replace the old snippet that you already had with this new one.

Phusion Passenger Standalone users don’t need to run anything else. Whenever you type

passenger start

it will automatically upgrade itself.

Via a native Linux package

John Leach from Brightbox has kindly provided Ubuntu packages for Phusion Passenger. The package is available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 3.0.0, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!

Phusion Passenger 3.0.0 RC 1 released

By Hongli Lai on October 1st, 2010

This is it, the first Release Candidate (internally named “pre 4”). A few more bugs have been fixed since beta 3, but if no more serious bugs are found then the next release will be 3.0.0 final.

What’s new

Autodetection of Ruby tool commands now take exe extension, –program-prefix and –program-suffix into account

Phusion Passenger had a pretty elaborate way of searching the system for Ruby tool commands like ‘gem’, ‘rake’, etc. The reason why all this search code exists is because the problem is much more complicated than just searching for “rake” in $PATH. A significant number of people have multiple Ruby versions installed on their system; let’s call these Ruby A and Ruby B. They run the Phusion Passenger installer with Ruby A and they installed the required gems with Ruby A’s RubyGems, but they have a Rake in $PATH that belongs to Ruby B. If we were to naively use the Rake in $PATH then the wrong Rake will be invoked and they would wonder why the gems they just installed aren’t detected. Our Ruby tool search code tries very hard to find the Rake that belongs to the Ruby interpreter that’s currently running.

However the code didn’t take the exe extension into consideration, as well as the –program-prefix and –program-suffix that Ruby could be configured with. For example, on some systems the Ruby command is called “ruby1.8” and Rake is called “rake1.8”. Similarly, the default JRuby binary distribution calls Rake “jrake”. We’ve now updated our search code to take this into account as well. Issue #537.

Usability and ease of installation remains to be one of our biggest goals. As you can see we’ve gone through great lengths to make sure that everything works as expected and that everything is as fool-proof as possible, even if the system might not be entirely correctly configured.

Fixed linking problems on some systems
Phusion Passenger uses math functions such as pow(). On some systems — .e.g. NetBSD, some versions of Solaris and some Linux distros — this requires Phusion Passenger to be linked to the Math library. Issue #539.
Phusion Passenger Standalone fix
When Phusion Passenger Standalone is daemonized with -d, it exits with a non-zero exit code. This has been fixed. Issue #536.
Preferred PCRE version upgraded to version 8.10
passenger-install-nginx-module automatically installs PCRE for you in case you didn’t already have it.

Installation/upgrade

First install the latest Phusion Passenger gem:

gem install passenger --pre

(Or you can download the tarball.)

If you want to install or upgrade the Apache or Nginx version, then run the installer as you’re used to:

passenger-install-apache2-module
passenger-install-nginx-module

At the end the installer will tell you to paste a configuration snippet. If you’re upgrading then replace the old snippet with the new one.

If you want to run Phusion Passenger Standalone, then run:

cd /path-to-your-app
passenger start

No special upgrade instructions needed.

The documentation

Users guide for Apache
Users guide for Nginx
Users guide for Standalone

Phusion Passenger 3.0.0 public beta 3 released

By Hongli Lai on September 20th, 2010

What’s new

More RVM issues fixed
A lot of people who happen to use RVM have reported various problems. Some of the symptoms include:

  • The spawn server crashes during startup.
  • Phusion Passenger Standalone prints ‘command not found’ errors.

It turned out that these problems occur when one has an older RVM version installed. Phusion Passenger depends on features provided by relatively new RVM versions. We’ve added sanity checks in Phusion Passenger so that it warns you if it detects an old RVM installation.

You can upgrade your RVM installation by running these commands:

rvm update --head
rvm reload
rvm repair all

We’ve also fixed support for system-wide RVM installs.

Aggressively checks for permissions
Many problems that people report on the Phusion Passenger discussion board are actually caused by wrong permissions set on the application files or on one of the parent directories. We’ve added aggressive permission checking code in Phusion Passenger. If any permissions are wrong then it will tell the user exactly which directory has wrong permissions and how to fix it. This should make Phusion Passenger a lot more fail-proof than before.
Fixed some more compilation problems
These problems occur on some systems but not on others.
Fixed passenger-status crash
When passenger-status tries to tell the user that it needs to be run as root, it used to crash instead because of a typo.
Phusion Passenger Standalone checks for GNU make on FreeBSD
Phusion Passenger Standalone depends on the GNU version of make in order to be able to show a compilation progress bar.

Installation/upgrade

First install the latest Phusion Passenger gem:

gem install passenger --pre

(Or you can download the tarball.)

If you want to install or upgrade the Apache or Nginx version, then run the installer as you’re used to:

passenger-install-apache2-module
passenger-install-nginx-module

At the end the installer will tell you to paste a configuration snippet. If you’re upgrading then replace the old snippet with the new one.

If you want to run Phusion Passenger Standalone, then run:

cd /path-to-your-app
passenger start

No special upgrade instructions needed.

The documentation

Users guide for Apache
Users guide for Nginx
Users guide for Standalone

Phusion Passenger 3.0.0 public beta 2 released

By Hongli Lai on September 17th, 2010

Thanks to all the users who have provided constructive feedback! This release fixes all newly reported issues so far.

Phusion Passenger Standalone now correctly handles paths with spaces in them
Beta 1 would error out with a weird message.
Fixed some RVM support issues
If you’re using RVM and you had problems with beta 1, try this version. Thanks to Wayne for helping out!
Improved watchdog startup error checking
If the watchdog crashes during startup then the reason wasn’t always obvious. We’ve improved the startup error checking code so that more error information can be collected.
Fixed Phusion Passenger Standalone binary compatibility string generation
The binary compatibility string is generated from many components, including part of the output of ‘uname -p’. It turns out that on some systems that command outputs in an unexpected format.
Fixed curl error detection
Curl is used to download things when wget is not available. Beta 1 did not correctly recognize cases in which curl exit because of an HTTP 404 error.
Fixed PassengerMinInstances default value on Apache
It’s supposed to be 1, but in beta 1 it was accidentally still 0.
Fixed support for older Apache 2 versions
Beta 1 used API calls which are only available in newer versions.
Fixed various compilation warnings
Including warnings on CentOS 5 and Ubuntu 10.04.

Installation/upgrade

First install the latest Phusion Passenger gem:

gem install passenger --pre

(Or you can download the tarball.)

If you want to install or upgrade the Apache or Nginx version, then run the installer as you’re used to:

passenger-install-apache2-module
passenger-install-nginx-module

At the end the installer will tell you to paste a configuration snippet. If you’re upgrading then replace the old snippet with the new one.

If you want to run Phusion Passenger Standalone, then run:

cd /path-to-your-app
passenger start

No special upgrade instructions needed.

The documentation

Users guide for Apache
Users guide for Nginx
Users guide for Standalone

Phusion Passenger 3.0.0 public beta 1 is out!

By Hongli Lai on September 15th, 2010

Phusion Passenger is software with which one can deploy Ruby web applications on the Apache or Nginx web server. Now, with the release of version 3.0.0 beta 1, it also comes with a standalone version that does not require an external web server. Please read http://www.modrails.com/ for details.

In the past period we’ve blogged extensively about the progress of Phusion Passenger 3. Many exciting changes have been introduced. It’s been a long ride, but today we are happy to announce the first Phusion Passenger 3 public beta! We’ve already tested Phusion Passenger 3 extensively in private, and now we’re giving the community a chance to test it as well in order to make the final release rock-solid.

We’ve received a lot of emails from people who are excited about Phusion Passenger 3. Thank you all for your support!

To recap, we’ve described Phusion Passenger 3 extensively in our past Technology Preview articles:

Except for mass deployment, all features described in the technology previews are available as open source in Phusion Passenger 3. A few more changes have been made since the last Technology Preview:

The ‘PassengerPreStart’ option
In Technology Preview 4 we described PassengerMinInstances. That option ensures that, at least x instances will be kept around once they’ve been spawned, but it does not ensure that instances are immediately spawned up during web server start. Instead we have a separate configuration option for that, called PassengerPreStart. PassengerPreStart and PassengerMinInstances and the reason why they’re separate configuration options are described extensively in the manual.
RVM support
RVM is becoming more and more popular. Previous versions of Phusion Passenger work fine with RVM, but one has to follow special installation instructions for it to work properly. We’ve noticed that many users are not aware of this and as a result couldn’t get Phusion Passenger working properly with RVM. In Phusion Passenger 3 we’ve collaborated with Wayne E. Seguin and added special support for RVM so that everything should work out-of-the-box, without special installation instructions.

Other minor changes

For completeness, here are a list of minor changes that have not been described in the Technology Previews so far:

  • Nginx is now compiled with SSL support by default.
  • Nginx can now be compiled on a noexec /tmp filesystem. Fixes bug #380.
  • Support for Nginx 0.8.
  • Dropped support for Nginx 0.6.
  • Much better Ruby 1.9.2 support. Our unit test suite has been refactored; Ruby 1.9.2 passes our unit test suite 100%.
  • The PassengerFriendlyErrorPages option has been added. This allows you to turn off the friendly error pages that Phusion Passenger normally displays in the event of an error, so that only a standard 500 error page is shown. This is documented in the manual.
  • The PassengerDebugLogFile option has been added. This is documented in the manual.
  • User switching support has been made more flexible through the PassengerUser, PassengerGroup, PassengerDefaultUser and PassengerDefaultGroup options. These are documented in the manual.
  • RailsSpawnMethod has been renamed to PassengerSpawnMethod because we now support smart spawning for Rack apps as well.
  • passenger-install-apache2-module and passenger-install-nginx-module can now be run concurrently without the one deleting files compiled by the other.
  • #at_exit blocks are now called during application process shutdown.
  • passenger-install-apache2-module now supports a –snippet option. When this option is given, the command outputs the Apache configuration snippet that should be pasted, but doesn’t do anything else.
  • Temporary files that Phusion Passenger places in /tmp are touched every 6 hours. This prevents /tmp cleaner daemons that are present on many systems from deleting these temp files that are essential for Phusion Passenger’s functioning. Bug #365.

Install

Installation is almost the same as Phusion Passenger 2. Type:

gem install passenger --pre

If you want to install the Apache or Nginx version, then run the installer as you’re used to:

passenger-install-apache2-module
passenger-install-nginx-module

Or you can download the tarball.

If you want to run Phusion Passenger Standalone, then run:

cd /path-to-your-app
passenger start

The documentation

Users guide for Apache
Users guide for Nginx
Users guide for Standalone

Things to come

Although Phusion Passenger 3.0 brings many exciting improvements, it is only the beginning. In parallel to Phusion Passenger 3, we have been working on some interesting things which we plan to reveal to the world shortly after the release of Phusion Passenger 3.0. Please stay tuned for future developments.

The Road to Passenger 3: Technology Preview 4 – Adding new features and removing old limitations

By Hongli Lai on July 29th, 2010

In the past two years that we’ve been developing Phusion Passenger, we’ve received not only many feature requests but also many criticisms about certain limitations. Some feature requests have been implemented in Phusion Passenger 2.x, some have not. Some limitations were solved in the life time of Phusion Passenger 2, others were not because they require a lot of refactoring first. In Phusion Passenger 3 we’ve extensively refactored the code to not only make a lot of new cool features possible, but also to lift a lot of the old limitations. In this Technology Preview we are pleased to announce these changes.

Asynchronous spawning

Previously, when application processes are being spawned, Phusion Passenger is unable to handle HTTP requests until the processes are done spawning, because Phusion Passenger is holding the lock on the application pool while this is happening. Some websites have apps that need a very long time to spawn (30+ seconds) and this would be a problem for them. This behavior would also cause a “thundering herd” problem: suppose that a traffic spike appears, then the first request will cause Phusion Passenger to spawn a process. The other requests in the spike are blocked until that’s done, and then all of a sudden they are processed. Phusion Passenger thinks it needs to spawn more, so it spawns another one and blocks the rest, and so on. This can cause a large number of processes to be spawned all of a sudden, causing a long delay.

In Phusion Passenger 3 spawning happens in the background so that no clients have to be blocked. This turns out to work so well that application process spawning is now virtually unnoticeable, except when spawning the first application process.

Ability to configure minimum number of processes

Phusion Passenger automatically shuts down processes that haven’t been accessed for a while, where “a while” is configurable through the PassengerPoolIdleTime directive. Many web applications are rarely accessed during the night, so what happens is that all application processes are shut down during the night and the first person in the morning who accesses the web application has to wait for some time while the first application process is being spawned. This problem could be solved by setting PassengerPoolIdleTime to 0 or to a large number which means that processes are kept around forever or for a long time, but this also means that application processes are not shut down during the night, which might still be desirable for resource utilization reasons.

Phusion Passenger 3 introduces a new, long-awaited configuration directive: PassengerMinInstances. As its name implies, PassengerMinInstances makes sure that at least the given number of processes will be started and kept around. This, in combination with asynchronous spawning, turns out to work so well that we’ve assigned a default value of 1 for PassengerMinInstances. With Phusion Passenger 3, spawning delays should become a thing of the past!

Smart spawning support for all Rack applications

Smart spawning is a core feature of Phusion Passenger since version 1.0. It can reduce the spawning time of Rails processes by as much as 90%, and in combination with Ruby Enterprise Edition it allows you to save 33% memory on average.

However, smart spawning was limited to Rails applications only, not for Rack applications. Starting from Rails 3, all Rails 3 applications are also Rack applications, and Phusion Passenger 2 only supports smart spawning of Rails 3 applications if you remove config.ru (thereby forcing Phusion Passenger to detect it as a Rails app, not a Rack app).

Phusion Passenger 3 now supports smart spawning for all Rack applications. This works transparently and without further configuration.

Ability to access individual application processes over HTTP

When one sends a request to the web server, Phusion Passenger routes the request to one of the application processes, but one never knows up front which one that’s going to be, nor is there a way to control it. This is fine for normal requests, but sometimes one wants to send a request to a specific application process, e.g. to debug something. Another use case for accessing individual application processes directly is broadcasting messages: e.g. telling every application process to clear some local in-memory caches.

This has always been possible with reverse proxy app servers like Mongrel and Thin because each of their processes listen on their own port. Phusion Passenger 3 now also allows one to access individual application processes directly. Each application process can now be accessed through its own TCP socket port. The port numbers are randomly allocated by the operating system and the protocol is plain HTTP, so you can use existing tools like ‘curl’.

We take security very seriously. These sockets are bound to 127.0.0.1 so it’s not possible to access them from remote computers. Furthermore, each socket is protected by its own unique randomly generated secure password. One can use ‘passenger-status’ to query the ports and passwords.

Global queuing now on by default

Many people with web apps that have long-running requests are familiar with the “slow Mongrel queue problem”. When one sets up Mongrel or Thin behind Nginx or Apache, it’s possible for new requests to be queued behind a Mongrel/Thin instance which is currently processing a long-running request. When other Mongrel/Thin instances become available, this new request is already queued behind the long-running request and cannot migrate to the other free instances. The more long-running requests one has the bigger of a problem this can become, resulting to very long response times for some users.

There are multiple ways to solve this problem, but Phusion Passenger has already solved this problem for a long time with a feature that we call global queuing. It was disabled by default because turning global queuing off would yield a little bit of extra performance in microbenchmarks. However for version 3.0 we’ve decided to turn it on by default: rather than saving those few milliseconds in benchmarks, we believe it’s much more important that all users get to have fair response times.

Ability to disable friendly error pages

One of the innovations that Phusion Passenger has brought us is the ability to show friendly error messages directly in the browser, e.g. when the web application fails to spawn because of a syntax error. This dramatically improves usability for developers, new and experienced alike, because one doesn’t have to manually dig into log files anymore. However this is not always desirable in production: although the error page is developer-friendly, it isn’t necessarily user-friendly. It might also expose information that the system administrator would rather not expose such as filenames.

Phusion Passenger 3 introduces a new configuration directive for controlling whether friendly error pages should be shown: ‘PassengerFriendlyErrorPages’. When turned off, Phusion Passenger will display the standard Apache/Nginx 500 Internal Server Error page instead, but all error messages are still logged to the web server error log file.

Nginx-specific improvements

In Phusion Passenger 3, Nginx is compiled with SSL support by default due to popular demand.

We’ve also introduced the following new configuration options:

passenger_set_cgi_param (name) (value)
This is the Phusion Passenger equivalent of proxy_module’s proxy_set_header. It allows you to pass arbitrary CGI environment parameters the web application.
passenger_buffer_response (on|off)
This is the Phusion Passenger equivalent of proxy_buffer. It was and still is off by default to allow streaming responses, but when streaming responses aren’t necessary one can turn this option on so that Nginx can gracefully handling clients that are slow at receiving responses.
passenger_ignore_client_abort (on|off)
This is the Phusion Passenger equivalent of proxy_ignore_client_abort.

In other news

In Technology Preview 3 we unveiled Phusion Passenger Lite. Based on various feedback that we’ve gotten since then, we’ve decided to rename this thing to Phusion Passenger Standalone in order to reduce confusion about what it is.

Towards the future

If there is one thing we’ve come to understand over time is that different businesses have different needs and constraints when it comes to deploying their applications. In order to provide the best experience and support to these businesses, we’re working on different versions of Phusion Passenger to accommodate them even better in their respective environments. In light of this, we want to underline that the technology previews we’re currently writing about will and have described the cool stuff that will be incorporated in the version intended for the most high demand environments. More information on this will follow. Having said that, almost everything we’ve blogged about up till this point will be included in the version that’s available for everybody.

The Road to Passenger 3: Technology Preview 3 – Closing the gap between development and production & rethinking the word “easy”

By Hongli Lai on July 1st, 2010

Update: “Phusion Passenger Lite” has been renamed to “Phusion Passenger Standalone” nowadays. The mass deployment feature has also been removed but will find its way back in the near future in a commercial version of Phusion Passenger.

Before Phusion Passenger came along, the most widely used Ruby app servers all implemented the same model which we refer to as the reverse proxy model. In this model, the user had to manually setup a bunch of app server processes and had to configure the web server to proxy requests to the app server processes. The technically inclined understand this model, but it is confusing to e.g. newcomers and to people who in general don’t have a lot of system administration skills or a reasonable understanding of HTTP. Most people were and still are much more familiar with PHP’s model, where you tell the web server where your app is and then have the web server take care of the rest for you. It was this confusion that caused all the uproar about sucky Rails deployment back in 2008.

While developing Phusion Passenger for Apache, we decided to follow a PHP-like model because ease of use was one of our main goals. No manual setups of app servers. No manual proxy configuration. Upload and go. For Phusion Passenger for Nginx, we continued to follow this model. Let’s call this the automatic model. As of 2010, Phusion Passenger appears to be the only widely-used Ruby app server that implements this model; the other widely-used Ruby app servers implement the reverse proxy model.

Reverse proxy vs automatic model: which one is better?

Ever since Phusion Passenger was first released, debates popped up about which one is superior. We believe that no model is inherently superior to the other. They are just different, meaning that both models have their own pros and cons. Which one is better for you depends a lot on your server infrastructure and your system administrators’ preferences.

Phusion Passenger’s automatic model:

  • Integrated into the web server. Processes are managed along with the web server itself, and configuration happens in the web server config file.
  • Easier to comprehend for most people. Appears more “standard stack” to system administrators who are not familiar with Ruby specifically.
  • Can spawn and shutdown processes dynamically according to traffic patterns.
  • Processes are automatically monitored: if they crash they are automatically restarted.
  • Less manual control over individual processes because they can come and go at any time.

Reverse proxy model as implemented by most other Ruby app servers:

  • App server is a separate entity. Processes are managed distinctly from the web server itself. Configuration happens outside the web server.
  • Many people have a hard time comprehending this and they generally find setups like this cumbersome, but to experts this model can be seen as simple, elegant and sensible.
  • Most app servers do not automatically restart crashed processes and one needs to monitor processes separately with things like Monit.
  • One needs to specify the number of processes up front: no dynamic process count scaling according to traffic.
  • Allows fine-grained manual control over individual processes.

We are not commenting on which points are supposed to be pros and which points are supposed to be cons because they are highly subjective. For us, integration into the web server is a strong plus because we host dozens of apps on our server(s) and we don’t like to spend time managing app server processes for each app, but other people are uncomfortable with having the web server manage things automatically and would prefer to keep a close eye on everything.

The automatic model can also be problematic to people who were on the reverse proxy model because they already had their web servers and infrastructures configured in a certain way. Switching to Phusion Passenger could mean changing a lot of web server configuration.

The hidden but unutilized potential

Reverse proxy model app servers can potentially have an extra advantage, but for some reason this hasn’t been implemented to its full potential so far:
Reverse proxy app servers are just easier to get started with. When you’ve just created a new Rails app, you can start it with script/server and you’re ready to go.

This works great in development but totally blows up in production. Reverse proxy model app servers must be put behind a reverse proxy e.g. Nginx or HAProxy for a variety of reasons such as security, load balancing between processes, handling of slow clients, etc. In production environments nobody exposes Mongrel or Thin directly to the Internet. Unicorn even explicitly documents that it is designed to be put behind a reverse proxy and that it doesn’t bother with slow clients at all.

In contrast, Phusion Passenger 2.x requires one to configure the web server, meaning the user must first install a web server. This is cumbersome when you’re in development and just want to get started. It is also cumbersome if you’re a newcomer and aren’t familiar with Apache or Nginx, and you just want to get your app running on your server.

Do you type script/server in development instead of creating a virtual host in the Apache or Nginx? Well you’re not the only one: we also do this until we eventually get sick of it, but there’s always a mental blockade that tells us that editing the web server configuration file is too much work to bother with.

Well, until Phusion Passenger 3 comes along.

Phusion Passenger Lite: fusion between the reverse proxy and the automatic model

In addition to Phusion Passenger for Apache and Phusion Passenger for Nginx, Phusion Passenger 3 introduces a new component to the existing lineup: Phusion Passenger Lite.

When it comes to usage, its interface is almost identical to that of Mongrel and Thin. To run your Ruby web app, just type this in the terminal and you’re ready to go:

passenger start

Closing the gap between development and production

Phusion Passenger Lite consists of an Nginx core. Nginx is known to be extremely scalable, high-performance and lightweight. You do not need to have Nginx already installed; this is automatically taken care of. You also do not need to have any Nginx experience: Nginx is hidden from the user but its power is automatically utilized.

Unlike Mongrel, Thin and Unicorn, Phusion Passenger Lite can be directly exposed to the Internet. It can serve static files at blazing speeds thanks to the Nginx core. Mongrel and Thin can serve static files but they aren’t very good at it. Unicorn doesn’t even try.

Easy migration from existing reverse proxy app servers

Because the interface is so similar, you can easily swap Mongrel, Thin or Unicorn in your existing reverse proxy setup and replace it with Phusion Passenger Lite. Unlike Mongrel and Thin, Phusion Passenger Lite only has to listen on a single socket instead of multiple, vastly simplifying your reverse proxy configurations. Phusion Passenger Lite can listen on a Unix domain socket instead of a TCP socket, just like Thin and Unicorn. In reverse proxy setups this can yield much higher performance than TCP sockets.

Advantages over existing reverse proxy app servers

Unlike Mongrel, Thin and Unicorn, Phusion Passenger Lite can dynamically spawn and shutdown processes according to traffic. However you can also configure it to use a static number of processes! In fact you can configure a minimum and a maximum and have Phusion Passenger Lite automatically figure out the number of processes to use for the current traffic.

Like Phusion Passenger for Apache/Nginx and Unicorn, worker processes that have crashed are automatically restarted.

That said, bear in mind that this advantage can be a disadvantage to some people. At its heart, Phusion Passenger Lite still manages processes for you, so you don’t have as much fine-grained control over the processes as you do with other reverse proxy app servers.

Advantage over Phusion Passenger for Apache and Phusion Passenger for Nginx

Another unintended advantage of Phusion Passenger Lite is that it runs as the same user as the shell and respects environment variables that are defined for your shell, e.g. things like PATH, LD_LIBRARY_PATH, RUBYOPT, GEM_PATH and GEM_HOME.

  • Some people find that their app cannot load a certain library when the app is started in Phusion passenger, but can when the app is started with e.g. Mongrel or Thin. This is almost always caused by some environment variable that’s set in the shell but not in the web server: everything you set in /etc/bashrc and friends don’t have effect on processes started from the web server.
  • Some people say that their application does not start on Phusion Passenger, but does under Mongrel/Thin. Very often this turns out to be just a permission issue or some web server configuration issue.

With Phusion Passenger Lite, even confusion like this will be a thing of the past.

Rethinking the word “easy”: automatic mass deployment

Phusion Passenger is considered by many people the easiest Ruby app server out there. But can it be easier? After some heavy thinking outside the box, we believe the answer is yes, it certainly can!

Imagine having a directory full of Ruby web apps, e.g. ~/Sites. To deploy an app, just drop your application’s directory into ~/Sites. To undeploy it, remove the application’s directory. The application directory’s name is used as the domain name. No manually signaling the web server for a restart.

This is exactly what we’ve done with Phusion Passenger Lite. We call this feature automatic mass deployment. Check out this demo.

So from now on, if you have a bunch of Ruby web apps in the same directory, just run this command in that directory

sudo passenger start -p 80 -u (some_unprivileged_username)

and you’ve immediately deployed every single app!

Conclusion

Phusion Passenger Lite does not replace Phusion Passenger for Apache or Phusion Passenger for Nginx. Rather, it is a complement to our existing lineup of Phusion Passenger products, optimized for different use cases. Phusion Passenger Lite closes the gap between development and production and can be used comfortably and easily in both. It can act as a drop-in replacement for your existing reverse proxy based setup. It makes Ruby web app deployment even easier than before: now you don’t even need a separate web server. On the other hand, if you need integration into the web server, then Phusion Passenger for Apache/Nginx is for you.

We hope you’ve enjoyed this Technology Preview. Please stay tuned for the next one because we have more exciting news for you!

The Road to Passenger 3: Technology Preview 2 – Stability, robustness, availability, self-healing

By Hongli Lai on June 18th, 2010

In Technology Preview 1 we’ve shown that Phusion Passenger 3 can be up to 55% faster on OS X. Performance is good and all, but it won’t do you any good unless the software keeps running. When Phusion Passenger goes down it’s an annoyance at best, but in the worst case any amount of down time can cost your organization real money. Any HTTP request that’s dropped can mean a lost transaction.

Although stability, robustness and availability aren’t as hot and fashionable as performance, for Phusion Passenger 3 we have not neglected these areas. In fact we’ve been working on hard on implementing additional safeguards, as well as refactoring our designs to make things more stable, robust and available.

Self-healing

In Phusion Passenger 2.2’s architecture, there are a number of processes that work together. At the very front there is the web server, which could consist of multiple processes. If you’ve ever typed passenger-memory-stats then you’ve seen the web server processes at work. Apache typically has a dozen of processes (prefork MPM) and Nginx typically has 3.

For Phusion Passenger to work, there must be some kind of global state, shared by all web server processes. In this global state information is stored such as which Ruby app processes exist, which ones are currently handling requests, etc. This allows Phusion Passenger to make decisions such as which Ruby app process to route a request to, whether there is any need to spawn another process, etc. This global state exists in a separate process which all web server processes communicate with. On Apache this is the ApplicationPoolServerExecutable, on Nginx this is the HelperServer. For simplicity’s sake, let’s call both of them HelperServer (in Passenger 3 they’ve both been renamed to PassengerHelperAgent). The HelperServer is written in C++ and is extremely fast and lightweight, consuming only about 500 KB of real memory.

As you can see the HelperServer is essentially the core of Phusion Passenger. The problem with 2.2 is that if the HelperServer goes down, Phusion Passenger goes down with it entirely. Phusion Passenger will stay down until the web server is restarted. For various architectural reasons in Apache and Nginx, it is not easily possible to restart the HelperServer upon a crash in a reliable way.

Now, why would the HelperServer ever crash?

  1. Bugs. We are humans too and we can make mistakes, so it’s possible that there are crasher bugs in the HelperServer. In the past 2 years we’ve spent a lot of effort into making the HelperServer stable. For example we check all system calls for error results, and we’ve spent a lot of effort into making sure that uncaught exceptions are properly logged and handled. However one can never prove that a system is entirely bug free. We aren’t aware of any crasher bugs at this time but they might still exist.
  2. Limited system resources. For example if the system is very low on memory, the kernel will invoke the Out-Of-Memory Killer (OOM Killer). Properly selecting a process to kill in low-memory conditions is actually a pretty hard problem, and more often than not the OOM Killer selects the wrong process, e.g. our HelperServer.
  3. System administrator mistakes. Passing the wrong PID to the kill command and things like that.
  4. System configuration problems, hardware problems (faulty RAM and stuff) and operating system bugs.

There are some people who have reported problems with the HelperServer. Their HelperServer crashes tend to happen sporadically and they usually cannot reproduce the problem reliably themselves. For many people (2) is often the cause of HelperServer crashes, and increasing the amount of swap is reported to help, but for other people the problems lie elsewhere.

The crashes aren’t always our fault (i.e. not bugs), but they are always our problem. It saddens us to say that we’ve been unable to help these people so far because we simply cannot reproduce their problems even when we mimic their system configuration.

But this is going to change.

Enter Phusion Passenger 3 with self-healing architecture

Phusion Passenger 3 now introduces a lightweight watchdog process into the architecture. It monitors both the web server and the HelperServer. If the HelperServer crashes, then the watchdog restarts the HelperServer immediately.

Of course if the watchdog is killed then it’s still game over, but we’ve taken extra care in terms of code to try to make this extremely unlikely to happen. The watchdog for starters is extremely lightweight, even more so than the HelperServer. It is written in C++ and uses about 150-200 KB of memory. Its only job is to start the HelperServer and other Phusion Passenger helper processes and to monitor them. The codebase extensively uses C++ idioms that promote code stability, such as smart pointers and RAII. By employing heavy testing as well, we’re expecting to have brought the possibility that the watchdog contains crashing bugs to a minimum. The small footprint and the fact that it does nothing most of the time minimizes the chances that it’s killed by the OOM Killer. In fact, if the watchdog is running on Linux and has root access, it will register itself as not OOM-killable.

No longer will HelperServer crashes take down Phusion Passenger, even if the crash isn’t our fault.

Restarts are fast

It only takes a few hundred miliseconds to restart the HelperServer.

Crashing signals are logged

If the HelperServer crashes then the watchdog will tell you whether it crashed because of a signal, e.g. SIGSEGV. This makes it much easier for system administrators to see why a component crashed so that they might fix the underlying cause. In Phusion Passenger 2.2 this was not possible.

Guaranteed cleanup

Upon shutting down or restarting the web server, Phusion Passenger 2.2 gracefully notifies application processes to shut down. It does not force them to. This would pose a problem for broken web applications that don’t shut down properly, e.g. web applications that are stuck in an infinite loop, stuck in a database call, etc.

Phusion Passenger 3 guarantees that all application processes are properly shut down when you shutdown/restart the web server. It gives application processes a deadline of 30 seconds to shutdown gracefully; if any of them fail to do that, they’ll be terminated with SIGKILL.

This mechanism works so well that it even extends to background processes that have been spawned off by the web application processes. All of those processes belong to the same process group. Phusion Passenger sends SIGKILL to the entire process group and terminates everything. No longer will you have to manually clean up processes; you can be confident that everything is gone if you shutdown/restart the web server.

Zero-downtime web server restart

In Phusion Passenger 2.2, whenever you restart the web server, HTTP requests that are currently in progress are dropped and the clients receive ugly “Connection reset by server” or similar error messages. This can be a major problem for large websites, because during the 1 second that Phusion Passenger is restarting hundreds of people could be getting errors. If your visitor happens to be clicking on that “Buy” button, well, tough luck.

In Phusion Passenger 3 we’ve implemented zero-downtime web server restart. Phusion Passenger and the web server are restarted in the background, and while this is happening, the old web server instance (with the old Phusion Passenger instance) will continue to process requests.

The architecture is actually a little bit more complicated than what’s shown in the diagram because behind the web server there are a bunch of Phusion Passenger processes, but you get the gist of it.

When the new web server (along with the new Phusion Passenger) has been started, it will immediately begin accepting new requests. Old requests that aren’t finished yet will continue to be processed by the old web server and Phusion Passenger instance. The old instance will shut down 5 seconds after all requests have been finished, to counter the possibility that the kernel still has leftover requests in the socket backlog that hit the old instance after its done processing everything already in its queue.

This works so well that we can restart the web server while running an ‘ab’ benchmark with 100 concurrent users without dropping a single request!

Zero-downtime application shutdown

Suppose that a Ruby application process has gone rogue and you want to shut it down. The most obvious way to do that is by sending it a SIGTERM or SIGKILL signal. However this would also abort whatever request it is currently processing.

In Phusion Passenger 2.2, you could also send SIGUSR1 to the process, causing it to shut down gracefully after it has processed all requests in its socket backlog. However this introduces two problems:

  • If the website is very busy then the process’s socket backlog will never be empty, and so the process will never exit.
  • Exiting after the process has detected an empty backlog can introduce a race condition. Suppose that, right after the process has determined that its backlog is empty but before it has shutdown completely, Phusion Passenger tries to send another request to the process. This request would be lost.

In Phusion Passenger 3, SIGUSR1 will now cause the application process to first unregister itself so that Phusion Passenger won’t route any new requests to it anymore. It will then proceed with exiting 5 seconds after its socket backlog has become empty. This way you can gracefully shutdown a process without losing a single request.

Conclusion

Although Phusion Passenger has been powering many high-traffic Ruby websites for a while now, some people still have some doubts about whether Phusion Passenger is fit for production. Instead of using words convince them, we would rather convince them with real results. Phusion Passenger 3 raises the bar in the areas of performance, stability, robustness and availability yet higher, but it doesn’t stop here. Please stay tuned for the next Technology Preview in which will unveil even more of Phusion Passenger 3.

The Road to Passenger 3: Technology Preview 1 – Performance

By Hongli Lai on June 10th, 2010

It has already been two years since we’ve first released Phusion Passenger. Time sure flies and we’ve come a long way since then. We were the first to implement a working Ruby web app deployment solution that integrates seamlessly in the web server, and all the features that we’ve developed over time – smart spawning and memory reduction, upload buffering, Nginx support, etc – have served us for a long time. Nevertheless, it is time to say goodbye to the old Phusion Passenger 2.2 codebase. In the past we had focused primarily on three things:

  • Ease of use.
  • Stability.
  • Robustness.

Notice that “performance” is not on the above list. We strived to make Phusion Passenger “fast enough”, e.g. not ridiculously slower than the alternatives. Lately it would appear that competitors are once again focusing on performance. We can of course not afford to stay behind. We’ve been working on Phusion Passenger 3 for a while now. Today we will begin unveiling the technology behind this new major Phusion Passenger version. This blog post is the first of the multiple technology previews to come.
Read on…