Phusion white papers Phusion overview

Phusion Blog

Docker-friendly Vagrant boxes

By Hongli Lai on November 8th, 2013

Vagrant

We heavily utilize Vagrant in our development workflow. Vagrant is a tool for easily setting up virtual machines as development environments, making it easy to distribute development environments and making them reconstructible and resetable. It has proven to be an indispensable tool when working in development teams with more than 1 person, especially when not everybody uses the same operating system.

Lately we’ve been working with Docker, which is a cool new OS-level virtualization technology. Docker officially describes it as “iPhone apps for your server”, but being the hardcore system-level guys that we are, we dislike this description. Instead we’d like to describe Docker as “FreeBSD jails for Linux + an ecosystem to make it a joy to use”. Docker, while still young and not production-ready, is very promising and can make virtualization cheap and efficient.

Googling for Vagrant and Docker will yield plenty of information and tutorials.

Today, we are releasing Docker-friendly Vagrant boxes based on Ubuntu 12.04. Docker requires at least kernel 3.8, but all the Ubuntu 12.04 Vagrant boxes that we’ve encountered so far come with kernel 3.2 or 3.5, so that installing Docker on them requires a reboot. This makes provisioning a VM to be significantly more painful than it should be.

Update April 23 2014: we now also provide Vagrant boxes based on Ubuntu 14.04. The 12.04 boxes are still available. See the website for more information.

These boxes differ from the ones provided by vagrantup.com in the following ways as well:

  • We provide 2 virtual CPUs by default, so that the boxes can make better use of multicore hosts.
  • We provide more RAM by default: 1 GB.
  • We provide a bigger virtual hard disk: around 40 GB. So that you don’t have to worry about running out of disk space inside your VM.
  • We use LVM so that partitioning is easier.
  • The memory cgroup and swap accounting are turned on, for some Docker features.
  • Chef is installed via the Ubuntu packages that they provide, instead of via RubyGems. This way the box doesn’t have to come with Ruby by default, making the environment cleaner.
  • Our VMWare Fusion boxes recompile VMWare Tools on every kernel upgrade, so that Shared Folders keep working even if you change the kernel.

These base boxes are automatically built from Veewee definitions. These definitions makes building boxes quick and unambigious. The entire building process is described in the definitions; no manual intervention is required.

We provide prebuilt boxes at https://oss-binaries.phusionpassenger.com/vagrant/boxes/, but you can build them yourself if you so wish.

The boxes are also available on Vagrant Cloud.

Related resources: Github | Prebuilt boxes | Vagrant Cloud | Discussion forum | Twitter

Using these boxes in Vagrant

If you have Vagrant 1.5, you can use our boxes through Vagrant Cloud:

vagrant init phusion/ubuntu-14.04-amd64
# -OR-
vagrant init phusion/ubuntu-12.04-amd64

On older Vagrant versions, you can modify your Vagrantfile to use our boxes. Here is an example Vagrantfile which works with both VirtualBox and VMWare Fusion. It also automatically installs the latest version of Docker.

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "phusion-open-ubuntu-14.04-amd64"
  config.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vbox.box"
  # Or, for Ubuntu 12.04:
  #config.vm.box = "phusion-open-ubuntu-12.04-amd64"
  #config.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-12.04-amd64-vbox.box"

  config.vm.provider :vmware_fusion do |f, override|
    override.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-14.04-amd64-vmwarefusion.box"
    #override.vm.box_url = "https://oss-binaries.phusionpassenger.com/vagrant/boxes/latest/ubuntu-12.04-amd64-vmwarefusion.box"
  end

  if Dir.glob("#{File.dirname(__FILE__)}/.vagrant/machines/default/*/id").empty?
    # Install Docker
    pkg_cmd = "wget -q -O - https://get.docker.io/gpg | apt-key add -;" \
      "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list;" \
      "apt-get update -qq; apt-get install -q -y --force-yes lxc-docker; "
    # Add vagrant user to the docker group
    pkg_cmd << "usermod -a -G docker vagrant; "
    config.vm.provision :shell, :inline => pkg_cmd
  end
end

You can login with username vagrant and password vagrant. This user has sudo privileges. The root user also has password vagrant.

The prebuilt boxes are available at https://oss-binaries.phusionpassenger.com/vagrant/boxes/.

Final words

Enjoy these Vagrant boxes!

You may also want to check our other products, such as Phusion Passenger, which is an application server which makes Ruby, Python, Node.js and Meteor deployment extremely simple. Or simply follow us on Twitter.

Use Nginx + SPDY, without compiling Nginx and without a recent OpenSSL

By Hongli Lai on August 21st, 2013

google-spdy

We use the SPDY network protocol extensively to improve the performance of our websites. SPDY — pronounced “speedy” — is a new-ish protocol from Google with the goal of reducing latency, improving throughput and improving pipelining. Many articles have been written about the advantages of SPDY. We have observed 20%-30% better loading time on www.phusionpassenger.com by switching from plain HTTP to SPDY, mostly because of the better pipelining that SPDY offers over plain HTTP.

SPDY is built on top of TLS. Nginx has supported SPDY through external patches for a while. Since version 1.4.0, Nginx has SPDY support builtin, with two caveats:

  1. SPDY support must be enabled by compiling Nginx with --with-http_spdy_module.
  2. It requires OpenSSL 1.0.1+, because SPDY requires the Next Protocol Negotiation TLS extension.

Many users prefer to use the Nginx binary provided by their distribution. But not all of the currently widely used distributions provide OpenSSL 1.0.1, and of those that do, very few of them have Nginx with SPDY enabled.

We started providing prebuilt Nginx binaries since Phusion Passenger 4.0.13 (learn more at “No more compiling Phusion Passenger”). These Nginx binaries not only have Phusion Passenger support enabled, but also SPDY support! Furthermore, we’ve spent great effort on ensuring that these binaries are compatible with a wide range of Linux distributions, whether they’re running on x86 or x86_64. Best of all: you can use these Nginx binaries without Phusion Passenger, and as a drop-in replacement for your distribution’s Nginx binary! This means:

  • You install Nginx using your distribution’s preferred method (e.g. apt-get install nginx).
  • You overwrite the Nginx binary with the one that we provide.
  • You get to keep all the nice things that your distribution package offers, such as init scripts, conf.d directories, etc.
  • No compilation is necessary.

Getting started on Debian or Ubuntu

This guide is taylored for Debian and Ubuntu. The instructions may also work on other distributions, but the paths may be different, and the init script format may also be different. You can use this guide as a starting point for figuring out how to achieve the same for your specific distribution.

Install Nginx using apt:

sudo apt-get install nginx

Next, download our Nginx binary. There are multiple versions of Nginx and of Phusion Passenger. You can find all available versions at the Phusion download server, indexed by Phusion Passenger version. At the time of writing, Nginx 1.4.2 and Phusion Passenger 4.0.13 are the most recent versions:

# 32-bit systems
curl -O https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/4.0.13/nginx-1.4.2-x86-linux.tar.gz
# 64-bit systems
curl -O https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/4.0.13/nginx-1.4.2-x86_64-linux.tar.gz

Extract the downloaded tarball:

tar xzvf nginx-*.tar.gz

Update April 9 2014: the tarball and the binary have been renamed. It’s called “webhelper” now. So if you want Nginx 1.4.7 from Phusion Passenger 4.0.41, run:

# 32-bit systems
curl -O https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/4.0.41/webhelper-1.4.2-x86-linux.tar.gz
# 64-bit systems
curl -O https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/4.0.41/webhelper-1.4.2-x86_64-linux.tar.gz

# Then rename the tarball and the binary:
tar xzvf webhelper-*.tar.gz
mv PassengerWebHelper nginx

The next steps are a little more complicated, although not difficult. The Nginx binary that we provide is compiled with the prefix /tmp. This is because Nginx requires several data directories (e.g. client_body_temp_path) to properly operate. Since our Nginx binary is designed to be portable, we can’t assume any specific directory structure, which is why we use the /tmp prefix.

Luckily, there is a way to tell the Nginx binary during runtime to a different directory structure, and that’s exactly what we’re going to do.

Overwrite the original Nginx binary and create a bunch of symlinks:

sudo cp nginx /usr/sbin/
sudo ln -s /etc/nginx /var/lib/nginx/conf
sudo ln -s /var/log/nginx /var/lib/nginx/logs

Then edit /etc/default/nginx and add:

DAEMON_OPTS="$DAEMON_OPTS -p /var/lib/nginx"

Next, edit /etc/nginx/nginx.conf, and set the following options:

pid /var/run/nginx.pid;
lock_file /var/lock/nginx.lock;

Finally, restart Nginx using your distribution’s Nginx init script:

sudo /etc/init.d/nginx restart

Testing SPDY

To test SPDY, you need an SSL certificate for your domain name. There are many cheap SSL certificate providers our there, which you can easily find through Google. Once you have an SSL certificate, create a virtual host entry:

sudo tee /etc/nginx/conf.d/spdy_test.conf <<EOF
server {
    listen 443 ssl spdy;
    server_name your_domain_name.com;
    ssl on;
    ssl_certificate /path-to-your-cert.crt;
    ssl_certificate_key /path-to-your-key.key;
    root /tmp/spdy_test;
}
EOF

Then create a web directory with a test document:

mkdir /tmp/spdy_test
echo it works > /tmp/spdy_test/index.html

Restart Nginx:

sudo /etc/init.d/nginx restart

Finally, use SPDYCheck to check your website at https://your_domain_name.com.

Distribution updates

Whenever the distribution has an update for Nginx, you must replace the Nginx binary after the distribution’s update tool has installed the update. For example, suppose that Ubuntu releases Nginx 1.4.3 tomorrow:

$ sudo apt-get upgrade
[some error messages will appear during restarting of Nginx]

apt-get upgrade will probably fail to restart Nginx, but this is normal! This is because you will probably have SPDY-specific configuration options, but the distribution’s Nginx doesn’t support that.

Ignore the error, and download the latest version of the Phusion Nginx binary from the Phusion download server:

curl -O https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/<SOME VERSION>/nginx-1.4.3-<SOME ARCHITECTURE>-linux.tar.gz

Next, extract the Nginx binary and overwrite the distribution’s binary:

tar xzvf nginx-*.tar.gz
sudo cp nginx /usr/sbin/

Finally, finalize the apt-get upgrade and restart Nginx:

sudo apt-get upgrade
sudo /etc/init.d/nginx restart

What about security?

Downloading random binaries from the Internet is very dangerous. If an attacker intercepts and modifies the communication channel, anything goes. To combat this problem, we’ve employed two security measures:

  • All our binaries are hosted on HTTPS.
  • All our binaries are signed with PGP. The PGP key is Phusion Automated Software Signing (software-signing@phusion.nl), fingerprint 1637 8A33 A6EF 1676 2922 526E 561F 9B9C AC40 B2F7.

Reinstalling Nginx if something goes wrong

If our binary doesn’t work for some reason, then reverting to the original Nginx binary is easy:

sudo apt-get remove nginx
sudo apt-get install nginx

Conclusion

Installing Nginx with SPDY support through our prebuilt binaries is quite easy and requires just a few config file changes. We’ve love to know whether it works well for you. Please leave feedback at the comment form below. Thank you for reading.

SHA-3 extensions for Ruby and Node.js

By Hongli Lai on October 6th, 2012

A few days ago, NIST announced the winner of the SHA-3 competition: Keccak (prounced [kɛtʃak], ketchak). The researchers who authored Keccak released a reference implementation in C.

We’ve created Ruby and Node.js extensions for Keccak. Our extensions utilize the code from the official reference implementation and come with a extensive suite of unit tests. But note however that I do not claim to be a security expert. Feel free to review the code for any flaws.

Install with:

gem install digest-sha3
npm install sha3

We’ve strived to emulate the languages’ standard hash libraries’ interfaces, so using these extensions is straightforward:

require 'digest/sha3'
Digest::SHA3.hexdigest("foo")

and

var SHA3 = require('sha3');
var hash = SHA3.SHA3Hash();
hash.update('foo');
hash.digest();

Both libraries are MIT licensed. Enjoy!

Why does the world need SHA-3?

If you’re not a security researcher then you’ve undoubtedly asked the same questions as I did. What’s wrong with SHA-1, SHA-256 and SHA-512? Why does the world need SHA-3? Why was Keccak the winner?

According to to well-known security researcher Bruce Schneier, there’s nothing wrong with the SHA-2 family of hashing functions. However he likes SHA-3 because it’s completely different. SHA-1 and SHA-2 are both based on the Merkle–Damgård construction. It may be feasible to find a flaw in SHA-1 that also affects SHA-2. In contrast, Keccak is based on the sponge construction so any attacks on SHA-1 and SHA-2 will probably not affect Keccak. Indeed, it appears that NIST chose Keccak because it was looking for some kind of insurance in case SHA-2 would be broken. Many people commented that they expected Skein to win.

Do not hash your passwords

In any case, the following cannot be repeated enough. Do not use SHA-3 (or SHA-256, SHA-512, RIPEMD-160 or whatever fast hash) to hash passwords! Do not even use SHA-3 + salt to hash passwords. Instead, use a slow hash like bcrypt. As Coda Hale explained in his article, you’ll want the hash to be slow so you can defend against attackers effectively.

Tutorial: setting up Gitlab on Debian 6

By Hongli Lai on April 21st, 2012

We host many git repositories on our servers using SSH and file system ACLs for access control. However, having been spoiled by Github for so long, this method feels archaic and cumbersome. While Github provides private repositories, sometimes it’s just not an option because there are some things that may never leave the organization. If you still want to have a fancy web interface for your Git repositories, then there are several alternatives:

Gitlab is an excellent option. It’s not as fully featured as Github, but like Gitorious it is open source. We’ve found that it’s not only more user friendly than Gitorious but also easier to install.

This tutorial teaches you how to setup Gitlab on Debian 6, Ruby 1.9.3, Phusion Passenger and Nginx. It assumes that Gitlab and the git repositories are hosted on the same machine. Your Gitlab installation will be protected by SSL. Your users will be able to pull from and push to your repositories using the ssh:// protocol, but they won’t have actual shell access and you don’t need to have separate system accounts for each user to control access. Gitlab (or to be more correct, gitolite, which Gitlab uses) manages access without system accounts.

Step 1: Setup sudo

In this tutorial we’re going to use sudo to switch between user accounts. Sudo is not installed by default on Debian so install it if you don’t already have it:

$ su
# apt-get update
# apt-get install sudo

Now add your own account to /etc/sudoers:

# visudo

Add something like this to the end of the file:

your_username_here   ALL=(ALL) ALL

Step 2: Install the base software

Install basic dependencies:

$ sudo apt-get update
$ sudo apt-get install build-essential git-core wget curl gcc checkinstall libxml2-dev libxslt-dev sqlite3 libsqlite3-dev libcurl4-openssl-dev libreadline-dev libc6-dev libssl-dev libmysql++-dev make build-essential zlib1g-dev libicu-dev redis-server openssh-server python-dev python-pip libyaml-dev

Install Pygments, which Gitlab needs for syntax highlighting:

$ sudo pip install pygments

Gitlab needs the sendmail command in order to send emails (for things like lost password recovery). This command is provided by the exim4, postfix and sendmail packages but you can only have one of them installed. If you don’t already have one of them installed, then we recommend postfix.

First, check whether you already have the sendmail command:

$ ls /usr/sbin/sendmail

If you get a ‘file not found’ then install Postfix:

$ sudo apt-get install postfix

Step 3: Install gitolite

We need gitolite. Gitolite is a tool used by Gitlab for managing access control to git repositories. It works by providing a bunch of config files in which you can register users and their public keys. Gitolite modifies ~/.ssh/authorized_keys appropriately based on the config files’ contents. Gitolite is not supposed to run as root; instead, it runs as a single user.

Install gitolite:

$ sudo apt-get install gitolite

The Debian package will automatically create an account called gitolite. Gitlab controls gitolite by logging into gitolite’s user account through SSH, so we want to create a passwordless SSH keypair for this. This SSH keypair is what we call the gitolite admin key.

$ sudo -u gitolite ssh-keygen
...
Enter file in which to save the key (/var/lib/gitolite/.ssh/id_rsa):    <--- enter nothing here and press enter
...
Enter passphrase (empty for no passphrase):    <--- enter nothing here and press enter
Enter same passphrase again:                   <--  enter nothing here and press enter
...
Your public key has been saved in /var/lib/gitolite/.ssh/id_rsa.pub.
...

Now you need to tell gitolite that you want to use this key as the admin key. First, let’s print the content of the public key and copy it to the clipboard:

$ sudo -u gitolite cat /var/lib/gitolite/.ssh/id_rsa.pub
(now select the output in your terminal and copy it to the clipboard)

Now let’s configure gitolite:

$ sudo dpkg-reconfigure gitolite

When asked for a username, keep it at the default:

When asked for a repository path, keep it at the default:

When asked for the admin key, paste the contents of the key you copied to clipboard:

Finally, we need to set the REPO_MASK option to 0007.

$ sudo -u gitolite vi /var/lib/gitolite/.gitolite.rc

Look for:

$REPO_UMASK = 0077;         # gets you 'rwx------'

Change it to:

$REPO_UMASK = 0007;  # rwxrwx---

Tighten security

We think the default gitolite home directory security is a bit weak: everybody on the system can access the config files and the repository files.

$ ls -ld /var/lib/gitolite
drwxr-xr-x 5 gitolite gitolite 4096 Apr 21 08:40 /var/lib/gitolite

Let’s tighten it up a little bit so that only gitolite can access the files:

$ sudo -u gitolite chmod o-rx /var/lib/gitolite

Step 4: Install Ruby 1.9

Gitlab requires Ruby 1.9 and their developers recommend Ruby 1.9.2, but we don’t recommend 1.9.2 because it has a lot of critical bugs compared to 1.9.3. We’ve been running Gitlab in production on Ruby 1.9.3 for a while now and so far everything works great, so in this tutorial we’ll teach you how to install Ruby 1.9.3. But feel free to install a different version if you disagree with us.

Debian does not provide a recent enough version of Ruby through apt, so we need to install it manually.

$ wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p194.tar.gz
$ tar xzvf ruby-1.9.3-p194.tar.gz
$ cd ruby-1.9.3-p194
$ ./configure
$ make
$ sudo make install

Install Bundler:

$ sudo gem install bundler

Step 5: Install Gitlab

Download the source

$ cd /opt
$ sudo git clone git://github.com/gitlabhq/gitlabhq.git
$ sudo chown -R gitolite:gitolite gitlabhq
$ cd gitlabhq
$ sudo -u gitolite git checkout 9af14e4bda35

(You can also checkout the stable branch for the latest stable version, but this tutorial is written specifically for commit 9af14e4bda35.)

Configure

Setup the database configuration. Make sure you fill in the database details under the production section.

$ sudo -u gitolite cp config/database.yml.example config/database.yml
$ sudo -u gitolite vi config/database.yml

Setup other Gitlab configuration.

$ sudo -u gitolite cp config/gitlab.yml.example config/gitlab.yml
$ sudo -u gitolite vi config/gitlab.yml
  • Set email.from to what should be filled in the ‘From’ header in emails.
  • Set email.to to the domain name on which you want to host Gitlab, without the protocol scheme and without the path.
  • Set email.protocol to https.
  • Set git_host.admin_uri to gitolite@localhost:gitolite-admin.
  • Set git_host.base_path to /var/lib/gitolite/repositories/.
  • Set git_host.host to the system’s SSH domain name.
  • Set git_host.git_user to gitolite.

Now tighten up security:

$ sudo -u gitolite chmod o-rwx config/*.yml

Install gems and setup database

$ sudo -u gitolite -H bundle install --without development test --deployment
$ sudo -u gitolite bundle exec rake db:setup RAILS_ENV=production
$ sudo -u gitolite bundle exec rake db:seed_fu RAILS_ENV=production

This last command will output an initial administrator account’s username and password. Write it down somewhere.

Check status

$ sudo -u gitolite bundle exec rake gitlab:app:status RAILS_ENV=production

You should get all YES:

Starting diagnostic
config/database.yml............exists
config/gitlab.yml............exists
/home/git/repositories/............exists
/home/git/repositories/ is writable?............YES
remote: Counting objects: 603, done.
remote: Compressing objects: 100% (466/466), done.
remote: Total 603 (delta 174), reused 0 (delta 0)
Receiving objects: 100% (603/603), 53.29 KiB, done.
Resolving deltas: 100% (174/174), done.
Can clone gitolite-admin?............YES
UMASK for .gitolite.rc is 0007? ............YES

Run a Resque worker

This worker daemon is for processing background tasks.

$ sudo -u gitolite bundle exec rake environment resque:work QUEUE=* RAILS_ENV=production BACKGROUND=yes

Step 6: Generate an SSL certificate

Generate a self-signed certificate. You may enter arbitrary certificate information but the Common Name must be set to the domain name on which you want to host Gitlab.

$ cd /opt/nginx/conf
$ sudo openssl req -new -x509 -nodes -days 3560 -out gitlab.crt -keyout gitlab.key
$ sudo chmod o-r gitlab.key

Installing the certificate into your browser

This self-signed certificate by itself provides encryption, but not authentication, making it vulnerable to man-in-the-middle attacks. To solve this problem, install this certificate into your browser.

Here’s how you do it on Google Chrome on OS X. First, copy the certificate to your local machine:

$ scp your-server.com:/opt/nginx/conf/gitlab.crt .

Now open Keychain Access. Under the “Keychains” list, select “System”. Then click on the “+” button and add the certificate file.

Step 7: Deploy to Phusion Passenger and Nginx

Install Nginx and Phusion Passenger:

$ sudo gem install passenger --no-rdoc --no-ri
$ sudo passenger-install-nginx-module
(Choose option 1 and install Nginx to /opt/nginx)

Now edit the Nginx config file

$ sudo vi /opt/nginx/conf/nginx.conf

and add this to the http block:

# This is a normal HTTP host which redirects all traffic to the HTTPS host.
server {
    listen 80;
    server_name gitlab.yourdomain.com;
    root /nowhere;
    rewrite ^ https://gitlab.phusion.nl$request_uri permanent;
}

# The actual Gitlab HTTPS host.
server {
    listen 443;
    server_name gitlab.yourdomain.com;
    root /opt/gitlabhq/public;
    ssl on;
    ssl_certificate gitlab.crt;
    ssl_certificate_key gitlab.key;
    add_header Strict-Transport-Security "max-age=315360000";
    location / {
        passenger_enabled on;
    }
    location /assets {
        expires max;
        add_header Cache-Control public;
        passenger_enabled on;
    }
}

Now start Nginx:

$ sudo /opt/nginx/sbin/nginx

And viola, you’re up and running! You can access Gitlab through https://gitlab.yourdomain.com/.

Already have Phusion Passenger installed and already running Ruby 1.8?

Phusion Passenger for Nginx does not yet support multiple Ruby versions (but it will in 3.2). If you already have Ruby apps deployed on your server using Phusion Passenger, and those apps require Ruby 1.8, then you’ll have to run Gitlab on Ruby 1.9 using Phusion Passenger Standalone, and connect it to Nginx through a reverse proxy setup.

$ cd /opt/gitlabhq
$ sudo -u gitolite passenger start -d -e production --max-pool-size 2 -S gitlab.socket

The HTTPS server block should then look like this:

upstream gitlab {
    server unix:/opt/gitlabhq/gitlab.socket;
}

# The actual Gitlab HTTPS host.
server {
    listen 443;
    server_name gitlab.yourdomain.com;
    root /opt/gitlabhq/public;
    ssl on;
    ssl_certificate gitlab.crt;
    ssl_certificate_key gitlab.key;
    add_header Strict-Transport-Security "max-age=315360000";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Ssl on;
    location / {
        if (!-f $request_filename) {
            proxy_pass http://gitlab;
            break;
        }
    }
    location /assets {
        expires max;
        add_header Cache-Control public;
        if (!-f $request_filename) {
            proxy_pass http://gitlab;
            break;
        }
    }
}

Donate

Gitlab started a donation campaign a few weeks ago. It is excellent, high-quality open source software so if you like it, consider giving them a donation to support their development.

Don’t be perfectionist: big code dumps suck

By Hongli Lai on April 13th, 2012

description

We’ve been developing Phusion Passenger 3.2 on local repositories for a while now. We didn’t want to show the changes to the world until we know that it’s at least somewhat ready for public consumption. For a long time 3.2 didn’t even fully compile. The end results speak for themselves: Phusion Passenger 3.2 introduces quite a large number of changes. And yet there’s something that does not feel quite right here. Not only was there a large time gap between 3.0 and 3.2, we also feel that we haven’t properly communicated our development progress to the public, which could possibly have resulted in the perception that development was slow.

We have been too perfectionist. Dr Nic was right in booing our development method.

From an open source perspective as well, development behind closed doors is not the way to go. We plan on publishing changes in a more gradual manner in the future.

default_value_for 1.0.5 released: default attribute values for ActiveRecord models

By Hongli Lai on August 10th, 2011

We’ve just released default_value_for version 1.0.5. default_value_for is a Rails plugin for supporting default values in ActiveRecord models. Like this:

class User < ActiveRecord::Base
  default_value_for :age, 10
end

User.new.age   # => 10

You can read more about this plugin at Github.

Version 1.0.5 fixes support for Rails 3.0 and Rails 3.1. Unfortunately we had to remove one feature due to changes in ActiveRecord 3.1. It is no longer possible to access associations in the default_value_for_block. The following is no longer possible:

class User < ActiveRecord::Base
  belongs_to :organization
  
  default_value_for :name do |model|
    model.name = "Drone for #{model.organization.name}"
  end
end

evil_organization.name  # => "Evil Organization"
user = evil_organization.users.create
user.name               # => "Drone for Evil Organization"??

One would expect the last expression to evaluate to “Drone for Evil Organization”. On Rails 2 and Rails 3.0, it does, but on Rails 3.1 it does not. Because of the way the ‘organization’ attribute is assigned to the model by ActiveRecord, we are no longer able to support this behavior. Inside the default_value_for block, model.organization would evaluate to nil.

Launching unofficial, automatically updated Github mirror for the Nginx SVN repository

By Hongli Lai on August 9th, 2011

I probably don’t need to tell you what a great web server Nginx is. It’s lightweight, it’s fast, it’s scalable, and many people like it for its configuration file syntax yet flexible features. It’s no surprise that Nginx is doing a great job serving as a core for Phusion Passenger Standalone (our Ruby/Rails web application server which supports Apache and Nginx).

Nginx’s development has traditionally been behind closed doors. Nginx is written for the most part by one brilliant man, Igor Sysoev, who accepts patches on the Nginx mailing list. But for a long time there was no source repository with which contributors and interested people can track the development process, and no official bug tracker. All of that have changed in 2011. Igor has established Nginx as a company. The Nginx SVN repository is now open to the public and a few days ago they even opened a Trac. In short: a lot of great news lately.

Announcing the Github Nginx mirror

These days a lot of people have switched from Subversion to Git. When you’ve worked with Git for a while, Subversion probably feels archaic and unproductive. The “killer app” for Git is Github, which provides an unbeatable collaboration experience. Indeed, many projects that have switched to Github reported a dramatic increase in contributions!

In order to stimulate Nginx development, we’ve launched a Github mirror of the Nginx SVN repository:

A few notes about this mirror:

  • It’s automatically updated twice a day.
  • It’s read-only. We don’t accept pull requests. Changes should be sent to the Nginx mailing list in the form of patches.
  • We intentionally don’t mirror all the branches and tags, only the most recent ones, because we don’t believe the old branches and tags are useful to anybody.
  • Please feel free to contact us if you have an issue with our mirror.

Happy developing!

Announcing EncryptedCookieStore plugin for Rails 2.3

By Hongli Lai on April 13th, 2010

EncryptedCookieStore is similar to Ruby on Rails’s CookieStore (it saves session data in a cookie), but it uses encryption so that people can’t read what’s in the session data. This makes it possible to store sensitive data in the session.

EncryptedCookieStore is written for Rails 2.3. Other versions of Rails have not been tested.

Note: This is not ThinkRelevance’s EncryptedCookieStore. In the Rails 2.0 days they wrote an EncryptedCookieStore, but it seems their repository had gone defunct and their source code lost. This EncryptedCookieStore is written from scratch by Phusion.

Source code at http://github.com/FooBarWidget/encrypted_cookie_store

Installation and usage

First, install it:

./script/plugin install git://github.com/FooBarWidget/encrypted_cookie_store.git

Then edit config/initializers/session_store.rb and set your session store to EncryptedCookieStore:

ActionController::Base.session_store = EncryptedCookieStore

You need to set a few session options before EncryptedCookieStore is usable. You must set all options that CookieStore needs, plus an encryption key that EncryptedCookieStore needs. In session_store.rb:

ActionController::Base.session = {
        # CookieStore options...
        :key            => '_session',     # Name of the cookie which contains the session data.
        :secret         => 'b4589cc9...',  # A secret string used to generate the checksum for
                                           # the session data. Must be longer than 64 characters
                                           # and be completely random.

        # EncryptedCookieStore options...
        :encryption_key => 'c306779f3...', # The encryption key. See below for notes.
}

The encryption key must be a hexadecimal string of exactly 32 bytes. It should be entirely random, because otherwise it can make the encryption weak.

You can generate a new encryption key by running rake secret:encryption_key. This command will output a random encryption key that you can then copy and paste into your environment.rb.

Operational details

Upon generating cookie data, EncryptedCookieStore generates a new, random initialization vector for encrypting the session data. This initialization vector is then encrypted with 128-bit AES in ECB mode. The session data is first protected with an HMAC to prevent tampering. The session data, along with the HMAC, are then encrypted using 256-bit AES in CFB mode with the generated initialization vector. This encrypted session data + HMAC are then stored, along with the encrypted initialization vector, into the cookie.

Upon unmarshalling the cookie data, EncryptedCookieStore decrypts the encrypted initialization vector and use that to decrypt the encrypted session data + HMAC. The decrypted session data is then verified against the HMAC.

The reason why HMAC verification occurs after decryption instead of before decryption is because we want to be able to detect changes to the encryption key and changes to the HMAC secret key, as well as migrations from CookieStore. Verifying after decryption allows us to automatically invalidate such old session cookies.

EncryptedCookieStore is quite fast: it is able to marshal and unmarshal a simple session object 5000 times in 8.7 seconds on a MacBook Pro with a 2.4 Ghz Intel Core 2 Duo (in battery mode). This is about 0.174 ms per marshal+unmarshal action. See rake benchmark in the EncryptedCookieStore sources for details.

EncryptedCookieStore vs other session stores

EncryptedCookieStore inherits all the benefits of CookieStore:

  • It works out of the box without the need to setup a seperate data store (e.g. database table, daemon, etc).
  • It does not require any maintenance. Old, stale sessions do not need to be manually cleaned up, as is the case with PStore and ActiveRecordStore.
  • Compared to MemCacheStore, EncryptedCookieStore can “hold” an infinite number of sessions at any time.
  • It can be scaled across multiple servers without any additional setup.
  • It is fast.
  • It is more secure than CookieStore because it allows you to store sensitive data in the session.

There are of course drawbacks as well:

  • It is prone to session replay attacks. These kind of attacks are explained in the Ruby on Rails Security Guide. Therefore you should never store anything along the lines of is_admin in the session.
  • You can store at most a little less than 4 KB of data in the session because that’s the size limit of a cookie. “A little less” because EncryptedCookieStore also stores a small amount of bookkeeping data in the cookie.
  • Although encryption makes it more secure than CookieStore, there’s still a chance that a bug in EncryptedCookieStore renders it insecure. We welcome everyone to audit this code. There’s also a chance that weaknesses in AES are found in the near future which render it insecure. If you are storing *really* sensitive information in the session, e.g. social security numbers, or plans for world domination, then you should consider using ActiveRecordStore or some other server-side store.

JRuby: Illegal Key Size error

If you get this error (and your code works with MRI)…

    Illegal key size
    
    [...]/vendor/plugins/encrypted_cookie_store/lib/encrypted_cookie_store.rb:62:in `marshal'

…then it probably means you don’t have the “unlimited strength” policy files installed for your JVM. Download and install them. You probably have the “strong” version if they are already there.

As a workaround, you can change the cipher type from 256-bit AES to 128-bit by
inserting the following in config/initializer/session_store.rb:

EncryptedCookieStore.data_cipher_type = 'aes-128-cfb'.freeze  # was 256

Please note that after changing to 128-bit AES, EncryptedCookieStore still requires a 32 bytes hexadecimal encryption key, although only half of the key is actually used.

Securely store passwords with bcrypt-ruby; now compatible with JRuby and Ruby 1.9

By Hongli Lai on August 13th, 2009

When writing web applications, or any application for that manner, any passwords should be stored securely. As a rule of thumb, one should never store passwords as clear text in the database for the following reasons:

  • If the database ever gets leaked out, then all accounts are compromised until every single user resets his password. Imagine that you’re an MMORPG developer; leaking out the database with clear text passwords allows the attacker to delete every player’s characters.
  • Many people use the same password for multiple sites. Imagine that the password stored in your database is also used for the user’s online banking account. Even if the database does not get leaked out, the password is still visible to the system administrator; this can be a privacy breach.

There are several “obvious” alternatives, which aren’t quite secure enough:

Storing passwords as MD5/SHA1/$FAVORITE_ALGORITHM hashes
These days MD5 can be brute-force cracked with relatively little effort. SHA1, SHA2 and other algorithms are harder to brute-force, but the attacker can still crack these hashes by using rainbow tables: precomputed tables of hashes with which the attacker can look up the input for a hash with relative ease. This rainbow table does not have to be very large: it just has to contain words from the dictionary, because many people use dictionary words as passwords.

Using plain hashes also makes it possible for an attacker to determine whether two users have the same password.

Encrypting the password
This is not a good idea because if the attacker was able to steal the database, then there’s a possibility that he’s able to steal the key file as well. Plus, the system administrator is able to read everybody’s passwords, unless he’s restricted access to either the key file or the database.

The solution is to store passwords as salted hashes. One calculates a salted hash as follows:

salted_hash = hashing_algorithm(salt + cleartext_password)

Here, salt is a random string. After calculating the salted hash, one should store the salted hash in the database, along with the (cleartext) salt. It is not necessary to keep the salt secret or to obfuscate it.

When a user logs in, one can verify his password by re-computing the salted hash and comparing it with the salted hash in the database:

salted_hash = hashing_algorithm(salt_from_database + user_provided_password)
if (salted_hash == salted_hash_from_database):
    user is logged in
else:
    password incorrect

The usage of the salt forces the attacker to either brute-force the hash or to use a ridiculously large rainbow table. In case of the latter, the sheer size of the required rainbow table can make it unpractical to generate. The larger the salt, the more difficult it becomes for the cracker to use rainbow tables.

However, even with salting, one should still not use SHA1, SHA2, Whirlpool or most other hashing algorithms because these algorithms are designed to be fast. Although brute forcing SHA2 and Whirlpool is hard, it’s still possible given sufficient resources. Instead, one should pick a hashing algorithm that’s designed to be slow so that brute forcing becomes unfeasible. Bcrypt is such a slow hashing algorithm. A speed comparison on a MacBook Pro with 2 Ghz Intel Core 2 Duo:

  • SHA-1: 118600 hashes per second.
  • Bcrypt (with cost = 10): 7.7 hashes per second.

Theoretically it would take 4*10^35 years for a single MacBook Pro core to crack an SHA-1 hash, assuming that the attacker does not harness any weaknesses in SHA-1. To crack a bcrypt hash one would need 6*10^39 years, or 10000 more times. Therefore, we recommend the use of bcrypt to store passwords securely.

There’s even a nice Ruby implementation of this algorithm: bcrypt-ruby! Up until recently, bcrypt-ruby was only available for MRI (“Matz Ruby Interpreter”, the C implementation that most people use). However, we’ve made it compatible with JRuby! The code can be found in our fork at Github. The current version also has issues with Ruby 1.9, which we’ve fixed as well. The author of bcrypt-ruby has already accepted our changes and will soon release a new version with JRuby and Ruby 1.9 support.

Further recommended reading

How to Safely Store a Password by Coda Hale.

default_value_for Rails plugin: declaratively define default values for ActiveRecord models

By Hongli Lai on October 3rd, 2008

Introduction

The default_value_for plugin allows one to define default values for ActiveRecord models in a declarative manner. For example:

class User < ActiveRecord::Base
  default_value_for :name, "(no name)"
  default_value_for :last_seen do
    Time.now
  end
end

u = User.new
u.name       # => "(no name)"
u.last_seen  # => Mon Sep 22 17:28:38 +0200 2008

We at Phusion use it for generating UUIDs for models.

Note: critics might be interested in the “When (not) to use default_value_for?” section. Please read on.

Installation

Install with:

./script/plugin install git://github.com/FooBarWidget/default_value_for.git

See also the AgileWebDevelopment Plugins entry.

If you like this plugin, then please consider donating and/or recommending us:

Hongli Lai Ninh Bui

The default_value_for method

The default_value_for method is available in all ActiveRecord model classes.

The first argument is the name of the attribute for which a default value should be set. This may either be a Symbol or a String.

The default value itself may either be passed as the second argument:

default_value_for :age, 20

…or it may be passed as the return value of a block:

default_value_for :age do
  if today_is_sunday?
    20
  else
    30
  end
end

If you pass a value argument, then the default value is static and never changes. However, if you pass a block, then the default value is retrieved by calling the block. This block is called not once, but every time a new record is instantiated and default values need to be filled in.

The latter form is especially useful if your model has a UUID column. One can generate a new, random UUID for every newly instantiated record:

class User < ActiveRecord::Base
  default_value_for :uuid do
    UuidGenerator.new.generate_uuid
  end
end

User.new.uuid  # => "51d6d6846f1d1b5c9a...."
User.new.uuid  # => "ede292289e3484cb88...."

Note that record is passed to the block as an argument, in case you need it for whatever reason:

class User < ActiveRecord::Base
  default_value_for :uuid do |x|
    x   # <--- a User object
    UuidGenerator.new.generate_uuid
  end
end

Rules

Instantiation of new record

Upon instantiating a new record, the declared default values are filled into the record. You’ve already seen this in the above examples.

Retrieval of existing record

Upon retrieving an existing record, the declared default values are not filled into the record. Consider the example with the UUID:

user = User.create
user.uuid   # => "529c91b8bbd3e..."

user = User.find(user.id)
# UUID remains unchanged because it's retrieved from the database!
user.uuid   # => "529c91b8bbd3e..."

Mass-assignment

If a certain attribute is being assigned via the model constructor’s mass-assignment argument, that the default value for that attribute will not be filled in:

user = User.new(:uuid => "hello")
user.uuid   # => "hello"

However, if that attribute is protected by attr_protected or attr_accessible, then it will be filled in:

class User < ActiveRecord::Base
  default_value_for :name, 'Joe'
  attr_protected :name
end

user = User.new(:name => "Jane")
user.name   # => "Joe"

Inheritance

Inheritance works as expected. All default values are inherited by the child
class:

class User < ActiveRecord::Base
  default_value_for :name, 'Joe'
end

class SuperUser < User
end

SuperUser.new.name   # => "Joe"

Attributes that aren’t database columns

default_value_for also works with attributes that aren’t database columns. It works with anything for which there’s an assignment method:

# Suppose that your 'users' table only has a 'name' column.
class User < ActiveRecord::Base
  default_value_for :name, 'Joe'
  default_value_for :age, 20
  default_value_for :registering, true
  
  attr_accessor :age
  
  def registering=(value)
    @registering = true
  end
end

user = User.new
user.age    # => 20
user.instance_variable_get('@registering')    # => true

Caveats

A conflict can occur if your model class overrides the ‘initialize’ method, because this plugin overrides ‘initialize’ as well to do its job.

class User < ActiveRecord::Base
  def initialize  # <-- this constructor causes problems
    super(:name => 'Name cannot be changed in constructor')
  end
end

We recommend you to alias chain your initialize method in models where you use default_value_for:

class User < ActiveRecord::Base
  default_value_for :age, 20
  
  def initialize_with_my_app
    initialize_without_my_app(:name => 'Name cannot be changed in constructor')
  end
  
  alias_method_chain :initialize, :my_app
end

Also, stick with the following rules:

  • There is no need to alias_method_chain your initialize method in models that don’t use default_value_for.
  • Make sure that alias_method_chain is called after the last default_value_for occurance.

When (not) to use default_value_for?

You can also specify default values in the database schema. For example, you can specify a default value in a migration as follows:

create_table :users do |t|
  t.string    :username,  :null => false, :default => 'default username'
  t.integer   :age,       :null => false, :default => 20
  t.timestamp :last_seen, :null => false, :default => Time.now
end

This has the same effect as passing the default value as the second argument to default_value_for:

user = User.new
user.username   # => 'default username'
user.age        # => 20
user.timestamp  # => Mon Sep 22 18:31:47 +0200 2008

It’s recommended that you use this over default_value_for whenever possible.

However, it’s not possible to specify a schema default for serialized columns. With default_value_for, you can:

class User < ActiveRecord::Base
  serialize :color
  default_value_for :color, [255, 0, 0]
end

And if schema defaults don’t provide the flexibility that you need, then default_value_for is the perfect choice. For example, with default_value_for you could specify a per-environment default:

class User < ActiveRecord::Base
  if RAILS_ENV == "development"
    default_value_for :is_admin, true
  end
end

Or, as you’ve seen in an earlier example, you can use default_value_for to generate a default random UUID:

class User < ActiveRecord::Base
  default_value_for :uuid do
    UuidGenerator.new.generate_uuid
  end
end

Or you could use it to generate a timestamp that’s relative to the time at which the record is instantiated:

class User < ActiveRecord::Base
  default_value_for :account_expires_at do
    3.years.from_now
  end
end

User.new.account_expires_at   # => Mon Sep 22 18:43:42 +0200 2008
sleep(2)
User.new.account_expires_at   # => Mon Sep 22 18:43:44 +0200 2008

Finally, it’s also possible to specify a default via an association:

# Has columns: 'name' and 'default_price'
class SuperMarket < ActiveRecord::Base
  has_many :products
end

# Has columns: 'name' and 'price'
class Product < ActiveRecord::Base
  belongs_to :super_market
  
  default_value_for :price do |product|
    product.super_market.default_price
  end
end

super_market = SuperMarket.create(:name => 'Albert Zwijn', :default_price => 100)
soap = super_market.products.create(:name => 'Soap')
soap.price   # => 100

What about before_validate/before_save?

True, before_validate and before_save does what we want if we’re only interested in filling in a default before saving. However, if one wants to be able to access the default value even before saving, then be prepared to write a lot of code. Suppose that we want to be able to access a new record’s UUID, even before it’s saved. We could end up with the following code:

# In the controller
def create
  @user = User.new(params[:user])
  @user.generate_uuid
  email_report_to_admin("#{@user.username} with UUID #{@user.uuid} created.")
  @user.save!
end

# Model
class User < ActiveRecord::Base
  before_save :generate_uuid_if_necessary
  
  def generate_uuid
    self.uuid = ...
  end
  
  private
    def generate_uuid_if_necessary
      if uuid.blank?
        generate_uuid
      end
    end
end

The need to manually call generate_uuid here is ugly, and one can easily forget to do that. Can we do better? Let’s see:

# Controller
def create
  @user = User.new(params[:user])
  email_report_to_admin("#{@user.username} with UUID #{@user.uuid} created.")
  @user.save!
end

# Model
class User < ActiveRecord::Base
  before_save :generate_uuid_if_necessary
  
  def uuid
    value = read_attribute('uuid')
    if !value
      value = generate_uuid
      write_attribute('uuid', value)
    end
    value
  end
  
  # We need to override this too, otherwise User.new.attributes won't return
  # a default UUID value. I've never tested with User.create() so maybe we
  # need to override even more things.
  def attributes
    uuid
    super
  end
  
  private
    def generate_uuid_if_necessary
      uuid  # Reader method automatically generates UUID if it doesn't exist
    end
end

That’s an awful lot of code. Using default_value_for is easier, don’t you think?

What about other plugins?

I’ve only been able to find 2 similar plugins:

Default Value appears to be unmaintained; its SVN link is broken. This leaves only ActiveRecord Defaults. However, it is semantically dubious, which leaves it wide open for corner cases. For example, it is not clearly specified what ActiveRecord Defaults will do when attributes are protected by attr_protected or attr_accessible. It is also not clearly specified what one is supposed to do if one needs a custom initialize method in the model.

I’ve taken my time to thoroughly document default_value_for’s behavior.

Credits

I’ve wanted such functionality for a while now and it baffled me that ActiveRecord doesn’t provide a clean way for me to specify default values. After reading http://groups.google.com/group/rubyonrails-core/browse_thread/thread/b509a2fe2b62ac5/3e8243fa1954a935, it became clear that someone needs to write a plugin. This is the result.

Thanks to Pratik Naik for providing the initial code snippet on which this plugin is based on: http://m.onkey.org/2007/7/24/how-to-set-default-values-in-your-model

If you like this plugin, then please consider donating and/or recommending us:

Hongli Lai Ninh Bui