Phusion white papers Phusion overview

Phusion Blog

The Phusion founders’ PGP keys have been updated

By Hongli Lai on March 25th, 2014

Phusion takes security very seriously. This is why we strongly believe in protecting the authenticity and integrity of our communications and our software, and why we employ the use of PGP digital signatures. Using our PGP keys, you can verify the authenticity and integrity of all emails and files that we publish to you or to the world. All software releases that we make are signed with one of our PGP keys.

The founders’ keys have changed

As Phusion’s founders, we – Hongli Lai and Ninh Bui – have our own personal PGP keys as well, which we use to encrypt or sign some of our emails and git commits. We’ve recently run a security audit and noticed that our PGP keys are no longer deemed as secure as they should be. The keys that we’ve been using until today were made back in 2009, but the recommended algorithms and key sizes in 2014 are quite different from what they were 5 years ago. For this reason, we’ve decided to revoke our old keys and to create new ones, with stronger security settings.

Nothing has been compromised. We are simply renewing our keys as a precaution.

Effective immediately, our new PGP keys are as follows:

  • Hongli Lai (hongli@phusion.nl)
    Short key ID: 8C59158F
    Long key ID: CD70085E8C59158F
    Fingerprint: 218A 7255 83D0 2ECE F3A9 C2A7 CD70 085E 8C59 158F
  • Ninh Bui (ninh@phusion.nl)
    Short key ID: 69481265
    Long key ID: AE405F7869481265
    Fingerprint: A77C 9CEF 766D 0E7D A95B 8778 AE40 5F78 6948 1265

If you had our old keys in your keyring, please update them so that you see the revocations:

gpg --refresh-keys --keyserver pool.sks-servers.net
# -OR-
gpg --refresh-keys --keyserver keyserver.ubuntu.com

No effect on the signatures of our file releases

Please note that Phusion’s software releases and Ruby gems are not signed with our personal keys. Instead, they’re signed with the Phusion Software Signing key, which is still considered strong enough.

Our git commits, though, are often signed with our personal keys.

If you’re using Phusion Passenger, we strongly recommend you to cryptographically verify every release. The Phusion Passenger documentation contains comprehensive instructions that explains how you can verify our tarballs, Ruby gems, Git commits and more.

Onward and upwards!

With kind regards,
Hongli Lai
Ninh Bui

Enterprise customers: please update your gem source and APT repository URL

By Hongli Lai on November 1st, 2013

A short while ago, we introduced a gem server and an APT repository for Phusion Passenger Enterprise customers. HTTP basic authentication was used to authenticate with these servers. Customers were supposed to use URLs that look like this:

https://YOUR_ORDER_REFERENCE:YOUR_PASSWORD@www.phusionpassenger.com/enterprise_gems/
deb https://YOUR_ORDER_REFERENCE:YOUR_PASSWORD@www.phusionpassenger.com/enterprise_apt precise main

Today, we are deprecating the ORDER_REFERENCE:PASSWORD authentication string in favor of download tokens. Each customer has a unique download token, which can be found in the Customer Area. The new RubyGems source URL and APT URL look like this:

https://download:YOUR_DOWNLOAD_TOKEN@www.phusionpassenger.com/enterprise_gems/
deb https://download:YOUR_DOWNLOAD_TOKEN@www.phusionpassenger.com/enterprise_apt precise main

Why are we doing this?

We made this change as a security precaution. Just to be clear: there is no security incident. Nothing has been breached or leaked, and the old mechanism was not vulnerable.

Since the authentication string can be read as clear text in configuration files on the system, anybody who can access the configuration files will be able to login to the Customer Area. This in itself has no security consequences for customers: the only thing you can do in the Customer Area is to view licensing details and to download Phusion Passenger Enterprise. Credit card details cannot be retrieved from the Customer Area.

Since customer passwords are automatically generated by our system and are completely random, being able to read the password is of no use other than being able to login to the Customer Area.

However, we might extend the functionality of the Customer Area in the future. As such, we are changing the authentication string so that those who have access to the system configuration files cannot login to the Customer Area. With this new download token mechanism, the only thing they will be able to do is to download Phusion Passenger Enterprise files, no matter how much we extend the Customer Area in the future.

Migration instructions

The old ORDER_REFERENCE:PASSWORD mechanism will continue to work for 1 month. We will disable it after 1 month so please migrate now.

Step 1: remove your old URLs

Remove your existing Enterprise gem source (you might have to run this with sudo or rvmsudo depending on the circumstances):

gem source --remove https://YOUR_ORDER_REFERENCE:YOUR_PASSWORD@www.phusionpassenger.com/enterprise_gems/

Remove your Enterprise APT source:

sudo rm -f /etc/apt/sources.list.d/passenger.list

Step 2: add the new URLs

If you’re using RubyGems to install Passenger Enterprise, add the new gem source (you might have to run this with sudo or rvmsudo depending on the circumstances):

gem source --add https://download:YOUR_DOWNLOAD_TOKEN@www.phusionpassenger.com/enterprise_gems/

If you’re using APT to install Passenger Enterprise, add the new APT URL to /etc/apt/sources.list.d/passengerlist:

deb https://download:YOUR_DOWNLOAD_TOKEN@www.phusionpassenger.com/enterprise_apt DISTRIBUTION_NAME main

Please refer to the manual for a full list of APT URLs for the different distributions.

Rails SQL injection vulnerability: hold your horses, here are the facts

By Hongli Lai on January 3rd, 2013

Update: see also the article Securing the Rails session secret.

Update 2: a statement from Michael Koziarski of the Rails security team regarding the severity of this bug has been added. He urges people to upgrade immediately. Please scroll to the “Conclusion” section for details.

Update 3: new advisories (CVE-2013-0155 and CVE-2013-0156) have been published. These vulnerabilities are unrelated to the one reported in this blog post, but are extremely critical. Upgrade immediately.

Yesterday a Ruby on Rails SQL injection vulnerability was announced which affects all versions. This immediately received widespread attention on Hacker News. Unfortunately the announcement doesn’t clearly explain how the vulnerability exactly works, which caused a lot of confusion and unnecessary panic, especially among people who are less familiar with Ruby or Rails.

Here are the facts, along with a clear explanation for non-Rails people.

Summary: what is this vulnerability?

  • The bug allows SQL injection through dynamic finder methods (e.g. find_by_foo(params[:foo])). I will explain dynamic finders in a bit.
  • The bug affects all Ruby on Rails versions.
  • A known exploitable scenario is when all of the following applies:
    1. You’re using Authlogic (a third party but popular authentication library).
    2. You must know the session secret token.

    There are other exploitable scenarios, but it really depends on what your app is doing. Since it is impossible to prove that something isn’t insecure, you should take the vulnerability seriously and upgrade anyway even if you think you aren’t affected.

What is this vulnerability NOT?

For those who know Rails:

  • The bug does not affect normal finder methods (e.g. find(params[:id])).
  • The bug is not exploitable through request parameters.
  • The bug is not in Authlogic. It’s in Rails. It just so happens that Authlogic triggers it.
  • Devise (another third-party authentication library) does not trigger the bug.
  • Point 6, as described at Rails Security Digest. ‘params’ Case, is a totally different and unrelated issue. The issue described there is quite severe and deserves serious attention, so please keep your eye open on any new advisories.

For those who do not know Rails:

  • It does not mean all unupgraded Rails apps are suddenly widely vulnerable.
  • It does not mean Rails doesn’t escape SQL inputs.
  • It does not mean Rails doesn’t provide parameterized SQL APIs.
  • It does not mean Rails encourages code that are inherently prone to SQL injection. The code should be safe but due to a subtlety was not. This has been fixed.

The main exploitable scenario

Rails provides finder methods for all ActiveRecord (database) models. For example, to lookup a user using a primary key that was provided through the “id” request parameter, one would usually write:

User.find(params[:id])

Rails also provides so-called “dynamic finder methods”. It generates a “find_by_*” method for all database columns in model. If your “users” table have the “id”, “name” and “phone” columns, then it will generate methods so you can write things like this:

User.find_by_id(params[:id])
User.find_by_name(params[:name])
User.find_by_phone(params[:name])

The vulnerability is in these dynamic finder methods, not in the normal and often-used find method.

ActiveRecord protects you against SQL injection by escaping input for you. For example the following works as expected, with no vulnerability:

User.find_by_name("kotori'; DROP TABLE USERS; --")
# => SELECT * FROM users WHERE name = 'kotori\'; DROP TABLE USERS; --' LIMIT 1

But ActiveRecord also defines ways for the programmer to inject SQL fragments into the query so that the programmer can customize the query when necessary. The injection interfaces are documented and the programmer is not supposed to pass user input to those interfaces. Normally, the strings passed to the injection interfaces are constant strings that never change. One of those injection interfaces is the options parameter (normally second parameter) for the “find_by_*” methods:

# Fetches a user record by name, but only fetch the 'id' and 'name' fields.
User.find_by_name("kotori", :select => "id, name")
# => SELECT id, name FROM users WHERE name = 'kotori' LIMIT 1

# You can inject arbitrary SQL if you wish:
User.find_by_name("kotori", :select => "id, name FROM users; DROP TABLE users; --")
# => SELECT id, name FROM users; DROP TABLE users; -- FROM users WHERE name = 'kotori' LIMIT 1

The vulnerability lies in the fact that “find_by_*” also accepted calls in which only the options parameter is given. In that case, it thinks that the value parameter is nil.

User.find_by_name(:select => "1; DROP TABLE users; --")
# => SELECT 1; DROP TABLE users; -- FROM users WHERE name IS NULL LIMIT 1;

Not many people ever use the second parameter, but code of the following form is quite common:

User.find_by_name(params[:name])

params[:name] is normally a string. Can an attacker somehow ensure that params[:name] is an options hash? Yes. Rails converts request parameters of a certain form into hashes. Suppose you call the controller method like this:

/example-url?name[select]=whatever&name[limit]=23

params[:name] is now a hash: { "select" => "whatever", "limit" => 23 }

However, this is not exploitable. Ruby has two datatypes, strings and symbols. Symbols are kind of like string constants. You’ve seen them before in this article: :select is a symbol. The vulnerability can only be triggered when the keys are symbols, but the Rails-generated request parameter hashes all have string keys thanks to the way HashWithIndifferentAccess works.

An attacker can only exploit this if the application somehow passes an arbitrary hash to “find_by_*”, yet with symbol keys. We now bring in the second part of the puzzle: Authlogic. This exploit is described here and works as follows.

Authlogic accepts authentication credentials through multiple ways: cookies, Rails session data, HTTP basic authentication, etc. All user accounts have a so-called persistence token, and the user must provide this persistence token through one of the authentication methods in order to authenticate himself. Authlogic looks up the user associated with the persistence token using roughly the following call:

User.find_by_persistence_token(the_token)

Can an attacker ensure that the_token is an options hash? Yes, but only through the Rails session data authentication method. In all the other methods, the_token is always a string.

The Rails session mechanism allows storing arbitrary Ruby objects, including hashes with symbol keys. Rails provides a variety of session stores, the default being the cookie store which stores session data in a cookie on the client. The cookie data is not encrypted, but is signed with an HMAC to prevent tampering. The cookie store is fast, does not require any server-side maintenance, and is only meant for session data that do not contain sensitive information such as credit card numbers. Apps that store sensitive information in the session should use the database session store instead. Nevertheless, it turned out that 95% of all Rails apps only ever store the user authentication credentials in the session, so the cookie store was made the default.

So to inject arbitrary SQL, you need to tamper with the cookie, which requires the HMAC key. The HMAC key is the so-called session secret. As the name implies, it is supposed to be secret. Rails generates a random 512-bit secret upon project creation. This is why most Rails apps that are running Authlogic are not exploitable: the attacker does not know the secret. Open source Rails apps however can form a problem. Many of them come with a default session secret, but the user never customizes them, so all those instances end up using the same HMAC key, making them very easily exploitable. Of course, in this case the operator have to worry about more than just SQL injection. If the HMAC key is known then anybody can send fake credentials to the app.

Other exploitable scenarios

Your code is vulnerable if you call Foo.find_by_whatever(bar), where bar can be an arbitrary user-specified hash with symbol keys. HashWithIndifferentAccess stores keys as strings, not symbols, so that does not trigger the vulnerability.

Mitigation

There are several ways to mitigate this.

  1. Upgrade to the latest Rails version. This solves everything, you don’t need to do anything else. “find_by_*” has been patched so that the first parameter may not be an options hash.
  2. Ensure that you only pass strings or integers to “find_by_*”, e.g. find_by_name(params[:name].to_s). This requires changing all code, including third party code. I do not recommend this as you’ll likely overlook things. If you upgrade Rails, you don’t need to do this.
  3. Keep your session secret secret! If you write open source Rails apps, make sure the user generates a different session secret upon installation. Don’t let them use the default one.

Demo app

We’ve put together a demo app which shows that the bug is not exploitable through request parameters. Setup this app, run it, and try to attack it as follows:

curl 'http://127.0.0.1:3000/?id\[limit\]=1'

You will see that this attack does not succeed in injecting SQL.

Conclusion

So here you have it. Some folks on Hacker News asked “how can this giant bug be overlooked”? As you can see, it is not a “giant bug”, it is much more subtle than that and requires a specific combination of code and circumstances to work. Most apps are not vulnerable.

Update: Michael Koziarski of the Rails security team said the following:

“When we told people they should upgrade immediately we meant it. It *is* exploitable under some circumstances, so people should be upgrading immediately to avoid the risk.”

References

SHA-3 extensions for Ruby and Node.js

By Hongli Lai on October 6th, 2012

A few days ago, NIST announced the winner of the SHA-3 competition: Keccak (prounced [kɛtʃak], ketchak). The researchers who authored Keccak released a reference implementation in C.

We’ve created Ruby and Node.js extensions for Keccak. Our extensions utilize the code from the official reference implementation and come with a extensive suite of unit tests. But note however that I do not claim to be a security expert. Feel free to review the code for any flaws.

Install with:

gem install digest-sha3
npm install sha3

We’ve strived to emulate the languages’ standard hash libraries’ interfaces, so using these extensions is straightforward:

require 'digest/sha3'
Digest::SHA3.hexdigest("foo")

and

var SHA3 = require('sha3');
var hash = SHA3.SHA3Hash();
hash.update('foo');
hash.digest();

Both libraries are MIT licensed. Enjoy!

Why does the world need SHA-3?

If you’re not a security researcher then you’ve undoubtedly asked the same questions as I did. What’s wrong with SHA-1, SHA-256 and SHA-512? Why does the world need SHA-3? Why was Keccak the winner?

According to to well-known security researcher Bruce Schneier, there’s nothing wrong with the SHA-2 family of hashing functions. However he likes SHA-3 because it’s completely different. SHA-1 and SHA-2 are both based on the Merkle–Damgård construction. It may be feasible to find a flaw in SHA-1 that also affects SHA-2. In contrast, Keccak is based on the sponge construction so any attacks on SHA-1 and SHA-2 will probably not affect Keccak. Indeed, it appears that NIST chose Keccak because it was looking for some kind of insurance in case SHA-2 would be broken. Many people commented that they expected Skein to win.

Do not hash your passwords

In any case, the following cannot be repeated enough. Do not use SHA-3 (or SHA-256, SHA-512, RIPEMD-160 or whatever fast hash) to hash passwords! Do not even use SHA-3 + salt to hash passwords. Instead, use a slow hash like bcrypt. As Coda Hale explained in his article, you’ll want the hash to be slow so you can defend against attackers effectively.

Announcing EncryptedCookieStore plugin for Rails 2.3

By Hongli Lai on April 13th, 2010

EncryptedCookieStore is similar to Ruby on Rails’s CookieStore (it saves session data in a cookie), but it uses encryption so that people can’t read what’s in the session data. This makes it possible to store sensitive data in the session.

EncryptedCookieStore is written for Rails 2.3. Other versions of Rails have not been tested.

Note: This is not ThinkRelevance’s EncryptedCookieStore. In the Rails 2.0 days they wrote an EncryptedCookieStore, but it seems their repository had gone defunct and their source code lost. This EncryptedCookieStore is written from scratch by Phusion.

Source code at http://github.com/FooBarWidget/encrypted_cookie_store

Installation and usage

First, install it:

./script/plugin install git://github.com/FooBarWidget/encrypted_cookie_store.git

Then edit config/initializers/session_store.rb and set your session store to EncryptedCookieStore:

ActionController::Base.session_store = EncryptedCookieStore

You need to set a few session options before EncryptedCookieStore is usable. You must set all options that CookieStore needs, plus an encryption key that EncryptedCookieStore needs. In session_store.rb:

ActionController::Base.session = {
        # CookieStore options...
        :key            => '_session',     # Name of the cookie which contains the session data.
        :secret         => 'b4589cc9...',  # A secret string used to generate the checksum for
                                           # the session data. Must be longer than 64 characters
                                           # and be completely random.

        # EncryptedCookieStore options...
        :encryption_key => 'c306779f3...', # The encryption key. See below for notes.
}

The encryption key must be a hexadecimal string of exactly 32 bytes. It should be entirely random, because otherwise it can make the encryption weak.

You can generate a new encryption key by running rake secret:encryption_key. This command will output a random encryption key that you can then copy and paste into your environment.rb.

Operational details

Upon generating cookie data, EncryptedCookieStore generates a new, random initialization vector for encrypting the session data. This initialization vector is then encrypted with 128-bit AES in ECB mode. The session data is first protected with an HMAC to prevent tampering. The session data, along with the HMAC, are then encrypted using 256-bit AES in CFB mode with the generated initialization vector. This encrypted session data + HMAC are then stored, along with the encrypted initialization vector, into the cookie.

Upon unmarshalling the cookie data, EncryptedCookieStore decrypts the encrypted initialization vector and use that to decrypt the encrypted session data + HMAC. The decrypted session data is then verified against the HMAC.

The reason why HMAC verification occurs after decryption instead of before decryption is because we want to be able to detect changes to the encryption key and changes to the HMAC secret key, as well as migrations from CookieStore. Verifying after decryption allows us to automatically invalidate such old session cookies.

EncryptedCookieStore is quite fast: it is able to marshal and unmarshal a simple session object 5000 times in 8.7 seconds on a MacBook Pro with a 2.4 Ghz Intel Core 2 Duo (in battery mode). This is about 0.174 ms per marshal+unmarshal action. See rake benchmark in the EncryptedCookieStore sources for details.

EncryptedCookieStore vs other session stores

EncryptedCookieStore inherits all the benefits of CookieStore:

  • It works out of the box without the need to setup a seperate data store (e.g. database table, daemon, etc).
  • It does not require any maintenance. Old, stale sessions do not need to be manually cleaned up, as is the case with PStore and ActiveRecordStore.
  • Compared to MemCacheStore, EncryptedCookieStore can “hold” an infinite number of sessions at any time.
  • It can be scaled across multiple servers without any additional setup.
  • It is fast.
  • It is more secure than CookieStore because it allows you to store sensitive data in the session.

There are of course drawbacks as well:

  • It is prone to session replay attacks. These kind of attacks are explained in the Ruby on Rails Security Guide. Therefore you should never store anything along the lines of is_admin in the session.
  • You can store at most a little less than 4 KB of data in the session because that’s the size limit of a cookie. “A little less” because EncryptedCookieStore also stores a small amount of bookkeeping data in the cookie.
  • Although encryption makes it more secure than CookieStore, there’s still a chance that a bug in EncryptedCookieStore renders it insecure. We welcome everyone to audit this code. There’s also a chance that weaknesses in AES are found in the near future which render it insecure. If you are storing *really* sensitive information in the session, e.g. social security numbers, or plans for world domination, then you should consider using ActiveRecordStore or some other server-side store.

JRuby: Illegal Key Size error

If you get this error (and your code works with MRI)…

    Illegal key size
    
    [...]/vendor/plugins/encrypted_cookie_store/lib/encrypted_cookie_store.rb:62:in `marshal'

…then it probably means you don’t have the “unlimited strength” policy files installed for your JVM. Download and install them. You probably have the “strong” version if they are already there.

As a workaround, you can change the cipher type from 256-bit AES to 128-bit by
inserting the following in config/initializer/session_store.rb:

EncryptedCookieStore.data_cipher_type = 'aes-128-cfb'.freeze  # was 256

Please note that after changing to 128-bit AES, EncryptedCookieStore still requires a 32 bytes hexadecimal encryption key, although only half of the key is actually used.

Securely store passwords with bcrypt-ruby; now compatible with JRuby and Ruby 1.9

By Hongli Lai on August 13th, 2009

When writing web applications, or any application for that manner, any passwords should be stored securely. As a rule of thumb, one should never store passwords as clear text in the database for the following reasons:

  • If the database ever gets leaked out, then all accounts are compromised until every single user resets his password. Imagine that you’re an MMORPG developer; leaking out the database with clear text passwords allows the attacker to delete every player’s characters.
  • Many people use the same password for multiple sites. Imagine that the password stored in your database is also used for the user’s online banking account. Even if the database does not get leaked out, the password is still visible to the system administrator; this can be a privacy breach.

There are several “obvious” alternatives, which aren’t quite secure enough:

Storing passwords as MD5/SHA1/$FAVORITE_ALGORITHM hashes
These days MD5 can be brute-force cracked with relatively little effort. SHA1, SHA2 and other algorithms are harder to brute-force, but the attacker can still crack these hashes by using rainbow tables: precomputed tables of hashes with which the attacker can look up the input for a hash with relative ease. This rainbow table does not have to be very large: it just has to contain words from the dictionary, because many people use dictionary words as passwords.

Using plain hashes also makes it possible for an attacker to determine whether two users have the same password.

Encrypting the password
This is not a good idea because if the attacker was able to steal the database, then there’s a possibility that he’s able to steal the key file as well. Plus, the system administrator is able to read everybody’s passwords, unless he’s restricted access to either the key file or the database.

The solution is to store passwords as salted hashes. One calculates a salted hash as follows:

salted_hash = hashing_algorithm(salt + cleartext_password)

Here, salt is a random string. After calculating the salted hash, one should store the salted hash in the database, along with the (cleartext) salt. It is not necessary to keep the salt secret or to obfuscate it.

When a user logs in, one can verify his password by re-computing the salted hash and comparing it with the salted hash in the database:

salted_hash = hashing_algorithm(salt_from_database + user_provided_password)
if (salted_hash == salted_hash_from_database):
    user is logged in
else:
    password incorrect

The usage of the salt forces the attacker to either brute-force the hash or to use a ridiculously large rainbow table. In case of the latter, the sheer size of the required rainbow table can make it unpractical to generate. The larger the salt, the more difficult it becomes for the cracker to use rainbow tables.

However, even with salting, one should still not use SHA1, SHA2, Whirlpool or most other hashing algorithms because these algorithms are designed to be fast. Although brute forcing SHA2 and Whirlpool is hard, it’s still possible given sufficient resources. Instead, one should pick a hashing algorithm that’s designed to be slow so that brute forcing becomes unfeasible. Bcrypt is such a slow hashing algorithm. A speed comparison on a MacBook Pro with 2 Ghz Intel Core 2 Duo:

  • SHA-1: 118600 hashes per second.
  • Bcrypt (with cost = 10): 7.7 hashes per second.

Theoretically it would take 4*10^35 years for a single MacBook Pro core to crack an SHA-1 hash, assuming that the attacker does not harness any weaknesses in SHA-1. To crack a bcrypt hash one would need 6*10^39 years, or 10000 more times. Therefore, we recommend the use of bcrypt to store passwords securely.

There’s even a nice Ruby implementation of this algorithm: bcrypt-ruby! Up until recently, bcrypt-ruby was only available for MRI (“Matz Ruby Interpreter”, the C implementation that most people use). However, we’ve made it compatible with JRuby! The code can be found in our fork at Github. The current version also has issues with Ruby 1.9, which we’ve fixed as well. The author of bcrypt-ruby has already accepted our changes and will soon release a new version with JRuby and Ruby 1.9 support.

Further recommended reading

How to Safely Store a Password by Coda Hale.