Phusion Blog

Executive summary: our web host Linode has been compromised and the
responsible hacker group appears to claim to have had access to one of
the Phusion servers, which prompted us to start a full investigation.
Until now, no evidence of third party access has been found, and no
tampering of the Phusion Passenger Enterprise files have been found.
In spite of this, we are taking precautionary action and we urge
customers to verify their Phusion Passenger Enterprise installations
through the instructions at the bottom of this message.

Dear users and customers,

About 3 weeks ago, our web host Linode issued several public statements[1][2]
claiming that one of their customers was the subject of an attack by a group
called HTP. From what we’ve been able to read from HTP[3] a few hours ago, we
believe that SwiftIRC and/or nmap was the target Linode was referring to.

In Linode’s initial statement[1], they also mentioned that law officials were
aware of the attack and that Linode had found no evidence of other customer
data being compromised. We too hadn’t noticed any suspicious activity on our
servers and weren’t notified by Linode about being the attacked target which
led us to believe that this initial statement held true.

A few hours ago however, a statement released by HTP was brought to our
attention wherein they claimed otherwise[3]. In particular, the statement
appears to claim that HTP has had root access to one of the Phusion servers and
this immediately prompted us to start a new investigation of our own. Up to this
point, we have found no evidence that they have had access to our data, but we
are checking our systems several times over to minimize the possibility of
having missed a potential attack vector on the first few passes. We have also
contacted Linode to get a clarification on their first statement[1] in light of
new events that seem to point to nmap’s server to have indeed been compromised.
Pending this response, we didn’t want to take any risks in waiting to notify
our customers of the current situation.

The absence of evidence after all doesn’t necessarily mean that the server has
not been accessed: even though we feel we have taken all the necessary steps to
ensure maximum security on our servers, we remain scrutinous of our systems’
integrity at all times. There are after all a myriad of components that comprise
a server, and each of them could be a potential attack vector as long as fault
free software is something developers in general can only hope to aspire to.
More specifically, as long as erring is human, we can only hope to minimize
these chances rather than believing we can prevent them completely 100% of the
time. Zero day exploits can always occur at any time and the best thing we can
do is to be as transparent about this to our customers as we can. To that end,
we’d like to notify our customers that we are moving our services to another
web host and will be reinstalling our servers as a precaution.

If HTP has indeed compromised our systems without us being able to tell, then we
would be interested in learning how and would encourage them to contact us
(info@phusion.nl). We value security and transparency over pride and are
extremely committed towards serving our customers. It is also the reason why we
are informing our customers about this in an open manner several hours after
seeing HTP’s claim despite not being able to verify this claim to be accurate
ourselves.

We would also like to take this opportunity to encourage all Phusion Passenger
users – that is, open source users and Enterprise customers alike – to make use of
the PGP digital signatures that we employed since February this year.[4] Checking
the signature of your Phusion Passenger download against the corresponding key
helps minimize the chances of the downloaded software being tampered with.
We have already manually reviewed the Phusion Passenger Enterprise source code
and have found no evidence of suspicious activity. For your own safety however,
we would always recommend you to take proper caution when downloading and
installing software from the internet. The PGP digital signatures are provided
to aid in that aspect and we would highly recommend you to use this at all times.

Having said this, if our servers actually were accessed, then it’s possible that
the attackers temporarily inserted compromised gems and tarballs and removed
them later. We therefore urge our Enterprise customers to verify the integrity
of their Phusion Passenger Enterprise installations. Instructions can be found
at the end of this message.

In any case, Phusion has not, does not and will not store customer creditcard
information on any of its servers. All credit card information is stored on
servers of third party, PCI-DSS compliant payment gateways, e.g. FastSpring
and Paypal. Phusion also does not store customer passwords in plain text; all
customer passwords are stored in BCrypt format.

The open source version of Phusion Passenger is hosted on another server, namely
GitHub, and we have also found no suspicious activity in its repository.

We understand that after reading all this, you might have concerns with regards
to your own server’s integrity. Even though we have found no evidence of
suspicious activity on our own servers or in Phusion Passenger’s code base, we
feel that we should still encourage you to remain scrutinous of your own
servers’ integrity and take the steps you deem necessary in maximizing its
security.

Needless to say, we remain committed in being transparent towards our customers
and will continue in keeping them up to date of any of our findings concerning
this matter. If you have any questions, please feel encouraged to contact
support@phusion.nl.

With warm regards,
Hongli Lai
Ninh Bui

References:

1. https://blog.linode.com/2013/04/12/security-notice-linode-manager-password-reset/
2. https://blog.linode.com/2013/04/16/security-incident-update/
3. http://straylig.ht/zines/HTP5/0x02_Linode.txt
4. http://www.modrails.com/documentation/Users%20guide%20Apache.html#_cryptographic_verification_of_installation_files

Instructions for verifying Phusion Passenger Enterprise installations

We have generated SHA-1 hashes of all Phusion Passenger Enterprise files
inside the gems and tarballs. You can use these hashes to verify your installed
Phusion Passenger files. If anything is amiss or if you require further
assistance, please contact support@phusion.nl.

1. Install GnuPG. Debian users can apt-get install gnupg, OS X users can use GPG Tools: https://gpgtools.org/
2. Login to the Customer Area: https://www.phusionpassenger.com/orders
3. Scroll down to the “Files” section.
4. Download the “sha1sums.txt” and “sha1sums.txt.asc” files that pertain to the
   version of Phusion Passenger Enterprise that you’re currently running.
   Ensure that both files are in the same directory.
5. Import the Phusion Software Signing PGP key: http://www.modrails.com/documentation/Users%20guide%20Apache.html#_importing_the_phusion_software_signing_key
   Name: Phusion Software Signing (software-signing@phusion.nl)
   Short key ID: 0x0A212A8C
   Long key ID: 0x2AC745A50A212A8C
   Fingerprint: D5F0 8514 2693 9232 F437 AB72 2AC7 45A5 0A21 2A8C
6. Set this key to trusted:
   gpg –edit-key software-signing@phusion.nl
   Then in the GPG prompt, type: trust
   Choose: 5 = I trust ultimately
   In the GPG prompt, type: save
7. Verify the downloaded sha1sums.txt against its signature:
   gpg –verify sha1sums.txt.asc
   You should see:
   Good signature from “Phusion Software Signing software-signing@phusion.nl“
8. Copy sha1sums.txt to your server.
9. On your server, find out where the Phusion Passenger files are by running: passenger-config –root
10. Run: cd
11. Run: sha1sum -c /path-to/sha1sums.txt –quiet