Phusion Blog

Mail in 2012 from an admin’s perspective

There has been a lot of discussion about the current state of e-mail and how we use it. Many blog posts popped up, describing what has changed in the past several decades and how we could adapt to the new way e-mail is used. Different approaches and possible solutions are presented, but they all have one thing in common: they try to solve problems on the side of the end-user, the one reading his or her e-mail. But over those same several decades, a lot has changed for the system administrators too. And while most of the e-mail users don’t know (or even want to know) what happens behind the scenes, admins have sleepless nights to make sure you get your e-mails.

This blog post comes from the fact that I had an unnecessary hard time setting up SRS in Postfix. If you want to know how to do just that, please find the instructions in the latter part of this post. I’ll start with a short intro on what SPF and SRS are, why we need them, and why you might need them too most likely. Lastly a short introduction to DKIM is given, which is another nice way of improving the e-mail system. All of the described here are techniques to improve your and others’ mailing experience by verifying the sender: this helps identifying and reducing spam, and also prevents legitimate servers being wrongfully accused for spamming or other unwanted behaviour. The techniques are not new, in fact they have proven themselves in the last couple of years. Many e-mail service providers check SPF and DKIM, and Google Gmail’s spam filtering mechanisms take the validity into account. They even enforce DKIM validity on eBay and Paypal messages as these domains are obviously interesting for phising and abuse alike.

Rocket Science

Is setting up an e-mail server that hard? In the basics, no. Setting up an MTA to send a message to another MTA is not anywhere near rocket science. And while e-mail is nothing more than exchanging aforementioned messages, we are already halfway, right? But the challenges administrators have to deal with are spam (end-users know that one of course) and all the problems that come with it. Ever got an e-mail from somecrap@yourdomain.com? Or got a complaint that someone@yourdomain.com sent an e-mail, but that someone doesn’t even exist? Forging of e-mail headers (especially the ‘From’ field) is easy, but as e-mail is a vital part of almost anyone’s daily routine, it has become problematic. So what can we do from an administrators point of view, to keep e-mail usable in 2012?

Identify and verify the mailing servers

As the forging of headers happens outside the control of the admin, methods of fighting it are limited by nature. Imagine a scenario where you control server A, but server B sends a forged e-mail to server C making it look like it came from server A (by forging the ‘From’ header). The message does not even pass your server A, so what can you do?
Only one thing in the forged e-mail is under your control: the domain name. And that is where the Sender Policy Framework (SPF) relies on. By adding special SPF records pointing to your mail servers to the DNS system, a receiving party can check whether the sending server is allowed to send mail from that domain name. Continuing on the previous example: hopefully, server C uses SPF to check the authenticity of the e-mail it got from B. As an admin, I published DNS records for server A containing the IP of server A (so A is allowed to send mail for domainA.com), but B’s IP is not in that record. Server C checks the SPF record from domainA.com and notices B is not allowed to use that domain name. A forged e-mail! Of course it is up to C to decide which way to handle the e-mail, be it dropping it, or bouncing it, or whatever they see fit.
Hopefully it has become clear that SPF is something that is not fully under your control. The only thing you can do, is provide the correct records in your DNS system, and hope that other mail servers to their SPF checks. To reduce spam towards your servers, you should of course do the SPF checking as well. I won’t go into detail about setting the records, but for the content and format check out this clear overview.

So far so good, right? Using SPF you can reduce the incoming spam, and you help other administrators to find out whether e-mails really came from your domain. But there is one problem: forwarding.
At Phusion, some of the guys like to have e-mail forwarded to their Gmail account. The forwarding itself is a breeze: Postfix has a map of virtual addresses pointing to the right Gmail address. But in an SPF-scenario, the following happens: someone@example.com sends an e-mail to info@phusion.nl, which is forwarded to personA@gmail.com and personB@gmail.com. Google checks SPF records, but the check will fail! The mail originates from example.com, so this domain is queried for an SPF record. But, the mail is received from the servers at phusion.nl. It is a shortcoming of the SPF paradigm, with a solution called the Sender Rewriting Scheme (SRS). As the name suggests, SRS rewrites the sender (on the envelope), so the next server/MTA in the chain can check the SPF records of the right domain, instead of the original domain where the e-mail originated from. So in our described scenario, someone@example.com will become someone@phusion.nl on the envelope, so when it is received by personA@gmail.com and personB@gmail.com Google’s server can check the SPF records for phusion.nl (as opposed to the records of example.com). Now, the check will pass.

As only the ‘Envelope’ is rewritten, the receiving end-user will see the normal ‘From’ headers and therefore the normal name of the sender. Almost completely invisible for the end-users, but some clients can show whether the e-mail comes from a valid source, like Gmail adds ‘via forwarder.com’ to the sender’s address.
Ever saw that one? That is SPF, maybe complemented by SRS. Or maybe you’re familiar with the Gmail popup shown above, telling you the SPF check passed (Mailed-by) but also the DKIM-signature (more on that later) was valid (Signed-by).

What is the hassle then? Maybe you already knew all of this, and you might think it is not that big of a deal. Maybe your MTA does this out of the box (please tell me then), but at Phusion we use Postfix and it costed me some sweat to get SRS up and running. To me, it seemed like almost nobody in the world uses SRS. If I’m new to a technology or tool, the first thing I do is engage the Google-fu to find out what other people use. What is hot on Github, what do people on StackOverflow suggest, which tools and software is still actively supported, that kind of things. The most recent stuff on SRS was from 2003 or 2004, and of course, mail is something well established but was there really no one interested in SRS for the last 8 years? Or is Postfix the problem?
Then we found pfix-tools (Github). It seems people are interested in SRS after all, or at least the developers behind this tool suite. As their readme states: “pfixtools is a suite of tools aiming at complementing postfix, to make it even more customizable, while keeping really high performance levels.” It contains pfix-srsd, a daemon to do the SRS rewritings for Postfix. Sweet! For those of you using Postfix, I will try to describe briefly but clearly how to set-up pfix-srsd.

Installation details: SRS for Postfix on Debian

FYI, this was done on a Debian 6 system.

Download and compile pfix-srsd

wget https://github.com/Fruneau/pfixtools/tarball/pfixtools-0.8
tar -xzvf pfixtools-0.8
aptitude install libev3 libev-dev libsrs2-0 libsrs2-dev libpcre3-dev
git clone https://github.com/Fruneau/pfixtools.git
cd pfixtools
git submodule init
git submodule update

cd common
make
cd ../pfix-srsd
make

Finally, move the resulting binary pfix-srsd to e.g. /usr/local/bin/ (the following steps will use this path, so be careful if you change it)

Create config and secrets files

/etc/default/pfix-srsd:

DOMAIN=phusion.nl
SECRETS=/etc/postfix/pfix-srs.secrets
OPTIONS=-I

In /etc/postfix/pfix-srs.secrets, enter lots of random stuff, preferably from some kind of random generator

Fix the permissions:

chown postfix:postfix /etc/postfix/pfix-srs.secrets
chmod 400 /etc/postfix/pfix-srs.secrets

In /etc/postfix/pfix-no-srs.cf you can put addresses that should not be SRS’ed.
Always compile this file after you’ve changed it with postmap /etc/postfix/pfix-no-srs.cf

Daemontools

NB: We use daemontools, but an example sysinit script is available here.

#!/bin/bash

PFIXSRSD_CONFIG="/etc/default/pfix-srsd" 
NAME="pfix-srsd" 
DAEMON="/usr/local/bin/pfix-srsd" 
PID_FILE="/var/run/pfix-srsd.pid" 

if [ -f $PFIXSRSD_CONFIG ]; then
. $PFIXSRSD_CONFIG
else
echo "Error reading config file, aborting.." 
exit 1
fi

exec setuidgid postfix $DAEMON -f $OPTIONS $DOMAIN $SECRETS >> /var/log/pfix-srsd.log 2>&1

Don’t forget to make it executable by chmod +x run

Modify Postfix config

add to /etc/postfix/main.cf:

# SRS Remapping
recipient_canonical_maps = hash:/etc/postfix/pfix-no-srs.cf, tcp:127.0.0.1:10002
recipient_canonical_classes = envelope_recipient
sender_canonical_maps = hash:/etc/postfix/pfix-no-srs.cf, tcp:127.0.0.1:10001
sender_canonical_classes = envelope_sender

Reload and test

Afterwards, reload the postfix service (and maybe your firewall):

invoke-rc.d postfix reload

To test, you can send a test message to a Gmail account and check the headers. It should contains something alike the following:

Received-SPF: pass (google.com: domain of SRS0=xtbD=G6=originaldomain.com=luuk@forwarder.nl designates 97.xxx.xxx.55 as permitted sender) client-ip=97.xxx.xxx.55;

Or use some of the available online testing tools, like this one. That might come in handy when you also get ur DKIM going, which you should!

Thanks and credits to Christoph Fischer who wrote a large part of the pfix-tools installation and configuration in German on his website.

What else can we do: an intro to DKIM

Another way to validate messages and their senders is DKIM, short for DomainKeys Identified Mail. It is based on digital signatures based on public-key cryptography. The public key is again available via DNS, the private key is used by the sender to create a signature and adding it in the DKIM-Signature: header. This header furthermore contains a specification of which fields are used to generate the signature, the signing algorithm, and of course the domain name. If you have a grasp of how public key cryptography works, you probably see the point already. But lets see how things are in the real world.

Google uses DKIM on their Gmail servers, not very surprisingly. Search for an e-mail from someone@gmail.com in your mail, and look for the headers. You want something like this:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
        :content-type:content-transfer-encoding;
        bh=2RwwqCV0x4QBCdFDkMasxwksfUflwAGP8M6WgWTq4Tg=;
        b=TypYGfooVtClvktv8igJcCGyNc/2HNcBkdKcxKJ8KcdeGjB6nLLH6HichqJiW8kPbE
        Sf6yu0Lg91qzvvt4THR3CMCzE7c1LGUi4nwQwWDrTbkJufXyzI6MkRfxAo9tyX3wblaz
        EmD2B7FJqFGj128++XCiKrawn8MHSU8OMjSYru9Dkq5fYUaOXUlXcINjt662RfqkfCbE
        GwHFA423ZA7mmHiwgiFNvn8rlDYNWP8WfehNU4+qc0ifsy+a+fTHjQM6cP42Xl+3VcTS
        f1A4USRwgIVG93I9iPdyoPn8ebnpg0y/ma1Y+i7qrgEi2fW2YPZZv7/WuSpQglKqs3u/
        jOzA==

What do we see? The (mostly used) rsa-sha256 algorithm (a), the domain is gmail.com (d), the headers used to construct the signature, the body hash (bh), and the signature itself (b). But very important is the selector (s), which is used in the DNS query. The following command –make sure dig is available on your system– shows the DKIM public key that we need to verify whether the signature is valid:

luuk@polecat> dig +short TXT 20120113._domainkey.gmail.com
"k=rsa\; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Kd87/UeJjenpabgbFwh+
eBCsSTrqmwIYYvywlbhbqoo2DymndFkbjOVIPIldNs/m40KF+yzMn1skyoxcTUGCQs8g3FgD2Ap3
ZB5DekAo5wMmk4wimDO+U8QzI3SD0" "7y2+07wlNWwIt8svnxgdxGkVbbhzY8i+RQ9DpSVpPbF7
ykQxtKXkv/ahW3KjViiAH+ghvvIhkx4xYSIc9oSwVmAl5OctMEeWUwg8Istjqz8BZeTWbf41fbNh
te7Y+YqZOwq1Sd0DbvYAD9NOZK9vlfuac0598HY+vtSBczUiKERHv1yRbcaQtZFh5wtiRrN04BLU
TD21MycBX5jYchHjPY/wIDAQAB"

Notice that this is a nice way to check if your DNS settings are as you expect them to be. To fetch the record, the selector (s) is used in conjunction with the domain (d): both were in the DKIM-signature header from the received e-mail. The ._domainkey. in between is defined by the standard (RFC6376) thus always should be there.

DKIM in practice

Not to hard in theory, is it? As an administrator you should take care of only two things: publish the TXT record in your DNS, and add the signatures to outgoing mails (and of course check the signatures on incoming mails, but that is taken care of by the same daemon most likely).

If you are interested in an exact and detailed way of setting up DKIM signing (again, for Debian/Postfix), please check out this guide at debiantutorials.com. In a nutshell, you need the dkim-filter package that provides, amongst others, dkim-genkey and the dkim-filter daemon listening at a socket on your system. In the Postfix config a milter is specified, so all mail is proxied through the DKIM signing daemon. This way, signatures on incoming mails are verified, and signatures are added before outgoing messages leave your system. Rest is up to the receiving (or intermediate) parties to check your signatures, using your published DNS records. Easy as pie, effective like a horse.

Useful links: