Mail Routing 1: Doing it Well and Not-So-Well

Mail is severely hampered by spam, as we all know. Interestingly, some spam prevention tactics are applied so softly that they work against a reliable mail system. Here is a tale from the crypt.

When I give you a piece of paper with my (new) phone number, you have no reason to doubt its validity. When a mutual friend gives you my new phone number, you are will probably also accept it. But how about if a stranger gave you my new phone number? You would probably reject it, and you should.

This logic has not been built into the email system from its early onset. Only recently have we starting adding this, using a technology called SPF. The working principle of SPF is to look at an incoming email, and compare the (envelope) sender address to the address from which the email was submmitted. This address must be on a list of addresses permitted to send on behalf of the sender's domain name.

Simple enough, right? Nope.

Email Complexions

Email is an incredibly complicated system. It does things like forwarding, which means that a server receiving mail for info@example.org can forward it to john@example.com. The sender may be from an unrelated domain, and this sender is traditionally mentioned while the email is being forwarded. This means that the mail server for the final destination address john@example.com sees an email arriving from the mail server for info@example.org, but the original sender's domain may not be permitting the info@example.org server as its origin.

This can be remedied. A system known as SRS has been defined to replace the (envelope) sender address to fix this. This sender address is a legitimate address that stems from the forwarding mail server, but it is coded so that when a bounce occurs, the original sender can be recovered and the bounce can be relayed back there.

Most email lists use SRS or VERP (though not all, there is a lot of crap out there) and should therefore not run into problems with SPF, so lists usually work. But email forwarding is not commonly given an SRS treatment, and it is so common that it has to be dealt with. Somehow.

Killing You Softly

To handle these problems, SPF has what is called a soft-fail option. Under this option, a mail server that is not a legitimate sender for an email address can still try to forward to a mailbox that checks SPF.

If SPF were completely black-and-white, an incoming mail server could refuse to take an email in, causing to a bounce that informs the originator that something went wrong. Bounces hold valuable technical information, so aside from a nuisance to the sender it is also useful. More useful than dropping the email anyway!

SPF is designed to constrain the routes that email travels to a path through valid domain mail servers, and ideally it would be checked black-and-white and upon entry. This works before forwarding, but is guaranteed to fail after a forwarding step. So, when a mailbox takes in forwarded email, which is ought to be a deliberate choice by the owner of that mailbox, exemption of SPF checking for these forwarded addresses should apply, and an email should get in anyway. This is a pattern that we see more often: party A must be willing to surrender data to party B, but it only becomes reality when party B is willing to accept from party A.

But in spite of much of it being automated, the setup of forwarding is rarely this tidy. To compensate for that flaw, many sites have setup SPF with the soft-fail option. Meaning, SPF is not cut off hard at the entrance port, but it merges with the statistics of a spam filter. This makes SPF look like an approach for spam filting. Unfortunately, spam filters are rather unwilling to send feedback to the sender, and all of a sudden SPF becomes a puzzle. Moreover, it becomes a game of chance, because the other variables in spam analysis will very. So what we have now is a system that is setup in a flawed manner, and to compensate it we made email delivery a game of chance without any feedback. Could we have broken the system any more? Just imagine debugging that, especially when the remote is another owner's domain and possibly manned with unknowing users!

It is the softness that makes SPF unworkable. And it is the large networks that lead the way in breaking email by using it. One might say that they have a vested interest to guide users away from communication systems that span the Internet and into their own walled-garden systems; but it is more likely a matter of making peace by being conservative. Meanwhile this is breaking the email system as a whole, so end users should not want to put up with this level of conservatism.

To get specific about the current-day setups:

• Gmail and HotMail specify SPF records for their domains, with ~all denoting soft-fail characteristics. As a result, email sent from these domains will not be fully reliable in reaching its targets.
• Yahoo specifies SPF records for their domains, but has a ?all never-fail characteristic. The result of this is a complete absense of protection against people using the email addresses under this domain.

It is the small, technical-savvy domains that do better by setting a -all policy. They will specify hard restrictions on the mail servers that may claim their email addresses. But, in light of SPF being used so often under soft-fail conditions, it is often incorporated into spam filtering, with the result of bounces being hushed.

Luckily, the ARPA2 project that aims to make such techy-savvy systems available to normal end users.

The ARPA2 Solution

The ARPA2 solution to SPF strictness is simple; never use (envelope) sender addresses except those that are locally hosted.

A good example are the groups that we defined for ARPA2; we allow local and remote users to be members of a group, and a group can act like as a sort of implied mailing list. And if you need forwarding, you can define a group with (initially) just one member receiving the email.

So, how do we handle groups? Well, in a few stages, really.

1. When Carl sends to his group of weavers, he does so under his dedicated group member name. If the group is called weavers@example.com then his member name extends the local part, like in weavers+carl@example.com. Behind this member address is a remote or local address, an alias, or anything else; but as far as the group goes, all his email comes from weavers+carl@example.com.

2. Mail sent to the group and originating from Carl's underlying email address is modified to the weavers+carl@example.com member address, setting both the local part and the domain. This is done for envelope and header addresses alike, to provide some privacy to the member.

3. Since the email is to the weavers group, its members are now listed. The sender weavers+carl will be among the members found, but this one may be dropped. Other members like weavers+mary would remain as a recipient. So the mail is forwarded to any such addresses.

4. The envelope recipient address for member Mary is translated into her underlying address, which again may be a remote or local address, or alias, or whatever else. As an example, let's say that it is a remote address mary.mungool@example.net (note the different domain). The original group address remains in the email headers, which is good for replies, but Mary's own address is now an envelope recipient.

5. We now relay the email to the mail server for Mary's envelope address, so the one serving example.net; since Carl's envelope sender address is still tied to the local domain, there should be no problem with SPF. We can make a hard claim about SPF if we trust the recipient to not forward it any further — which is not strictly needed because other list members only see the member address weavers+mary@example.com anyway.

6. When Mary replies to Carl's mail, she may either send to weavers or to weavers+carl. The first case works just like described above; the direct email to Carl works almost the same, except for the expansion of the group, which is replaced with a lookup of Carl's underlying address.

Do Try This At Home!

A simpler form of this scheme can be setup in a Postfix mail server. It takes three steps to do this.

Virtual Group. Define a virtual email address for the group in your /etc/postfix/virtual file, and be sure it is named in the virtual_alias_maps. The contents for this example would be

# Envelope recipient map for weavers group
weavers@example.com  mary.mungool@example.net
                     carl.czorski@example.org

Individual Forwarding. When group members contact members on their member address, we should forward the traffic to the underlying address in the same virtual map with

# Envelope recipient map for weavers members
weavers+mary@example.com  mary.mungool@example.net
weavers+carl@example.com  carl.czorsky@example.org

Incoming mapping. The last thing we need to facilitate is the local handling of member's underlying addresses. This is done in the /etc/postfix/canonical file, which must be listed in the canonical_maps setting to activate

# Envelope and header map for weavers members
mary.mungool@example.net  weavers+mary@example.com
carl.czorski@example.org  weavers+carl@example.com

This allows group members to contact each other directly, at their respective group member addresses, going through the mail server that defines the corresponding domain name. It also allows them (and anyone else, if you don't add access control to the group address) to send to the address of the weavers group.

There, you've done it. You created a simple DIY email list, without getting in trouble with SPF.

On top of that, there is (some) privacy because the canonical map replaces all addresses; it must do that for the envelope sender address that will later be the envelope sender for an outgoing message from this mail server; it may do that for the envelope recipient since a virtual mapping will correct that later on; and it may do that for sender and recipient headers in light of their membership privacy. Do note however, that there will still be Received: headers and possibly more traces of the original addresses, so privacy is not complete.

There are some limitations; you cannot subscribe multiple addresses to more than one list, for example. Interestingly, we were not planning to do it anyway; we wanted a user alias for every membership for any group, so there would be a one-to-one translation between the user's idea of his identity, and the group's.

Another limitation is that replies to the list will lead to duplicates. These can be filtered out in a IMAP server or mail client, but it is better when they are avoided. That is not possible with this simple setup in Postfix, but the approach of a mailing list is to remove addresses also found in To: and Cc: headers after expansion of the group to its members.

Note that Postfix also won't let you filter access to the group to be constrained to just the group members. This is because none of Postfix's maps looks at the combined sender and recipient. Such things are all more advanced, and need to go into a content_filter with custom code.

Forwarding is a bit different, because it entails arbitrary senders. To make this work, you really should use SRS to change the sender address. There is no premade solution for this in Postfix, though a plugin for SRS can be used. Though not ideal, as it applies SRS to all traffic, it does at least do it in the cases where it is required — and probably does no harm in others.

We are hoping to, someday, see a Postfix setting smtp_forwarding_srs = yes to indicate that any forwarding done by Postfix, perhaps defined as sending from a domain that is not in mydestination or a similar variable, should make the SMTP client use SRS. A similar setting for LMTP may also be needed.