Mail is severely hampered by spam, as we all know. Interestingly,
some spam prevention tactics are applied so softly that they work
against a reliable mail system. Here is a tale from the crypt.
Mail is severely hampered by spam, as we all know. Interestingly, some spam prevention tactics are applied so softly that they work against a reliable mail system. Here is a tale from the crypt.
When I give you a piece of paper with my (new) phone number, you have no reason to doubt its validity. When a mutual friend gives you my new phone number, you are will probably also accept it. But how about if a stranger gave you my new phone number? You would probably reject it, and you should.
This logic has not been built into the email system from its early onset. Only recently have we starting adding this, using a technology called SPF. The working principle of SPF is to look at an incoming email, and compare the (envelope) sender address to the address from which the email was submmitted. This address must be on a list of addresses permitted to send on behalf of the sender's domain name.
Simple enough, right? Nope.
Email is an incredibly complicated system. It does things like forwarding,
which means that a server receiving mail for
firstname.lastname@example.org can forward
email@example.com. The sender may be from an unrelated domain,
and this sender is traditionally mentioned while the email is being
forwarded. This means that the mail server for the final destination
firstname.lastname@example.org sees an email
arriving from the mail server for
email@example.com, but the original
sender's domain may not be permitting the
firstname.lastname@example.org server as
This can be remedied. A system known as SRS has been defined to replace the (envelope) sender address to fix this. This sender address is a legitimate address that stems from the forwarding mail server, but it is coded so that when a bounce occurs, the original sender can be recovered and the bounce can be relayed back there.
Most email lists use SRS or VERP (though not all, there is a lot of crap out there) and should therefore not run into problems with SPF, so lists usually work. But email forwarding is not commonly given an SRS treatment, and it is so common that it has to be dealt with. Somehow.
Killing You Softly
To handle these problems, SPF has what is called a soft-fail option. Under this option, a mail server that is not a legitimate sender for an email address can still try to forward to a mailbox that checks SPF.
If SPF were completely black-and-white, an incoming mail server could refuse to take an email in, causing to a bounce that informs the originator that something went wrong. Bounces hold valuable technical information, so aside from a nuisance to the sender it is also useful. More useful than dropping the email anyway!
SPF is designed to constrain the routes that email travels to a path through valid domain mail servers, and ideally it would be checked black-and-white and upon entry. This works before forwarding, but is guaranteed to fail after a forwarding step. So, when a mailbox takes in forwarded email, which is ought to be a deliberate choice by the owner of that mailbox, exemption of SPF checking for these forwarded addresses should apply, and an email should get in anyway. This is a pattern that we see more often: party A must be willing to surrender data to party B, but it only becomes reality when party B is willing to accept from party A.
But in spite of much of it being automated, the setup of forwarding is rarely this tidy. To compensate for that flaw, many sites have setup SPF with the soft-fail option. Meaning, SPF is not cut off hard at the entrance port, but it merges with the statistics of a spam filter. This makes SPF look like an approach for spam filting. Unfortunately, spam filters are rather unwilling to send feedback to the sender, and all of a sudden SPF becomes a puzzle. Moreover, it becomes a game of chance, because the other variables in spam analysis will very. So what we have now is a system that is setup in a flawed manner, and to compensate it we made email delivery a game of chance without any feedback. Could we have broken the system any more? Just imagine debugging that, especially when the remote is another owner's domain and possibly manned with unknowing users!
It is the softness that makes SPF unworkable. And it is the large networks that lead the way in breaking email by using it. One might say that they have a vested interest to guide users away from communication systems that span the Internet and into their own walled-garden systems; but it is more likely a matter of making peace by being conservative. Meanwhile this is breaking the email system as a whole, so end users should not want to put up with this level of conservatism.
To get specific about the current-day setups:
- Gmail and
specify SPF records for their domains,
~alldenoting soft-fail characteristics. As a result, email sent from these domains will not be fully reliable in reaching its targets.
specifies SPF records for their domains, but has a
?allnever-fail characteristic. The result of this is a complete absense of protection against people using the email addresses under this domain.
It is the small, technical-savvy domains that do better by
-all policy. They
will specify hard restrictions on the mail servers that may
claim their email addresses. But, in light of SPF being used
so often under soft-fail conditions, it is often incorporated
into spam filtering, with the result of bounces being hushed.
Luckily, the ARPA2 project that aims to make such techy-savvy systems available to normal end users.
The ARPA2 Solution
The ARPA2 solution to SPF strictness is simple; never use (envelope) sender addresses except those that are locally hosted.
A good example are the groups that we defined for ARPA2; we allow local and remote users to be members of a group, and a group can act like as a sort of implied mailing list. And if you need forwarding, you can define a group with (initially) just one member receiving the email.
So, how do we handle groups? Well, in a few stages, really.
When Carl sends to his group of weavers, he does so under his dedicated group member name. If the group is called
email@example.com his member name extends the local part, like in
firstname.lastname@example.org. Behind this member address is a remote or local address, an alias, or anything else; but as far as the group goes, all his email comes from
Mail sent to the group and originating from Carl's underlying email address is modified to the
email@example.com address, setting both the local part and the domain. This is done for envelope and header addresses alike, to provide some privacy to the member.
Since the email is to the
weaversgroup, its members are now listed. The sender
weavers+carlwill be among the members found, but this one may be dropped. Other members like
weavers+marywould remain as a recipient. So the mail is forwarded to any such addresses.
The envelope recipient address for member Mary is translated into her underlying address, which again may be a remote or local address, or alias, or whatever else. As an example, let's say that it is a remote address
firstname.lastname@example.org(note the different domain). The original group address remains in the email headers, which is good for replies, but Mary's own address is now an envelope recipient.
We now relay the email to the mail server for Mary's envelope address, so the one serving
example.net; since Carl's envelope sender address is still tied to the local domain, there should be no problem with SPF. We can make a hard claim about SPF if we trust the recipient to not forward it any further — which is not strictly needed because other list members only see the member address
When Mary replies to Carl's mail, she may either send to
weavers+carl. The first case works just like described above; the direct email to Carl works almost the same, except for the expansion of the group, which is replaced with a lookup of Carl's underlying address.
Do Try This At Home!
A simpler form of this scheme can be setup in a Postfix mail server. It takes three steps to do this.
Define a virtual email address for the group in your
/etc/postfix/virtual file, and be sure it is named
virtual_alias_maps. The contents for this
example would be
# Envelope recipient map for weavers group email@example.com firstname.lastname@example.org email@example.com
When group members contact members on their member
address, we should forward the traffic to the
underlying address in the same
virtual map with
# Envelope recipient map for weavers members firstname.lastname@example.org email@example.com firstname.lastname@example.org email@example.com
The last thing we need to facilitate is the local
handling of member's underlying addresses. This is done
/etc/postfix/canonical file, which must be
listed in the
canonical_maps setting to activate
# Envelope and header map for weavers members firstname.lastname@example.org email@example.com firstname.lastname@example.org email@example.com
This allows group members to contact each other
directly, at their respective group member addresses,
going through the mail server that defines the
corresponding domain name. It also allows them
(and anyone else, if you don't add access control
to the group address) to send to the address of
There, you've done it. You created a simple DIY email list, without getting in trouble with SPF.
On top of that, there is (some) privacy because the
canonical map replaces all addresses; it must do
that for the envelope sender address that will later
be the envelope sender for an outgoing message from
this mail server; it may do that for the envelope
recipient since a
virtual mapping will correct
that later on; and it may do that for sender and
recipient headers in light of their membership
privacy. Do note however, that there will still be
Received: headers and possibly more traces of the
original addresses, so privacy is not complete.
There are some limitations; you cannot subscribe multiple addresses to more than one list, for example. Interestingly, we were not planning to do it anyway; we wanted a user alias for every membership for any group, so there would be a one-to-one translation between the user's idea of his identity, and the group's.
Another limitation is that replies to the list will
lead to duplicates. These can be filtered out in a
IMAP server or mail client, but it is better when
they are avoided. That is not possible with this
simple setup in Postfix, but the
approach of a mailing list
is to remove addresses also found in
headers after expansion of the group to its members.
Note that Postfix also won't let you filter access
to the group to be constrained to just the group
members. This is because none of Postfix's maps
looks at the combined sender and recipient. Such
things are all more advanced, and need to go into
content_filter with custom code.
Forwarding is a bit different, because it entails arbitrary senders. To make this work, you really should use SRS to change the sender address. There is no premade solution for this in Postfix, though a plugin for SRS can be used. Though not ideal, as it applies SRS to all traffic, it does at least do it in the cases where it is required — and probably does no harm in others.
We are hoping to, someday, see a Postfix setting
smtp_forwarding_srs = yes to indicate that any
forwarding done by Postfix, perhaps defined as
sending from a domain that is not in
mydestination or a similar variable,
should make the SMTP client use SRS. A similar
setting for LMTP may also be needed.