SPF is the technology that assures that mail only arrives from authentic
senders. One problem remains that slows down its introduction as a
hard filter, and that is email forwarding. This article nails the
integration of SPF with forwarding.
The ARPA2 infrastructure uses user names with aliases for
all sorts of purposes,
including for registration in groups or for occupancy of a role.
At the same time, there is a (visibly related) name for a group member,
and for a role occupant, and these have a one-to-one coupling with an
alias. So, user john may speak of himself as john+admin while the
sysadm role knows him as sysadm+mrsmith. Note the mechanism of
attaching an extra label to the of name a user, role or group to turn it
into an alias, occupant or member.
In support of our Bring Your Own IDentity design, we have always permitted remote identities to participate in groups, and said that there should be a facility to turn it into a pseudonym for use with the external entity, for reasons of privacy. What we haven't done het, is connect the dots.
What we need, next to the aliases for participation in groups and roles,
is an alias for participation in other domains. Such an alias would count
as an external identity and it would take whatever form is used under
that external domain, though some general user@domain.name form would be
assumed. By declaring this, we can choose how to approach the other
service; as one of us or as one of them. Imagine email, where this construct
offers a choice between sending an email from the local realm, or the external
one.
Forwarding and SPF: Resolving the Conflict
Forwarding is still done a lot without modification to the envelope sender address, causing the receiving mailbox to reject the submission because the original sender is kept, but the email comes from another source. This is in some ways an internal responsibility, of the forwarding party, as they generally control both the forwarding step and the receiving mailbox. There are several solutions:
- Expensively — do not forward, but setup a mailing list or ARPA2 group
- Hopefully — introduce SRS on all those (very cheap) forwarding service
- Impossibly — disable SPF filtering on the receiving mailbox
- Perfectly — add an exception to the SPF filter on the receiving mailbox
The last option is presented here, and puts the work in your mailbox, where there already is more complexity and a more commercial breadth. The exception is also surprisingly simple to implement, given that external identities are explicitly setup for each of the recipients:
- When an email enters, lookup its envelope receiver
- From its external identities, collect the external domain(s)
- Mail may enter under SPF of those external domain(s)
- Mail may enter under SPF of the envelope sender domain
Note how this allows refined SPF-bypassing for the case of email forwarding. All it requires is a mapping, setup by the user, to acknowledge forwarded email from an external identity to a receiving mailbox. Specifically, there are no IP addresses to be entered, all that is needed are the names of forwarded domains — and that ought to be within the user's grasp!
Also note that domain names may be learnt from user actions who train
messages as not spam even though they overruled SPF due to forwarding.
All in all, this could be a very user-friendly approach to making SPF
function at full strength, with -all at its tail to reject anything
out of the ordinary; anything but forwarded email, that is.
Do Try This At Home!
You can now:
- Be sure to have SPF active on your forwarded domains
- Use email forwarding from forwarded domains to mailboxes with Perfect SPF
- Setup SPF on the forwarded domain to include the mailbox domain
- Receive email in the mailbox or under a sub-box for an alias
- You can now answer with the mailbox address or a forwarded address
- You can now send from the mailbox domain or the forwarded domain
It is still valuable if email forwarding adapts, because not every forwarded mailbox will follow this Perfect SPF strategy, and we do still want to be able to couple everything to everything.
What this does is give more control to the end user. That is basically our thing and a task we take rather seriously.
What this can also do, is help to speed up the introduction of SPF in all corners of the Internet — well, at least in all corners that we care about. Spammers are perfectly welcome to reject the technology and end up sending mail among themselves, of course.
Email is currently somewhat instable as a result of spam filtering, and the state of SPF is that it cannot be enforced hard, upon entry. This means that it ends up in spam filtering, which also means that bounce messages are dropped. Such messages are a nuisance to the user when they occur, but it is much more of a nuisance when email gets silently discarded, as is currently the case.
Given that the mailbox owner can now take responsibility of email forwarding to his mailbox (and because email lists behave properly and need no special care), it is possible to make the filter in the mailbox behave more strictly than required by sender domains! And that may mean that hard bounces can be produced once more. Senders with faulty setups can correct their announced SPF settings, and generally the World is moving forwards. Perhaps most importantly, the user is enabled to tighten his control over spam in a way that is very direct, and easy to understand. Suddenly, it is valuable for end users to understand the game of mail, and play it by the rules!
Update 2017-08-28:
As indicated, SRS is useful but may be too much work for the cheapest of forwarding deals. Indeed, there are excellent solutions but they require intricate understanding of one's MTA, not always the thing one is waiting for, and return traffic may always be captured, which potentially increases traffic. For this reason, we launched our own SRSrelay that acts just as a bump in the wire, an MTA on the way out to the upstream/smarter MTA. This is easier to configure, and it can be used with any other MTA because it speaks the general SMTP protocol. It is also usable without an upstream MTA; in that case the mail can be made to loop back into the forwarder's MTA.
We believe SRS is powerful because it holds a hint that mail has been
forwarded; the aforementioned settings on forwarding can in fact be
automated recognising the =SRS beginning of an envelope MAIL FROM
address, treating it as a hint of the original email address (no more
than a hint, as the signature is too weak and we cannot verify it)
and try sending a verification email through the originating service
when the user indicates that this is indeed an originating address for
the email. Note that the DKIM approach of comparing the From: header
to the envelope MAIL FROM should work perfectly with this reconstituted
address from SRS. Once the user agrees, we can even verify the path by
sending a test email through the system, with a self-signed instruction
to accept the forwarded route.
Even without SRS, it is possible to collect recurring discrepencies
between From: headers and the envelope MAIL FROM address. When
the same thing happens again and again, it is either spam or a sign
of forwarding. A similar procedure can then be followed, through a
suggestion to the user and a validating email sent to the address in
the From: header. Whatever the output would become a pattern for
spam-versus-ham detection. The trick now is to avoid doing this for
every possible attacker; to remedy that, the request is best posted
in a quarantine mail folder which the user occasionally scans for
email that was falsely classified of being spam.
Update 2017-08-29:
Don't be a forwarder — be a transporter.
Email forwarding is usually a straightforward translation of incoming email addresses or domains into outgoing variants. The outgoing address appears on the envelope, but the email itself is unchanged.
As indicated, the best solution is to make the final receiving mailbox aware of additional email addresses that are acceptable to it. It is more difficult than one would expect when forwarding takes place, because the outgoing recipient address looks like the expected address.
Given that a mailbox service is aware of any aliases for its receiving accounts, it would be much simpler to distinguish that email for such an alternate account is entering the system. Especially when a few steps of forwarding are made before the email arrives at the mailbox.
It is therefore a service to the owner of the forwarded address and the mailbox to not translate to the mailbox address, but instead to retain the forwarded address as an envelope recipient.
The way this would work is, to say it is Postfix terms, to move from a
virtual domain map to a transport. Definitions in /etc/postfix/virtual
that looked like
@example.com @example.org
are replaced with entries in /etc/postfix/transport that look like
example.com smtp:example.org
This triggers the SMTP client for the given domain name, forwarding
its entries to the MX for example.org without changing the envelope
name.
As a variation, if user names are to be substituted, a virtual map
info@example.com john@example.org
would stay in the same domain, to be transported as specified above, with the following virtual map for the name change within the domain:
info@example.com john@example.com
Note that various email filters may be supportive of this scheme, if
they allow alias domains. In such a setup, the example.com domain
would simply be configured as an alias to the primary domain example.org.
The one thing that this leaves open is the potential of mail loops. If you transport the email to something that functions like an open relay, you may receive the email back that you just transported, when your server is defined as its MX. You should detect such loops and take proper action if you choose this model. This approach is really just firt for delivery to those mail box servers that actuall embrace email from such a primary MTA.
Go Top