How SORBS ruined my life (old blog)

When this blog entry was written, Open Relay blockers like SORBS were considered the primary defence mechanism against spam email. They are still often used, but not as much as they used to be - for many of the reasons discussed in this article. SPF, DKIM & DMARC have largely taken their place.

 

 

I've had a helluva week so far, and I've got SORBS to thank for it.

SORBS (Spam and Open Relay Blocking System) is an email server reputation service. It (along with other reputation services) collect data on servers that are are either poorly configured (and open to abuse) or actively send spam. Reputation services are an excellent way of protecting against spam. They have very high hit rates and (generally) very low false positives. They also have a very low performance overhead compared with heuristic and bayesian filtering techniques. The mail servers I admister are configured to quarantine all email that is listed by either SORBS, SpamHaus or SpamCop (two other reputation services).

However, sometime last week (when I was off work sick), SORBS listed pretty much all of the hotmail, windows live, yahoo, bigpond, optusnet and myplace servers. There may have been others listed as well.

Responding to a few complaints that came in on Monday about email not being received, I began checking. Then began the sinking feeling that goes along with knowing that I'd have to change the email filtering rules, do some regression tests and then resubmit a week's work of spam through the email filters. SORBS is a very aggressive filter and I've been quite reliant on it for some time.

Less fun was trying to explain the problem to the CEO.

The first part of the quick fix was to move the SORBS check to the end of the anti-SPAM rules of the DMZ mail filter and set it to monitor only - not block. The DMZ mail server is exposed to the Internet and performs basic/quick checks only and does not look at the content of the email messages. It is highly robust and is meant as the front line of defence. It experiences an average of 90 security attacks per hour. It filters about 75% of the email traffic as inappropriate before passing the "possibly okay" messages to the second filter.

The second part was to add a check for zero day threats on the DMZ server and tag messages accordingly.

The third part was to tighten the screws a bit on the second mail filter. This meant decreasing the tolerance limits a little and added a few more checks which included looking for the tags on the email messages placed there by the DMZ mail filter. Usually, this server intercepts about 8% of messages pass to it. Messages quarantined by this server may result in a quarantine message and even a self-release option depending upon the spam score the email receives.

Now that these changes were made, I resubmitted 2143 spam quarantined messages. With the new rules in place, 1963 messages were blocked and 180 passed through to the second filter which blocked a further 68 messages leaving 112 messages sailing through to the mail server. I don't know how what the spam:ham ratio of the 112 messages was but the count seemed to be manageable so I released the remaining 14,000 quarantined messages and sent an email instructing people to forward any received spam to the spam submit mailbox for heuristic analysis. So far, I have 35 spam messages submitted by users which (if accurate) we now have a spam hit rate of 99.75% with (hopefully) a close to zero false positive rate. The hit rate has dropped to 59.4% for the DMZ server and risen to 12.5% for the second filter. Time and monitoring will determine how successful the changes have been.

Moving forward, I have written replacement rules that will no longer quarantine email based upon failed reputation. The new rules will look at SPF (Send Policy Framework - RFC4088) in addition to reputation. If the SPF check passes, the email will be accepted. If SPF fails, it will be dropped. If SPF is 'softfail' or 'none' then it will be subject to a reputation check. Any failure will drop then connection with a 46x error - Temporary Failure, with details as to why sending failed. The sending MTA will then notify the recipient that the email failed to send. The sending server then has the option of implementing SPF. I'd like at some point to add a DKIM check, but that's a reasonably difficult task.

BTW, I am heavily influenced by Ming Weng Wong's whitepaper on Messaging Anti-Abuse:

http://www.openspf.org/blobs/sender-authentication-whitepaper.pdf

 

 

Sender Policy Framework

Nearly ten years ago, Meng Weng Wong of pobox.com proposed an anti-spam methodology for SMTP email based upon the sending email domain by publishing a list of authorised mail sending agents in DNS. What followed was the highly controversial IETF MARID conference that produced four experimental RFCs including RFC4408 - Sender Policy Framework. That RFC is currently being revised by the SPFBIS working group which will ratify SPF as an IETF standard.

What the SPF experiment has demonstrated is that SPF will never be a complete solution to the problem of spam. However, it has been highly effective in mitigating spam and reducing the load on anti-spam engines. It is believed that SPF in conjunction with two other anti-spam measures (DKIM & DMARC) should be able to virtually eliminate spam with needing to check the content of email measures.

Part of the problem with SPF being widely successful in preventing spam is that most email systems administrators misunderstand SPF and implement it either poorly or incorrectly.

SPF is a "sender policy". It allows the sending system to define clearly what is and what is not a legitimate source of email. Before SPF, such checks were done using RDNS lookups - which are costly, expensive and simply do not work correctly if the domain is using an external email provider.

One thing the Marid conference did settle once and for all is who has the responsibility for the delivery of email.

The standard rule for email delivery post-Marid is that it is the sending server's responsibility to deliver the email until an SMTP code 250 is received from the target mail server. After that, it is the recipient MTA (Message Transfer Agent) that accepts responsibility to deliver the email.

That may sound really simple, however there are an enormous number of consequences resulting from this. For starters it means that email messages can no longer be trashed by anti-spam engines without notification to either the sender or the recipient. More importantly, if the receiving MTA cannot deliver the email after issuing a 250 to the sender, it now MUST issue an NDR (Non-Deliverable Report). However, it must do this without risking backscatter (sending a forged message back to a real recipient). MTAs that backscatter will quickly find themselves on a global blacklist.

The corollary to this is that the receiving MTA MUST have some way of verifying a legitimate source of email: Enter SPF.

Creating an SPF record for an email domain is quite simple: all you need to do is publish a DNS TXT record similar to the following:

<domain name> TXT "v=spf1 mx a ~all"

The "v=spf1" bit is simply identifying the string as an SPF record. What follows is a list of acceptable and unacceptable hosts and the degree to which they are acceptable. The host list is optionally prefixed by a character which designates the level of acceptance or non-acceptance as follows:

'+'  (or no prefix) PASS
'-' FAIL
'?' NEUTRAL
'~' SOFTFAIL

SPF PASS is essentially a guarantee that the email came from a legitimate mail server for the sending domain and standard spam tests may be bypassed.

SPF FAIL is a statement that the email did not come from a legitimate mail server and may safely be rejected.

SPF NEUTRAL means the sender policy has nothing to say about the email. Treat it as though no SPF existed.

SPF SOFTFAIL means the email probably did not come from a legitimate mail server, but the sender cannot 100% guarantee that it didn't.

There are two other SPF Results: TempError and PermError.  I have made comments on how these results are interpreted in the SPFBIS Working Group.

The above record could have been written as "v=spf1 +mx +a ~all". However, since '+' is assumed which no symbol is given, the '+' is rarely used.

The basic acceptable fields used are:

mx - all hosts defined by the mx record for the domain.
a - all hosts with an a record for the domain
ip4: - all hosts within the listed netblock (or ip address)
ip6: - all hosts within the listed ipv6 netblock
include: - include the SPF record for the listed email hosting domain
all - matches everything

So for our listed example, the receiving mail server would tested the sending servers IP address and check first to see if it matches the mx record for the sending domain. If it does, the return an SPF PASS result; otherwise test to see if it resolves to a 'a' record for the sending domain and if that fails, return an SPF SOFTFAIL.

I see a lot of SPF records that end in '-all'. I regard that as a very bold statement. It basically gives me the right to completely reject email messages with an SMTP code 550 if they fail the SPF check. The sender who publishes such a statement has to be completely SURE that no legitimate email will come from a host other than the one's listed. For the vast majority of organisations, that guarantee is simply impossible to make. The reason is that the sender cannot control email forwarding.

When you forward an email, the FROM address remains the same as does the SMTP MAIL FROM: line. This is what SPF uses to evaluate the sending domain against the connecting MTA. In the case of forwarded mail, they will not match and SPF will FAIL, resulting in a rejected message. However, now the intermediate server has a problem. It has accepted responsibility to deliver the mail and cannot do so. Under the post-MARID rules it must deliver an NDR, however with no current connection to the sending MTA the only way it can do so is send a DSN email (Delivery Status Notification) and risk backscatter.

There is a solution to this mess called the Sender Rewriting Scheme (SRS). This simply involves encapsulating the original destination address within the MAIL FROM: line in such a way that SPF can check both the source email addresses. It's a great idea that suffers from a simple three-part problem:

1) The sender has no control over whether SRS is implemented at the original destination or not.
2) Since SPF is a "sender policy", this means the sender still cannot make definitive statements in their SPF record.
3) To date, very few commercial MTAs implement SRS.

This means we cannot rely on SRS even if we implement SRS ourselves. You cannot know for certain how many of your email recipients implement SRS, nor can you know if they are forwarding email. This means, at best we can publish a "~all" at the end of our SPF records. Anything else risks both the possibility of incorrect non-delivery of an email and unnecessary backscatter.

At around the same time that SPF was floated, both Google and Yahoo came up with the idea of signing the email envelope using public-key cryptography. Yahoo's was called DomainKeys and Google's was called Domain Key Identified Mail (DKIM). Both methods were similar, but DKIM has become the standard. It works similarly to SPF with the public key published in a DNS TXT record. The good thing about DKIM is that it is not broken by forwarding since it looks at the envelope.

Mailing Lists, however, do break DKIM. This is because mailing lists (by their nature) modify the email envelope.

This gives us hope, because now we can use both SPF and DKIM to evaluate email. If either method passes, we can now regard the email as legitimate. Only if both fail do we stamp FAIL on the email message and reject it.

The problem with this idea is the take up of DKIM is very low. Few commercial mail servers can support DKIM. Publishing a DKIM record is also more difficult than publishing an SPF record. So, while the take up rate for SPF is around 40% (accounting for more that 60% of legitimate email), the take up rate for DKIM is around 5%. So, even if you do publish a DKIM record and stamp all outgoing email with a DKIM hash, chances are that few of the recipient MTAs will do anything about it.

Enter Domain Based Message Authentication, Reporting and Conformace or DMARC.

A DMARC record is published in DNS like SPF and DKIM. The difference is that DMARC provides a mechanism to report back to the original sending MTA to determine if the email is legitimate. In addition, it provides a "fuzzy" method for temporary rejection of non-conformant email similar to grey-listing, but with a more intuitive and sender-determined policy for doing so. With a DKIM record, the sender policy is not just isolated to SPF or DKIM alone. The sender can define a rejection policy based upon a combination of checks and receive reports back on rejected messages. Thus it removes all the guess work associated with message rejection.

Pretty much all the major players are now behind DMARC: Google, Yahoo, Microsoft, AOL, Facebook all currently implement DMARC even before the specification has been ratified by the IETF. See http://www.dmarc.org

In summary, if you administer an email system, you SHOULD be publishing an SPF record at the very least. You should also be implementing some form of SPF checking - but not too aggressively (making sure you whitelist regulars). If possible, look at implementing DKIM with a view to eventually implementing DMARC. As for SRS, if you can implement it, then do so. It will reduce the level for SPF rejections on forwarded email.

I will write another article another time on practical SPF implementations. Oh, and if you are running MailMarshal less than version 7, then upgrade it asap. The SPF implementation is broken. It took me nearly two weeks to convince them of this. Even with the fix they implemented, it still breaks under certain circumstances.