How SORBS ruined my life (old blog)

When this blog entry was written, Open Relay blockers like SORBS were considered the primary defence mechanism against spam email. They are still often used, but not as much as they used to be - for many of the reasons discussed in this article. SPF, DKIM & DMARC have largely taken their place.

 

 

I've had a helluva week so far, and I've got SORBS to thank for it.

SORBS (Spam and Open Relay Blocking System) is an email server reputation service. It (along with other reputation services) collect data on servers that are are either poorly configured (and open to abuse) or actively send spam. Reputation services are an excellent way of protecting against spam. They have very high hit rates and (generally) very low false positives. They also have a very low performance overhead compared with heuristic and bayesian filtering techniques. The mail servers I admister are configured to quarantine all email that is listed by either SORBS, SpamHaus or SpamCop (two other reputation services).

However, sometime last week (when I was off work sick), SORBS listed pretty much all of the hotmail, windows live, yahoo, bigpond, optusnet and myplace servers. There may have been others listed as well.

Responding to a few complaints that came in on Monday about email not being received, I began checking. Then began the sinking feeling that goes along with knowing that I'd have to change the email filtering rules, do some regression tests and then resubmit a week's work of spam through the email filters. SORBS is a very aggressive filter and I've been quite reliant on it for some time.

Less fun was trying to explain the problem to the CEO.

The first part of the quick fix was to move the SORBS check to the end of the anti-SPAM rules of the DMZ mail filter and set it to monitor only - not block. The DMZ mail server is exposed to the Internet and performs basic/quick checks only and does not look at the content of the email messages. It is highly robust and is meant as the front line of defence. It experiences an average of 90 security attacks per hour. It filters about 75% of the email traffic as inappropriate before passing the "possibly okay" messages to the second filter.

The second part was to add a check for zero day threats on the DMZ server and tag messages accordingly.

The third part was to tighten the screws a bit on the second mail filter. This meant decreasing the tolerance limits a little and added a few more checks which included looking for the tags on the email messages placed there by the DMZ mail filter. Usually, this server intercepts about 8% of messages pass to it. Messages quarantined by this server may result in a quarantine message and even a self-release option depending upon the spam score the email receives.

Now that these changes were made, I resubmitted 2143 spam quarantined messages. With the new rules in place, 1963 messages were blocked and 180 passed through to the second filter which blocked a further 68 messages leaving 112 messages sailing through to the mail server. I don't know how what the spam:ham ratio of the 112 messages was but the count seemed to be manageable so I released the remaining 14,000 quarantined messages and sent an email instructing people to forward any received spam to the spam submit mailbox for heuristic analysis. So far, I have 35 spam messages submitted by users which (if accurate) we now have a spam hit rate of 99.75% with (hopefully) a close to zero false positive rate. The hit rate has dropped to 59.4% for the DMZ server and risen to 12.5% for the second filter. Time and monitoring will determine how successful the changes have been.

Moving forward, I have written replacement rules that will no longer quarantine email based upon failed reputation. The new rules will look at SPF (Send Policy Framework - RFC4088) in addition to reputation. If the SPF check passes, the email will be accepted. If SPF fails, it will be dropped. If SPF is 'softfail' or 'none' then it will be subject to a reputation check. Any failure will drop then connection with a 46x error - Temporary Failure, with details as to why sending failed. The sending MTA will then notify the recipient that the email failed to send. The sending server then has the option of implementing SPF. I'd like at some point to add a DKIM check, but that's a reasonably difficult task.

BTW, I am heavily influenced by Ming Weng Wong's whitepaper on Messaging Anti-Abuse:

http://www.openspf.org/blobs/sender-authentication-whitepaper.pdf

 

 

Email status check

Okay, you're offsite and someone rings up to say the email system isn't working. Now, you KNOW that nine times out of ten the email system is working perfectly - it's just something the user is doing wrong. How can you quickly check to see if email is working without logging into the servers? Well, you could simply send an email from your gmail account to your work account and vice versa. That would be a good indication that everything is working, but if you don't get the email, it tells you absolutely nothing other than something might be wrong with the email system.

Most enterprise mail systems have a number of servers involved in the generation, transmission and reception of email. In generic terms we have the following elements:

MTA - Mail Transfer Agent
MDA - Mail Delivery Agent
MSA - Mail Submission Agent
MRA - Mail Retrieval Agent
MFA - Mail Filtering Agent
MUA - Mail Usage Agent

Many sysadmins may exclaim at this point "Hang on - I don't remember there being that many elements to email delivery!" and the reason for that is we are now in the post-Marid world of Internet based mail as explained by RFC5598 - 2009. Quite simply: Things are different now. If you are running a mail system that was setup before this time and not updated, chances are that you aren't RFC-compliant to the IETF standard. If you're running MS Exchange out-of-the-box, then you definitely aren't standards compliant. However, making your email system RFC-compliant is the stuff of another article...

RFC5598 divides the various agents into their respective areas of responsibility called "Responsible Actor Roles". These are:

 - User
 - Message Handling System (MHS)
 - ADministrative Management Domain (ADMD)

The traditional flow of email was:

MUA -> MTA -> .... -> MTA -> MUA

Now, the email flow is more commonly:

MUA -> MSA -> MTA -> ... -> MTA -> MFA -> MDA --> MRA --> MUA

where -> is a push operation and --> is a pull operation.

Obviously, in such a system there are a number of elements that can go wrong and be described as "the email system is down".

On email systems I administer, I usually create a dummy account called "Email-Check". At its most basic level, you set it up with an Out of Office reply that says "Email is working". However it doesn't end there. Each point in your message reception system can be setup to respond with diagnostics on each component. A fully working system will received replies from each component in the chain. In the second example, if you send your email to email-check@your-domain and receive a reply from the MTA and MFA, but not the MDA or the MRA, then you can reasonably assume the problem lies with the MDA - that should be the place you start looking.

Practical Examples

MailMarshal

1. Write a rule in MailMarshal that triggers when the to: address is email-check. Have the rule execute as an external command the file "mail-check.cmd" and pass the following parameters to it: servername@domain {ReplyTo} {SenderIP} {HelloName}

2. Write email-check as follows:

@echo off
c:
cd \scripts

echo Email check for [servername] > mmcheck.txt
echo. >> mmcheck.txt
echo.|time|grep current  >> mmcheck.txt
echo.>>mmcheck.txt
echo [Servername] Mail Marshal Service Information >>mmcheck.txt
echo. >>mmcheck.txt
start /wait msinfo32.exe /categories +SWEnvServices /report msinfo.txt
type msinfo.txt | grep MailMarshal >> mmcheck.txt
echo. >>mmcheck.txt
echo Sending IP  : %3 >>mmcheck.txt
echo Helo Name   : %4 >>mmcheck.txt

echo Sending Mail.
bmail -s 127.0.0.1 -t %2 -f %1 -h -a "MailMarshal Check [ServerName]" -m mmcheck.txt > sentmail.txt

Of course, you'll need to source the executables for grep.exe and bmail.exe or provide substitutes in order for this to work.

Postfix / Sendmail

If you are running a postfix or Sendmail, then this job can be done using a milter. A milter is generally written in C, Python or PERL. Personally, I prefer PERL. The way you write your script will depend on your actual setup. I plan on posting a postfix setup example sometime, I'll include a milter for email-check at that time.

Exchange

Unfortunately, dealing with actual messages in Exchange requires an MUA. I don't see any way around this except by setting one up to act on these messages. Technically, there's nothing to stop you running an Outlook client on an Exchange server with autologon (other than sheer common sense that is).

Groupwise

Being a full groupware system, there are a number of ways that Groupwise can respond and react to email messages at the server level. The easiest way is through the Groupwise API engine (GWAPI). The GWAPI can respond to the content in messages and trigger external scripts and is relatively simple to install and configure. The only downside is that ongoing development of the API has ceased since version 5 - so it will essentially run as an external system and only on a Netware server. The next easiest option is to write a Custom 3rd Party Object (C3PO), however that will essentially be an MUA that requires the Groupwise client to be installed. The elegant solution is to create a Trusted Application using the Groupwise TAPI that will directly access the message store.

Lotus Notes/Domino

Any decent Notes system will have at least one programmer managing the Notes/Domino infrastructure. Implementing a script to report on the status of the Domino system should be trivial.

Wildcard email addresses in MailMarshal

A few years back, there used to be a free service that enabled you to generate unique email addresses that would redirect to a single email account. This was great for web forms that required a valid email address. You would generate an email address for that particular website and disable it if they started spamming you.

Well, like all great "free" services, it eventually became "non-free", so that was the end of that. However, with a little ingenuity, it is possible to get MailMarshal to do something similar. Here's how:

Grammar

Firstly you need to identify the specific grammar of your email addresses, develop a secondary grammar for the wildcard addresses and then make sure there are no "collisions" in the grammar. For example, most organisations have their email addresses conforming (more or less) to the following grammar:

<first_name>.<last_name>@domain_name

That being the case, you can then define your wildcard grammer to be as follows:

<first_name>.<last_name>.<wildcard>@domain_name

Create Wildcard Group

The second step is to create a MailMarshal group called "Email Wildcard" and place the user names of everyone who will be using a wildcard, plus an entry with .* for every user as well. For example

joe.bloggs@example.com
joe.bloggs.*@example.com

It is possible to dispense with this step, however the group requirement gives you more control.

Rule 1

Some preliminary work is required here:

Firstly, create an external command called "Echo Old address to Threadno File". This is necessary because Mailmarshal can only work with non-SMTP header fields. You can work with the TO: field, but not the RCPT TO: field, which (unfortunately) is where the real stuff happens. So, we need to directly modify the text of the email outside of MailMarshal.

The external command will have the following properties:

Command Line: cmd
Parameters: /c echo {Recipient}>"{Install}\Address{Threadno}.txt
Tick the "single thread" and "execute once only" boxes. Leave the rest unchanged.

All this command will do is write the actual recipient email address to the file. Left like this, it will do nothing. We need to modify it later.

Next we need to write the rule to look for the email wildcard. This is done using the following Header-Match rule that needs to be manually defined as follows:

Match against: TO: CC: BCC:
Field parsing method: email addresses
Field search expression: (.+)\.(.+)\.(.+)@domain_name

For informational purposes, create a message stamp to indicate the email is from a wildcard source:

-----
Wildcard email address - {Recipient}

Next, a header rewrite rule needs to be created called "Address Change" as follows:

Match Against: X-AddressChange:
Field Parsing Method: Entire Line
Insert If Missing: Address Change

This will add the field X-AddressChange to the header indicating the email address has been changed and set us up for Rule 2. The complete Rule 1 will look as follows:

Standard Rule: Email Wildcard deletion - Rule 1
When a message arrives
Where message is incoming
Where addressed to 'Email Wildcard'
Where message contains one or more headers 'Email Wildcard'
Run the external command 'Echo Old address to Threadno File'
    And write log message(s) with 'Email Wildcard'
    And stamp message with 'Email Wildcard Stamp'
    And rewrite message headers using 'Address Change'
And pass message to the next rule for processing.

Rule 2

This rule is a lot simpler, it simply looks for the X-AddressChange field and then rewrites the email address to remove the wildcard.

The Header Match rule needs to be defined to look for "X-AddressChange" with the search expression '.+'

The Header Rewrite rule will be as follows:

Matching Fields: To:, Envelope Recipient:
Field Parsing Method: Email addresses
Field Search Expression: ^(.+)\.(.+)\.(.+)@(.+)
Substitute into field using expression: $1\.$2@$4
Enable Field Changes: (ticked) 

The final rule will be as follows:


Standard Rule: Email Wildcard deletion - Rule 2
When a message arrives
Where message is incoming
Where message contains one or more headers 'X-AddressChange Exists'
Rewrite message headers using 'Email Wilcard Deletion'
And pass message to the next rule for processing.

Rule 1 will stamp the message so you will know the original address used. If you start receiving spam from the source, add it to your blacklist of recipients. For example, suppose you sign up to a site using joe.bloggs.website1@example.com as your email address and you start being spammed by them. Add the address to your recipient blacklist and the spam will stop, however your regular email will still be delivered.

It is possible to add a "Rule 1.5" to add the recipient to the subject line - that way you can sort your emails by subject line. The rule would be very similar to Rule 2.

This is just one example of how you can push the boundaries of what MailMarshal is capable of by using external commands.