Configuring Evolution-EWS to connect to Exchange Online

Let me quickly get something off my chest:

I hate MS Exchange. 

I loathe it. There are few things I despise more. 

I would prefer almost any other integrated messaging system than it. The reasons why are far too long to go into here. It really deserves its own blog entry - which I may do some day.

Unfortunately, unless you are the decision maker for the organisation for which you work, or at least hold some good sway with your IT Steering Committee. You are probably stuck with it.

If your organisation is like most, you are probably migrating to Office 365 or have already done so. If Linux is your primary or preferred platform and you want to be able to receive your email, calendar etc. on your Linux desktop, this may be something of a problem. So, here's my explanation of how to use Evolution as your messaging client for Exchange Online using Exchange Web Services via evolution-ews.

What is "Exchange Web Services" (and why should I care)?

Exchange Web Services (EWS) is an application program interface (API) that allows programmers to access Microsoft Exchange items such as calendars, contacts and email. 

EWS first became available in Exchange Server 2007. It provides the ability to store, retrieve, move and modify email and related data for a single user, a group of users or an entire Exchange Server organisation on an Exchange server. EWS can be used to migrate Exchange data on-prem or to a third-party host in the cloud, or simply as an interface for a mail app or messaging client. Previously, if you wanted to do this you had to leverage Outlook Web Access (OWA).

As well as being available for On-prem editions of Exchange, EWS is available for Office 365, however that access is slightly different.

Evolution-EWS

Evolution-ews from 3.27.91 on has had the capability of connecting to office 365 with OAuth2 authentication. The administrator of the account must allow access via this method. Fortunately this is the default, so I suggest you don't mention it to your Exchange administrator as they may decide to block access.

Users of free accounts at live.com/hotmail.com/outlook.com addresses can also configure EWS account with the same Host URL, but they cannot use OAuth2, it's required to create an application specific password. Organisation accounts can use application specific passwords too, but administrators can disable this feature.
There are two existing application IDs, but the organisation administrators can create their own application and have everything under their control.

To be able to use OAuth2 with outlook.office365.com server an Application ID, which identifies the application which requests access to user data, and a Tenant ID, which basically identifies the organisation, are required.

First make sure everything is installed:

Ubuntu/Debian Instructions

sudo apt-get remove evolution
sudo apt-get update
sudo apt-get install evolution
sudo apt-get install evolution-ews

Note that the first line is there to remove the old version - which may not be up to date. This may not be necessary for you but YMMV.

CentOS/RedHat/Rocky instructions

sudo yum update
sudo yum install evolution
sudo yum install evolution-ews

If you are using DNF, replace 'yum' with 'dnf'

Finding your Tenant ID

Each organisation has assigned its directory (tenant) ID, which is used to identify the organisation when authenticating the user. How to find out the tenant ID is described here. A short walk-through is:

1. login to https://portal.azure.com

2. click the View button below Manage Azure Active Directory near the top of the page

3. the Overview section (selected by default) shows a Tenant information where is the Tenant ID

It can be seen also when:

1. login to https://portal.azure.com

2. click the View button below Manage Azure Active Directory near the top of the page

3. pick App registrations subsection

4. click Endpoints at the top of the subsection

It opens a table, where can be seen OAuth 2.0 authorization endpoint (v2) and OAuth 2.0 token endpoint (v2). Evolution-ews uses

https://login.microsoftonline.com/$TENANT/oauth2/token

and

https://login.microsoftonline.com/$TENANT/oauth2/authorize,

where the $TENANT is replaced with the same value as the Tenant ID from the above.

If you are a normal user and do not have the required permission to check in the portal.azure.com, you might find the tenant ID in your browser history. Search for URLs containing tenantId=........-....-....-....-.............

Copy this Tenant ID, it'll be used together with Application ID when configuring evolution-ews.

Choosing an Application ID

There are two unverified applications (for testing purposes). They are setup in a way that they can be used by the organisations, though, depending on the organisation settings, the administrators may or may not need to approve them first. These applications can be used with the organisation tenant ID or with the common (or left empty) tenant ID. Again, it depends on the actual organisation settings. 

1. 20460e5d-ce91-49af-a3a5-70b6be7486d1 - has both EWS and Microsoft Graph API usages enabled, but the Graph API is not enabled (in evolution-ews 3.38.x) due to its incompleteness. The application identifies itself as GNOME Evolution. It's preferred in case the Graph API would be used in the future, thus the transition to it will be smooth.

2. 751cf8be-ca07-484b-9308-fac4b9d85eff - has only the EWS protocol enabled. The application identifies itself as GNOME Evolution EWS.

3. Users can also try application IDs shown at point 12 of this Microsoft page, like d3590ed6-52b3-4102-aeff-aad2292ab01c or ea5a67f6-b6f3-4338-b240-c655ddc3cc8e, with the Redirect URI set to urn:ietf:wg:oauth:2.0:oob and other values kept empty.

If your organisation has its own Application ID, use that one. If you are the administrator for your organisation, and want to create your own Application ID by following this link.

Evolution Instructions

Launch the Evolution application. If this is a new installation, follow the prompts below to configure your account. Otherwise, select the menu items File -> New -> Mail Account, click Next, then follow the prompts.

• Enter your name and your primary SMTP address, uncheck Look up mail server details… and click Next
• For Server Type, select Exchange Web Services
• For Username, specify your email address
• Change Host URL to https://outlook.office365.com/EWS/Exchange.asmx
  Click the Fetch URL button, providing your EID password when prompted. (If this doesn't work, don't worry, it can be done later). This will result in the OAB URL field being populated, which is the address from which the client can download a copy of the Offline Address Book.
• Change Authentication to OAuth2 (Office365).
• Make sure that Override Office365 OAuth2 settings is checked and populate the Tenant ID and Application ID fields as follows:
  • Tenant ID: your tenant ID
    
  • Application ID: choose from above
    
• Finish the New Mail Account Wizard, and you will then be taken to the University’s login page. Authenticate with your EID credentials and perform Duo multi-factor authentication.

Please note that the initial synchronization with your account may take an extended period of time if you have a large mailbox.

If you've used EWS previously, you would have noticed that credential management is handled by the Gnome settings under online accounts. With O365 this does not happen. I'm not sure why it's different - it may have something to do with how the data is presented via the App ID, but I'm only guessing here. If anyone knows, please enlighten me in the comments.

Acknowledgements

The following links were helpful to me and acted as source material:

https://sites.utexas.edu/glenmark/2021/02/01/how-to-setup-your-office-365-email-using-evolution-ews-linux/

https://wiki.gnome.org/Apps/Evolution/EWS/OAuth2

https://bugzilla.redhat.com/show_bug.cgi?id=972562

How SORBS ruined my life (old blog)

When this blog entry was written, Open Relay blockers like SORBS were considered the primary defence mechanism against spam email. They are still often used, but not as much as they used to be - for many of the reasons discussed in this article. SPF, DKIM & DMARC have largely taken their place.

 

 

I've had a helluva week so far, and I've got SORBS to thank for it.

SORBS (Spam and Open Relay Blocking System) is an email server reputation service. It (along with other reputation services) collect data on servers that are are either poorly configured (and open to abuse) or actively send spam. Reputation services are an excellent way of protecting against spam. They have very high hit rates and (generally) very low false positives. They also have a very low performance overhead compared with heuristic and bayesian filtering techniques. The mail servers I admister are configured to quarantine all email that is listed by either SORBS, SpamHaus or SpamCop (two other reputation services).

However, sometime last week (when I was off work sick), SORBS listed pretty much all of the hotmail, windows live, yahoo, bigpond, optusnet and myplace servers. There may have been others listed as well.

Responding to a few complaints that came in on Monday about email not being received, I began checking. Then began the sinking feeling that goes along with knowing that I'd have to change the email filtering rules, do some regression tests and then resubmit a week's work of spam through the email filters. SORBS is a very aggressive filter and I've been quite reliant on it for some time.

Less fun was trying to explain the problem to the CEO.

The first part of the quick fix was to move the SORBS check to the end of the anti-SPAM rules of the DMZ mail filter and set it to monitor only - not block. The DMZ mail server is exposed to the Internet and performs basic/quick checks only and does not look at the content of the email messages. It is highly robust and is meant as the front line of defence. It experiences an average of 90 security attacks per hour. It filters about 75% of the email traffic as inappropriate before passing the "possibly okay" messages to the second filter.

The second part was to add a check for zero day threats on the DMZ server and tag messages accordingly.

The third part was to tighten the screws a bit on the second mail filter. This meant decreasing the tolerance limits a little and added a few more checks which included looking for the tags on the email messages placed there by the DMZ mail filter. Usually, this server intercepts about 8% of messages pass to it. Messages quarantined by this server may result in a quarantine message and even a self-release option depending upon the spam score the email receives.

Now that these changes were made, I resubmitted 2143 spam quarantined messages. With the new rules in place, 1963 messages were blocked and 180 passed through to the second filter which blocked a further 68 messages leaving 112 messages sailing through to the mail server. I don't know how what the spam:ham ratio of the 112 messages was but the count seemed to be manageable so I released the remaining 14,000 quarantined messages and sent an email instructing people to forward any received spam to the spam submit mailbox for heuristic analysis. So far, I have 35 spam messages submitted by users which (if accurate) we now have a spam hit rate of 99.75% with (hopefully) a close to zero false positive rate. The hit rate has dropped to 59.4% for the DMZ server and risen to 12.5% for the second filter. Time and monitoring will determine how successful the changes have been.

Moving forward, I have written replacement rules that will no longer quarantine email based upon failed reputation. The new rules will look at SPF (Send Policy Framework - RFC4088) in addition to reputation. If the SPF check passes, the email will be accepted. If SPF fails, it will be dropped. If SPF is 'softfail' or 'none' then it will be subject to a reputation check. Any failure will drop then connection with a 46x error - Temporary Failure, with details as to why sending failed. The sending MTA will then notify the recipient that the email failed to send. The sending server then has the option of implementing SPF. I'd like at some point to add a DKIM check, but that's a reasonably difficult task.

BTW, I am heavily influenced by Ming Weng Wong's whitepaper on Messaging Anti-Abuse:

http://www.openspf.org/blobs/sender-authentication-whitepaper.pdf

 

 

Email status check

Okay, you're offsite and someone rings up to say the email system isn't working. Now, you KNOW that nine times out of ten the email system is working perfectly - it's just something the user is doing wrong. How can you quickly check to see if email is working without logging into the servers? Well, you could simply send an email from your gmail account to your work account and vice versa. That would be a good indication that everything is working, but if you don't get the email, it tells you absolutely nothing other than something might be wrong with the email system.

Most enterprise mail systems have a number of servers involved in the generation, transmission and reception of email. In generic terms we have the following elements:

MTA - Mail Transfer Agent
MDA - Mail Delivery Agent
MSA - Mail Submission Agent
MRA - Mail Retrieval Agent
MFA - Mail Filtering Agent
MUA - Mail Usage Agent

Many sysadmins may exclaim at this point "Hang on - I don't remember there being that many elements to email delivery!" and the reason for that is we are now in the post-Marid world of Internet based mail as explained by RFC5598 - 2009. Quite simply: Things are different now. If you are running a mail system that was setup before this time and not updated, chances are that you aren't RFC-compliant to the IETF standard. If you're running MS Exchange out-of-the-box, then you definitely aren't standards compliant. However, making your email system RFC-compliant is the stuff of another article...

RFC5598 divides the various agents into their respective areas of responsibility called "Responsible Actor Roles". These are:

 - User
 - Message Handling System (MHS)
 - ADministrative Management Domain (ADMD)

The traditional flow of email was:

MUA -> MTA -> .... -> MTA -> MUA

Now, the email flow is more commonly:

MUA -> MSA -> MTA -> ... -> MTA -> MFA -> MDA --> MRA --> MUA

where -> is a push operation and --> is a pull operation.

Obviously, in such a system there are a number of elements that can go wrong and be described as "the email system is down".

On email systems I administer, I usually create a dummy account called "Email-Check". At its most basic level, you set it up with an Out of Office reply that says "Email is working". However it doesn't end there. Each point in your message reception system can be setup to respond with diagnostics on each component. A fully working system will received replies from each component in the chain. In the second example, if you send your email to email-check@your-domain and receive a reply from the MTA and MFA, but not the MDA or the MRA, then you can reasonably assume the problem lies with the MDA - that should be the place you start looking.

Practical Examples

MailMarshal

1. Write a rule in MailMarshal that triggers when the to: address is email-check. Have the rule execute as an external command the file "mail-check.cmd" and pass the following parameters to it: servername@domain {ReplyTo} {SenderIP} {HelloName}

2. Write email-check as follows:

@echo off
c:
cd \scripts

echo Email check for [servername] > mmcheck.txt
echo. >> mmcheck.txt
echo.|time|grep current  >> mmcheck.txt
echo.>>mmcheck.txt
echo [Servername] Mail Marshal Service Information >>mmcheck.txt
echo. >>mmcheck.txt
start /wait msinfo32.exe /categories +SWEnvServices /report msinfo.txt
type msinfo.txt | grep MailMarshal >> mmcheck.txt
echo. >>mmcheck.txt
echo Sending IP  : %3 >>mmcheck.txt
echo Helo Name   : %4 >>mmcheck.txt

echo Sending Mail.
bmail -s 127.0.0.1 -t %2 -f %1 -h -a "MailMarshal Check [ServerName]" -m mmcheck.txt > sentmail.txt

Of course, you'll need to source the executables for grep.exe and bmail.exe or provide substitutes in order for this to work.

Postfix / Sendmail

If you are running a postfix or Sendmail, then this job can be done using a milter. A milter is generally written in C, Python or PERL. Personally, I prefer PERL. The way you write your script will depend on your actual setup. I plan on posting a postfix setup example sometime, I'll include a milter for email-check at that time.

Exchange

Unfortunately, dealing with actual messages in Exchange requires an MUA. I don't see any way around this except by setting one up to act on these messages. Technically, there's nothing to stop you running an Outlook client on an Exchange server with autologon (other than sheer common sense that is).

Groupwise

Being a full groupware system, there are a number of ways that Groupwise can respond and react to email messages at the server level. The easiest way is through the Groupwise API engine (GWAPI). The GWAPI can respond to the content in messages and trigger external scripts and is relatively simple to install and configure. The only downside is that ongoing development of the API has ceased since version 5 - so it will essentially run as an external system and only on a Netware server. The next easiest option is to write a Custom 3rd Party Object (C3PO), however that will essentially be an MUA that requires the Groupwise client to be installed. The elegant solution is to create a Trusted Application using the Groupwise TAPI that will directly access the message store.

Lotus Notes/Domino

Any decent Notes system will have at least one programmer managing the Notes/Domino infrastructure. Implementing a script to report on the status of the Domino system should be trivial.

Wildcard email addresses in MailMarshal

A few years back, there used to be a free service that enabled you to generate unique email addresses that would redirect to a single email account. This was great for web forms that required a valid email address. You would generate an email address for that particular website and disable it if they started spamming you.

Well, like all great "free" services, it eventually became "non-free", so that was the end of that. However, with a little ingenuity, it is possible to get MailMarshal to do something similar. Here's how:

Grammar

Firstly you need to identify the specific grammar of your email addresses, develop a secondary grammar for the wildcard addresses and then make sure there are no "collisions" in the grammar. For example, most organisations have their email addresses conforming (more or less) to the following grammar:

<first_name>.<last_name>@domain_name

That being the case, you can then define your wildcard grammer to be as follows:

<first_name>.<last_name>.<wildcard>@domain_name

Create Wildcard Group

The second step is to create a MailMarshal group called "Email Wildcard" and place the user names of everyone who will be using a wildcard, plus an entry with .* for every user as well. For example

joe.bloggs@example.com
joe.bloggs.*@example.com

It is possible to dispense with this step, however the group requirement gives you more control.

Rule 1

Some preliminary work is required here:

Firstly, create an external command called "Echo Old address to Threadno File". This is necessary because Mailmarshal can only work with non-SMTP header fields. You can work with the TO: field, but not the RCPT TO: field, which (unfortunately) is where the real stuff happens. So, we need to directly modify the text of the email outside of MailMarshal.

The external command will have the following properties:

Command Line: cmd
Parameters: /c echo {Recipient}>"{Install}\Address{Threadno}.txt
Tick the "single thread" and "execute once only" boxes. Leave the rest unchanged.

All this command will do is write the actual recipient email address to the file. Left like this, it will do nothing. We need to modify it later.

Next we need to write the rule to look for the email wildcard. This is done using the following Header-Match rule that needs to be manually defined as follows:

Match against: TO: CC: BCC:
Field parsing method: email addresses
Field search expression: (.+)\.(.+)\.(.+)@domain_name

For informational purposes, create a message stamp to indicate the email is from a wildcard source:

-----
Wildcard email address - {Recipient}

Next, a header rewrite rule needs to be created called "Address Change" as follows:

Match Against: X-AddressChange:
Field Parsing Method: Entire Line
Insert If Missing: Address Change

This will add the field X-AddressChange to the header indicating the email address has been changed and set us up for Rule 2. The complete Rule 1 will look as follows:

Standard Rule: Email Wildcard deletion - Rule 1
When a message arrives
Where message is incoming
Where addressed to 'Email Wildcard'
Where message contains one or more headers 'Email Wildcard'
Run the external command 'Echo Old address to Threadno File'
    And write log message(s) with 'Email Wildcard'
    And stamp message with 'Email Wildcard Stamp'
    And rewrite message headers using 'Address Change'
And pass message to the next rule for processing.

Rule 2

This rule is a lot simpler, it simply looks for the X-AddressChange field and then rewrites the email address to remove the wildcard.

The Header Match rule needs to be defined to look for "X-AddressChange" with the search expression '.+'

The Header Rewrite rule will be as follows:

Matching Fields: To:, Envelope Recipient:
Field Parsing Method: Email addresses
Field Search Expression: ^(.+)\.(.+)\.(.+)@(.+)
Substitute into field using expression: $1\.$2@$4
Enable Field Changes: (ticked) 

The final rule will be as follows:


Standard Rule: Email Wildcard deletion - Rule 2
When a message arrives
Where message is incoming
Where message contains one or more headers 'X-AddressChange Exists'
Rewrite message headers using 'Email Wilcard Deletion'
And pass message to the next rule for processing.

Rule 1 will stamp the message so you will know the original address used. If you start receiving spam from the source, add it to your blacklist of recipients. For example, suppose you sign up to a site using joe.bloggs.website1@example.com as your email address and you start being spammed by them. Add the address to your recipient blacklist and the spam will stop, however your regular email will still be delivered.

It is possible to add a "Rule 1.5" to add the recipient to the subject line - that way you can sort your emails by subject line. The rule would be very similar to Rule 2.

This is just one example of how you can push the boundaries of what MailMarshal is capable of by using external commands.

Sender Policy Framework

Nearly ten years ago, Meng Weng Wong of pobox.com proposed an anti-spam methodology for SMTP email based upon the sending email domain by publishing a list of authorised mail sending agents in DNS. What followed was the highly controversial IETF MARID conference that produced four experimental RFCs including RFC4408 - Sender Policy Framework. That RFC is currently being revised by the SPFBIS working group which will ratify SPF as an IETF standard.

What the SPF experiment has demonstrated is that SPF will never be a complete solution to the problem of spam. However, it has been highly effective in mitigating spam and reducing the load on anti-spam engines. It is believed that SPF in conjunction with two other anti-spam measures (DKIM & DMARC) should be able to virtually eliminate spam with needing to check the content of email measures.

Part of the problem with SPF being widely successful in preventing spam is that most email systems administrators misunderstand SPF and implement it either poorly or incorrectly.

SPF is a "sender policy". It allows the sending system to define clearly what is and what is not a legitimate source of email. Before SPF, such checks were done using RDNS lookups - which are costly, expensive and simply do not work correctly if the domain is using an external email provider.

One thing the Marid conference did settle once and for all is who has the responsibility for the delivery of email.

The standard rule for email delivery post-Marid is that it is the sending server's responsibility to deliver the email until an SMTP code 250 is received from the target mail server. After that, it is the recipient MTA (Message Transfer Agent) that accepts responsibility to deliver the email.

That may sound really simple, however there are an enormous number of consequences resulting from this. For starters it means that email messages can no longer be trashed by anti-spam engines without notification to either the sender or the recipient. More importantly, if the receiving MTA cannot deliver the email after issuing a 250 to the sender, it now MUST issue an NDR (Non-Deliverable Report). However, it must do this without risking backscatter (sending a forged message back to a real recipient). MTAs that backscatter will quickly find themselves on a global blacklist.

The corollary to this is that the receiving MTA MUST have some way of verifying a legitimate source of email: Enter SPF.

Creating an SPF record for an email domain is quite simple: all you need to do is publish a DNS TXT record similar to the following:

<domain name> TXT "v=spf1 mx a ~all"

The "v=spf1" bit is simply identifying the string as an SPF record. What follows is a list of acceptable and unacceptable hosts and the degree to which they are acceptable. The host list is optionally prefixed by a character which designates the level of acceptance or non-acceptance as follows:

'+'  (or no prefix) PASS
'-' FAIL
'?' NEUTRAL
'~' SOFTFAIL

SPF PASS is essentially a guarantee that the email came from a legitimate mail server for the sending domain and standard spam tests may be bypassed.

SPF FAIL is a statement that the email did not come from a legitimate mail server and may safely be rejected.

SPF NEUTRAL means the sender policy has nothing to say about the email. Treat it as though no SPF existed.

SPF SOFTFAIL means the email probably did not come from a legitimate mail server, but the sender cannot 100% guarantee that it didn't.

There are two other SPF Results: TempError and PermError.  I have made comments on how these results are interpreted in the SPFBIS Working Group.

The above record could have been written as "v=spf1 +mx +a ~all". However, since '+' is assumed which no symbol is given, the '+' is rarely used.

The basic acceptable fields used are:

mx - all hosts defined by the mx record for the domain.
a - all hosts with an a record for the domain
ip4: - all hosts within the listed netblock (or ip address)
ip6: - all hosts within the listed ipv6 netblock
include: - include the SPF record for the listed email hosting domain
all - matches everything

So for our listed example, the receiving mail server would tested the sending servers IP address and check first to see if it matches the mx record for the sending domain. If it does, the return an SPF PASS result; otherwise test to see if it resolves to a 'a' record for the sending domain and if that fails, return an SPF SOFTFAIL.

I see a lot of SPF records that end in '-all'. I regard that as a very bold statement. It basically gives me the right to completely reject email messages with an SMTP code 550 if they fail the SPF check. The sender who publishes such a statement has to be completely SURE that no legitimate email will come from a host other than the one's listed. For the vast majority of organisations, that guarantee is simply impossible to make. The reason is that the sender cannot control email forwarding.

When you forward an email, the FROM address remains the same as does the SMTP MAIL FROM: line. This is what SPF uses to evaluate the sending domain against the connecting MTA. In the case of forwarded mail, they will not match and SPF will FAIL, resulting in a rejected message. However, now the intermediate server has a problem. It has accepted responsibility to deliver the mail and cannot do so. Under the post-MARID rules it must deliver an NDR, however with no current connection to the sending MTA the only way it can do so is send a DSN email (Delivery Status Notification) and risk backscatter.

There is a solution to this mess called the Sender Rewriting Scheme (SRS). This simply involves encapsulating the original destination address within the MAIL FROM: line in such a way that SPF can check both the source email addresses. It's a great idea that suffers from a simple three-part problem:

1) The sender has no control over whether SRS is implemented at the original destination or not.
2) Since SPF is a "sender policy", this means the sender still cannot make definitive statements in their SPF record.
3) To date, very few commercial MTAs implement SRS.

This means we cannot rely on SRS even if we implement SRS ourselves. You cannot know for certain how many of your email recipients implement SRS, nor can you know if they are forwarding email. This means, at best we can publish a "~all" at the end of our SPF records. Anything else risks both the possibility of incorrect non-delivery of an email and unnecessary backscatter.

At around the same time that SPF was floated, both Google and Yahoo came up with the idea of signing the email envelope using public-key cryptography. Yahoo's was called DomainKeys and Google's was called Domain Key Identified Mail (DKIM). Both methods were similar, but DKIM has become the standard. It works similarly to SPF with the public key published in a DNS TXT record. The good thing about DKIM is that it is not broken by forwarding since it looks at the envelope.

Mailing Lists, however, do break DKIM. This is because mailing lists (by their nature) modify the email envelope.

This gives us hope, because now we can use both SPF and DKIM to evaluate email. If either method passes, we can now regard the email as legitimate. Only if both fail do we stamp FAIL on the email message and reject it.

The problem with this idea is the take up of DKIM is very low. Few commercial mail servers can support DKIM. Publishing a DKIM record is also more difficult than publishing an SPF record. So, while the take up rate for SPF is around 40% (accounting for more that 60% of legitimate email), the take up rate for DKIM is around 5%. So, even if you do publish a DKIM record and stamp all outgoing email with a DKIM hash, chances are that few of the recipient MTAs will do anything about it.

Enter Domain Based Message Authentication, Reporting and Conformace or DMARC.

A DMARC record is published in DNS like SPF and DKIM. The difference is that DMARC provides a mechanism to report back to the original sending MTA to determine if the email is legitimate. In addition, it provides a "fuzzy" method for temporary rejection of non-conformant email similar to grey-listing, but with a more intuitive and sender-determined policy for doing so. With a DKIM record, the sender policy is not just isolated to SPF or DKIM alone. The sender can define a rejection policy based upon a combination of checks and receive reports back on rejected messages. Thus it removes all the guess work associated with message rejection.

Pretty much all the major players are now behind DMARC: Google, Yahoo, Microsoft, AOL, Facebook all currently implement DMARC even before the specification has been ratified by the IETF. See http://www.dmarc.org

In summary, if you administer an email system, you SHOULD be publishing an SPF record at the very least. You should also be implementing some form of SPF checking - but not too aggressively (making sure you whitelist regulars). If possible, look at implementing DKIM with a view to eventually implementing DMARC. As for SRS, if you can implement it, then do so. It will reduce the level for SPF rejections on forwarded email.

I will write another article another time on practical SPF implementations. Oh, and if you are running MailMarshal less than version 7, then upgrade it asap. The SPF implementation is broken. It took me nearly two weeks to convince them of this. Even with the fix they implemented, it still breaks under certain circumstances.