Email. The most common means of internet communication, that has been around for ages and nobody has been able to replace it. And yet, it is so hard to get it right in terms of configuration so that messages don’t get sent in spam.
Setting up your own email server is a nightmare, of course, but even cloud email providers don’t let you skip any of the steps – you have to know them and you have to do them. And you if miss something, business will suffer, as a portion of your important emails will get in spam. The sad part is that even if you get all of them right, emails can still get in spam, but this is on the spam filters and there’s little you can do about it. (I dread the moment when an outgoing server will run the email through standard spam filters with standard sets of configurations to see if it would get flagged as spam or not)
This won’t be the first “how to configure email” overview article that mentions all the pitfalls, but most of the ones I see omit some important details. But if you’ve seen one, you are probably familiar with what has to be done – configure SPF, DKIM and DMARC. But doing that in practice is trickier, as his meme implies:
So, you have an organization that wants to send email from its “example.com” domain. Most tutorials assume that you want to send email from one server, which is almost never the case. You need it for the corporate emails (which would in many cases be a hosted or cloud MS Exchange, or Google Suite), you need it for system emails from your applications (one or more of them), which use either another internal server or a cloud email provider, e.g. Amazon SES, you also need it for your website, which uses the hosting provider email server, and you need it for your email campaigns, e.g. via Mailchimp or Sendgrid.
All of the email providers mentioned above have some parts of the picture in their documentation but it doesn’t work in combination. As most providers wrongly assume they are the only one. Their examples assume that and their automated verifications assume that – e.g. Microsoft checks if your SPF record matches exactly what they provide, rather than checking if their servers are allowed by your more complex SPF record which includes all of the above providers.
So let’s get to the individual items you have to configure. Most of them are DNS records, which explains why a technical person in each organization has to do it manually, rather than each service pushing it there automatically after some API authentication:
- SPF (Sender Policy Framework) – a DNS recrod that lists the permitted senders (IP addresses) and an instruction flag on what to do with those that don’t match. In the typical scenario you need to include multiple senders’ policies rather than listing IP addresses, as they can change. E.g. in order to use Office365, you have to add
include:spf.protection.outlook.com. Note that this should be a TXT record, but some DNS providers support a special type of record – SPF. So some older software may expect an SPF header, which means you should support both records with identical values. The syntax is straighforward but sometimes tricky, so you can use a tool to generate and validate it. - DKIM (DomainKeys Identified Mail) – a DNS record that lets email senders sign their emails. The DNS record includes the public key used to verify the signature. Why is it needed if there’s SPF? Among other things (like non-repudiation), because with SPF the From header can still be spoofed. Not that DKIM always helps with that, but in combination with DMARC it does. How does DKIM work in multi-sender scenario? You have multiple DKIM selectors which means multiple TXT records. Usually every provider will recommend its own selector (e.g.
selector1._domainkey.example.com). Some may insist on beingdefault._domainkey, and if two of them insist on that, you should contact support (the message contains the selector and then verification will fail if it does not match) - DMARC (Domain-based Message Authentication, Reporting and Conformance) – this DNS record contains the policy according to which your emails should be validated – it enforces SPF and DKIM and tells the receiving side what to do if they fail. You can have one DMARC policy (again as a TXT record). The syntax is not exactly human readable, so use a tool to generate and validate it. An important aspect of DMARC is that you can receive reports in case of failures – you can specify an email where reports are sent and you can analyze them. There are services (like ReportURI) that can aggregate and analyze these reports. I prefer setting multiple report emails – one administrative and one for ReportURI.
- PTR (pointer) – this is used for reverse DNS loopkups – it maps a domain name to an IP address (as opposed to A records which map IP adresses to domain names). Spam filters use it to check incoming email. The PTR records should be there for the servers that send the email, e.g. mail.example.com. External providers are likely to already have that record so no need to worry about it.
- Service-specific settings – you may configure your headers properly but the service sending the emails (e.g. Office365, Mailchimp) might still be missing some configuration. In some cases you have to manually confirm your headers in order to enable DKIM signing. With MS Exhange, for example, you have to execute a few PowerShell commands to generate and then confirm the DKIM records.
- Blacklists – if you haven’t had everything setup correctly, or if one of your sending servers/services has been compromised, your domain and/or servers may be present in some blacklist. You have to check that. There are tools that aggregate blacklists and check against the, e.g. this one
After everything is done, you can run a spam test using some of the available online tools: e.g. this, this or this. And speaking of tools, MXToolbox has many useful tools to verify all aspects of email configuration
By now (even if you already knew most of the things above) you are probably wondering “why did we get here?”. Why do we have to do so many things just to send simple email. Well, first, it’s not that simple to have a universal messaging protocol. It looks simple to use and that’s the great part of it, but it does hide some complexities. The second reason is that the SMTP protocol was not designed with security in mind. Spam and phishing were maybe not seen as such a big issue and so the protocol does not have built-in guarantees for anything. It doesn’t have encryption, authentication, non-repudiation, anything.
That’s why this set of instruments evolved over time to add these security features to email (I haven’t talked about encryption, as it’s handled differently). Why did it have to be DNS-based? It’s the most logical solution, as it guarantees the ownership of the domain, which is what matters even visually to the recipient in the end. But it makes administration more complicated, as you are limited to one-line, semicolon or space separated formats. I think it would be helpful to have a way to delegate all of that to external services, e.g. by a single authenticating DNS record which points to a URL which provide all these policies. For example an EML record to point to https://example.com/email-policies which can publish them in a prettier and more readable (e.g. JSON) format and does that in a single place rather than having to generate multiple records. Maybe that has its own cons, like having the policy server compromised.
But if anything is obvious it is that everything should be designed with security in mind. And every malicious scenario should be taken into account. Because adding security later makes things even more complicated.
The post Getting Email Sending Settings Right appeared first on Bozho's tech blog.
Recent Comments