Headaches from Configuring Authentication and Encryption for Unix MTAs

Though I'm now officially an Exchange hater, there's still one thing that really sucks about running a Unixy email server: authentication and encryption.

It's been hard to configure for years - Sendmail, qmail, and now exim4, have put me through the grinder when it comes to setting up authentication, particularly authentication for SMTP relaying.

Out of box, the mail servers in the Debian and Ubuntu distros seem to be configured to relay email for your localhost, and that's all.

On the flipside, the most common configuration sysadmins want is IMAP, local SMTP relay, controlled relay for their LAN, and authenticated relay for the WAN. They want to support all the clients, meaning CRAM-MD5, CRAM-SHA1 or other digest authentication over cleartext, and both digest and plaintext auth over STARTTLS and plain old SSL, over ports 25, 465, and 587.

Here's a grid showing what configurations need support:

Password \ Connection
Unencrypted
START/TLS
SSL
Plaintext
NO
YES
YES
CRAM-MD5, CRAM-SHA1
YES
YES
YES

What "YES" means is that a user can relay mail to remote hosts from a client on the internet.

That's six basic variations, and five of them need to be supported. I'm counting both MD5 and SHA1 as one category, but when you do the tests, you need to actually find a client that will let you choose the digest type.

This omits another factor, which is the port used to connect: 465 is always SSL, but ports 25 and 587 are both for SMTP. Nominally, port 25 is for incoming mail, and 587 is for outgoing mail, and within LANs, port 25 is used between servers. So a client in a LAN can use port 25, but out on the internet, they need to use 587 because ISPs now seem to block port 25.

The good news is that the mail servers all support ways to use these digests and encryptions, but it's up to the sysadmin to decide the logic. Exim4 packaged by Debian and Ubuntu come with these elaborate config files that basically conform to the above rules, but they are not well documented, and thus hard to modify.

(For Exim4 I just got plaintext over START/TLS working, and it took a couple hours because I'm a noob. That's one configuration.... I didn't get digest authentication, or SSL on port 465 going right. For qmail, I managed to get SSL over 465 going right quickly, but START/TLS took a while to do.)