A mail server faces two kinds of threats. People try to abuse it: open relays, cracked mailboxes used to send spam. And people try to intercept or spoof its traffic. A compromised mail server lands your IP on blocklists within hours. This guide covers the security settings every self-hosted Postfix server should have. The examples use Postfix and Dovecot, the most common open-source pair.

Ports and their roles

Port Name Used for Encryption
25 SMTP Server-to-server delivery Opportunistic STARTTLS
587 Submission Your mail clients sending mail STARTTLS, required
465 Submissions (SMTPS) Clients sending mail Implicit TLS from the first byte
993 IMAPS Clients reading mail Implicit TLS
110/143 POP3/IMAP Legacy plaintext Avoid, or require STARTTLS

Rule: clients authenticate only on 587 or 465 over TLS. Port 25 is for receiving from other servers and doesn’t accept logins.

On VPS.DO, outbound mail (port 25) and reverse DNS are available on USA VPS plans, which is what you need to run a mail server. See email VPS and PTR records.

1. TLS certificates

Use a real certificate for the mail hostname (mail.example.com), from Let’s Encrypt:

sudo certbot certonly --standalone -d mail.example.com      # or the webroot or DNS method

Postfix (/etc/postfix/main.cf):

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file  = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_security_level = may          # port 25: offer STARTTLS to other servers
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

smtp_tls_security_level = may           # outgoing: use TLS when the other side offers it
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_loglevel = 1

Reload Postfix and Dovecot after certificate renewal. A Certbot deploy hook does this automatically:

echo 'systemctl reload postfix dovecot' | sudo tee /etc/letsencrypt/renewal-hooks/deploy/mail.sh
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/mail.sh

2. Submission ports with mandatory TLS and auth

In /etc/postfix/master.cf, enable 587 and 465 with authentication required:

submission inet n - y - - smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject

submissions inet n - y - - smtpd
  -o syslog_name=postfix/submissions
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject

And globally, never allow authentication without TLS:

smtpd_tls_auth_only = yes

Dovecot: ssl = required and disable_plaintext_auth = yes.

3. Not an open relay

An open relay forwards mail for anyone, and gets abused and blocklisted fast. Postfix’s defaults are safe. Keep the relay restrictions strict:

smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
mynetworks = 127.0.0.0/8 [::1]/128

Test it from an outside server. Relaying to a foreign domain without authentication must be rejected with Relay access denied.

4. Stop abuse from compromised accounts

Passwords get phished, and a cracked mailbox can send tens of thousands of spam messages in an hour.

smtpd_client_message_rate_limit = 100
smtpd_client_recipient_rate_limit = 300
anvil_rate_time_unit = 60s

Per-user limits need a policy daemon (postfwd, or rspamd’s ratelimit module).

  • Sender login maps, so authenticated users can only send as their own addresses (smtpd_sender_login_maps + reject_sender_login_mismatch).
  • Monitor the queue: mailq | tail -1. Alert when it suddenly grows.

5. Domain authentication: SPF, DKIM, DMARC

These protect your domain from spoofing and improve deliverability:

  • SPF — which servers may send for the domain.
  • DKIM — cryptographic signatures on outgoing mail (OpenDKIM or rspamd).
  • DMARC — policy and reporting that tie SPF and DKIM together.

See SPF, DKIM and DMARC.

6. Enforce TLS between servers: MTA-STS and TLS-RPT

Opportunistic STARTTLS on port 25 can be downgraded by an attacker who strips the STARTTLS offer. MTA-STS lets your domain publish a policy saying “deliver to us only over valid TLS”:

  • DNS TXT record: _mta-sts.example.com "v=STSv1; id=20261014"
  • A policy file at https://mta-sts.example.com/.well-known/mta-sts.txt:
version: STSv1
mode: enforce
mx: mail.example.com
max_age: 604800

TLS-RPT (_smtp._tls.example.com TXT "v=TLSRPTv1; rua=mailto:[email protected]") sends you reports about TLS failures. Start with mode: testing.

7. Test

openssl s_client -starttls smtp -connect mail.example.com:25 -servername mail.example.com </dev/null | head -20
openssl s_client -connect mail.example.com:465 </dev/null | head -5
swaks --to [email protected] --server mail.example.com:587 --tls --auth LOGIN --auth-user [email protected]

Online checkers (MXToolbox, internet.nl, mail-tester) verify TLS, SPF, DKIM, DMARC, MTA-STS and blocklist status.

Quick reference

Goal Setting
No auth without TLS smtpd_tls_auth_only = yes
Old protocols off smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
No open relay reject_unauth_destination in relay restrictions
Client submission 587 (STARTTLS) / 465 (TLS), auth required
Brute force Fail2Ban postfix-sasl, dovecot
Enforce inbound TLS MTA-STS + TLS-RPT

Related: mail server on a VPS and email deliverability.

  • Email
  • Security
  • Postfix