Email Deliverability for Self-Hosted Mail: Get Out of Spam

Running your own mail server on a VPS is very doable. Getting its messages into inboxes, not spam folders, takes a few specific steps. Gmail, Outlook and Yahoo judge every message on the sending IP’s setup, the domain’s authentication, reputation and content. Here’s the checklist that covers most deliverability problems.

1. Server identity: PTR, HELO and hostname

Receiving servers first check who is connecting:

The PTR (reverse DNS) record of your IP should be a proper hostname, like mail.example.com.
That hostname’s A record must point back to the same IP (forward-confirmed rDNS).
Your server should greet with the same hostname (Postfix myhostname = mail.example.com).

dig -x 203.0.113.25 +short          # → mail.example.com.
dig +short mail.example.com         # → 203.0.113.25
postconf myhostname

See PTR records. On VPS.DO, PTR is set in the client area for USA VPS IPs, which is also where outbound mail is supported. See email VPS.

2. Authentication: SPF, DKIM, DMARC, aligned

SPF authorises your server’s IP for the domain.
DKIM signs every message with a key published in DNS.
DMARC requires that SPF or DKIM passes and aligns with the visible From: domain.

Since 2024, Gmail and Yahoo require SPF or DKIM for all senders, and SPF and DKIM and DMARC for bulk senders (5,000+ messages a day to their users). Full setup: SPF, DKIM and DMARC.

3. Encryption

Offer and use TLS for SMTP (STARTTLS on port 25 for server-to-server). Gmail marks unencrypted mail with a red padlock. See SMTP security.

4. Reputation: IP and domain

Check blocklists for your IP. See IP blacklists.
New IPs and domains have no reputation. Warm up: send small volumes to people who expect your mail, then increase gradually.
Never send spam, and protect accounts from compromise. One hacked mailbox ruins reputation fast.
Monitor: Google Postmaster Tools (domain reputation, spam rate) and Microsoft SNDS (IP reputation).

Keep the spam complaint rate below 0.1% (Gmail’s guidance; above 0.3% causes serious problems).

5. Bulk and marketing mail rules

For newsletters and marketing (and required by Gmail and Yahoo for bulk senders):

One-click unsubscribe (List-Unsubscribe and List-Unsubscribe-Post headers), honoured within 2 days;
a visible unsubscribe link in the message;
confirmed opt-in lists, and removing bounces and inactive addresses;
sending marketing from a separate subdomain or IP from transactional mail (password resets, receipts), so one doesn’t hurt the other.

Many teams send bulk mail through a dedicated email service, and keep the VPS for transactional or personal mail. See transactional email services.

6. Content and format

Content matters less than authentication and reputation, but it still counts:

Send multipart messages (HTML plus plain text).
Avoid spammy patterns: all caps, excessive punctuation, link shorteners, image-only emails, mismatched link text and URLs.
Use a consistent From name and address.
Keep HTML clean, and avoid huge attachments.

7. Test before you blame the filters

swaks --to [email protected] --from [email protected] --server localhost   # send a test (swaks package)

mail-tester.com: send a message to the address it gives you, and get a score with specific problems (SPF, DKIM, rDNS, blocklists, content).
Gmail “Show original”: shows SPF/DKIM/DMARC results for a received message.
Headers of delivered messages show which checks passed (Authentication-Results).

8. Read bounces and logs

sudo grep -E 'status=(bounced|deferred)' /var/log/mail.log | tail -20

Bounce messages explain rejections: 550 5.7.26 … unauthenticated (DMARC), 550 5.7.1 … blocked using (a blocklist), 421 4.7.0 … rate limited (reputation, so slow down). Gmail and Microsoft include links to their sender guidelines in the bounce.

Common problems and fixes

Symptom	Likely fix
Rejected: “PTR record” / “reverse DNS”	Set the PTR, matching the HELO
Rejected: “unauthenticated” / DMARC fail	Fix SPF and DKIM alignment
Delivered to spam at Gmail	Check Postmaster Tools, authentication, warm-up, content
Rejected: listed in Spamhaus	Fix the cause, request delisting
Outlook junk only	Check SNDS, join JMRP, warm up gradually
Delays / 421 rate limits	Lower the sending rate, improve reputation

Quick reference

Area	Check
Identity	PTR = HELO = A record
Authentication	SPF + DKIM + DMARC, aligned
Encryption	STARTTLS on port 25
Reputation	Blocklists, Postmaster Tools, SNDS
Bulk rules	One-click unsubscribe, complaints < 0.1%
Testing	mail-tester.com, Gmail “Show original”

Email
DNS
Postfix