How to Set Up a Mail Server on a VPS: Postfix, Dovecot, and Anti-Spam Configuration

Running your own mail server on a VPS gives you complete control over your email infrastructure — full privacy, custom domain email, and no dependence on third-party providers. It is also one of the more complex server administration tasks due to the number of components required and the anti-spam measures necessary to ensure your outbound email is delivered reliably. This guide covers the complete stack: Postfix for SMTP, Dovecot for IMAP, and the critical DNS and anti-spam configuration that determines whether your emails land in the inbox or the spam folder.

Is Self-Hosting Email Right for You?

Before proceeding, honestly assess whether self-hosted email suits your situation:

• Self-hosted email is appropriate for: Organizations with privacy requirements, developers who want to understand email infrastructure, businesses that have exhausted hosted email options, or those who want custom integrations impossible with SaaS providers.
• Self-hosted email requires ongoing maintenance: Monitoring spam blacklists, keeping software updated, managing disk space for mailboxes, and diagnosing deliverability issues are ongoing responsibilities.
• Alternatives to consider: For most small businesses, a managed email service (Google Workspace, Fastmail, Migadu) provides better deliverability with less maintenance effort.

If self-hosted email is the right choice, proceed with the setup below.

Prerequisites

• A KVM VPS with Ubuntu 22.04 LTS and at least 1 GB RAM (2 GB recommended for Rspamd)
• A dedicated IP address — shared IPs may be blacklisted by previous tenants
• A domain name with DNS management access
• Reverse DNS (PTR record) configured for your VPS IP — contact your provider to set this up if it is not self-service
• Ports 25, 465, 587, 993 open in your firewall

Step 1: Configure DNS Records (Do This First)

DNS configuration must be done before or simultaneously with server setup, as DNS propagation takes time. Add these records to your domain:

A Record

mail.yourdomain.com.   IN  A  YOUR_VPS_IP

MX Record

yourdomain.com.   IN  MX  10  mail.yourdomain.com.

SPF Record (TXT)

yourdomain.com.   IN  TXT  "v=spf1 mx a ip4:YOUR_VPS_IP ~all"

SPF tells receiving mail servers which IPs are authorized to send email for your domain. Without a valid SPF record, your email will frequently be marked as spam.

DMARC Record (TXT) — Configure after DKIM is set up

_dmarc.yourdomain.com.   IN  TXT  "v=DMARC1; p=quarantine; rua=mailto:dmarc@yourdomain.com"

Reverse DNS (PTR)

Contact your VPS provider to set the PTR record for your IP to mail.yourdomain.com. This is critical — many receiving mail servers reject email from IPs without a matching PTR record. At VPS.DO, this can be requested through the support ticket system.

Step 2: Prepare the Server

sudo apt update && sudo apt upgrade -y

# Set the hostname to match your mail server's FQDN
sudo hostnamectl set-hostname mail.yourdomain.com

# Update /etc/hosts
sudo nano /etc/hosts
# Add: YOUR_VPS_IP  mail.yourdomain.com mail

Step 3: Install and Configure Postfix (SMTP Server)

sudo apt install postfix postfix-mysql -y

During installation, select “Internet Site” and enter your domain name (yourdomain.com) when prompted.

Edit the main Postfix configuration:

sudo nano /etc/postfix/main.cf

# Basic settings
myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
inet_interfaces = all
inet_protocols = ipv4

# Accepted domains
mydestination = $myhostname, localhost.$mydomain, localhost

# Relay settings
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
relayhost =

# Mailbox delivery
home_mailbox = Maildir/
mailbox_size_limit = 0

# TLS settings
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# Anti-spam
smtpd_helo_required = yes
smtpd_recipient_restrictions =
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination,
    reject_rbl_client zen.spamhaus.org,
    reject_rbl_client bl.spamcop.net

Step 4: Issue SSL Certificate for Mail Server

sudo apt install certbot -y
sudo certbot certonly --standalone -d mail.yourdomain.com

Note: Stop Nginx or any service using port 80 temporarily if needed during certificate issuance.

Step 5: Install and Configure Dovecot (IMAP Server)

sudo apt install dovecot-core dovecot-imapd dovecot-lmtpd -y

Configure Dovecot’s main settings:

sudo nano /etc/dovecot/dovecot.conf

protocols = imap lmtp
listen = *, ::

sudo nano /etc/dovecot/conf.d/10-mail.conf

mail_location = maildir:~/Maildir
namespace inbox {
    inbox = yes
}

sudo nano /etc/dovecot/conf.d/10-ssl.conf

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
ssl_min_protocol = TLSv1.2
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256

sudo nano /etc/dovecot/conf.d/10-auth.conf

disable_plaintext_auth = yes
auth_mechanisms = plain login

Configure Postfix to use Dovecot LMTP for local delivery:

sudo nano /etc/postfix/main.cf

Add:

mailbox_transport = lmtp:unix:private/dovecot-lmtp

sudo systemctl restart postfix dovecot
sudo systemctl enable postfix dovecot

Step 6: Set Up DKIM Signing with OpenDKIM

DKIM cryptographically signs outbound email, proving it originated from your server and was not modified in transit:

sudo apt install opendkim opendkim-tools -y

sudo nano /etc/opendkim.conf

Domain                  yourdomain.com
KeyFile                 /etc/opendkim/keys/yourdomain.com/mail.private
Selector                mail
Socket                  inet:12301@localhost
RequireSafeKeys         false

Generate the DKIM key pair:

sudo mkdir -p /etc/opendkim/keys/yourdomain.com
cd /etc/opendkim/keys/yourdomain.com
sudo opendkim-genkey -s mail -d yourdomain.com
sudo chown opendkim:opendkim mail.private

View the DNS record to add:

sudo cat /etc/opendkim/keys/yourdomain.com/mail.txt

Add this TXT record to your DNS exactly as shown (it will look like mail._domainkey.yourdomain.com).

Configure Postfix to use OpenDKIM:

sudo nano /etc/postfix/main.cf

Add:

milter_protocol = 2
milter_default_action = accept
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

sudo systemctl restart opendkim postfix

Step 7: Open Firewall Ports

sudo ufw allow 25/tcp comment 'SMTP'
sudo ufw allow 465/tcp comment 'SMTPS'
sudo ufw allow 587/tcp comment 'SMTP Submission'
sudo ufw allow 993/tcp comment 'IMAPS'

Step 8: Test Email Delivery

Use these tools to verify your mail server configuration before sending real email:

• mail-tester.com — Sends a test email and scores your SPF, DKIM, DMARC, content, and blacklist status. Aim for 10/10.
• mxtoolbox.com — Check MX records, SPF, DKIM, DMARC, and IP blacklist status.
• DMARC Analyzer — Verify your DMARC record is correctly parsed.

Common Deliverability Issues and Fixes

Getting Started

A mail server needs a dedicated IP address — verify your VPS provider offers this. USA VPS plans at VPS.DO include a dedicated public IP, and support can assist with PTR (reverse DNS) configuration via ticket. For mail server deployments, a VPS with at least 1 GB RAM is the practical minimum, with 2 GB recommended if you add Rspamd for advanced spam filtering.

Conclusion

A self-hosted mail server with Postfix, Dovecot, and proper DKIM/SPF/DMARC configuration delivers full control over your email infrastructure. The setup is complex but well-documented, and the ongoing maintenance is manageable for technically capable teams. The key to reliable deliverability is the DNS configuration — SPF, DKIM, DMARC, and a correctly set PTR record are non-negotiable for inbox placement.