VPS DDoS Mitigation: Practical Defenses Against Volumetric, Protocol, and Application Attacks

A VPS has limited bandwidth and CPU compared to a dedicated server — a sustained DDoS attack can exhaust resources and take down your services. While true volumetric attacks (10+ Gbps) require CDN-level protection, most VPS attacks are smaller application-layer floods that practical server-side defenses can survive. This guide implements layered DDoS mitigation: kernel TCP hardening, Nginx rate limiting, Cloudflare proxy, and detection automation.

Types of Attacks and Defenses

Layer 1: Kernel SYN Flood Protection

<code">sudo nano /etc/sysctl.d/99-ddos-protection.conf

<code"># SYN Cookies: respond to SYN without allocating state until ACK received
# Prevents SYN flood from exhausting connection table
net.ipv4.tcp_syncookies = 1

# Increase SYN backlog — more half-open connections before dropping
net.ipv4.tcp_max_syn_backlog = 65535

# Reduce SYN-ACK retries — fail faster on incomplete handshakes
net.ipv4.tcp_synack_retries = 2

# Increase accept queue — more connections waiting in fully established queue
net.core.somaxconn = 65535

# Enable RFC 1337 — reject RST packets from TIME_WAIT connections
net.ipv4.tcp_rfc1337 = 1

# Ignore ICMP broadcast requests (Smurf attack protection)
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Ignore bogus ICMP error responses
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log martian packets (impossible source addresses)
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1

<code">sudo sysctl -p /etc/sysctl.d/99-ddos-protection.conf

Layer 2: UFW Rate Limiting and Protocol Drops

<code"># Drop invalid packets (not part of any established connection)
sudo iptables -A INPUT -m state --state INVALID -j DROP

# Rate-limit new TCP connections (SYN packets) per source IP
sudo iptables -A INPUT -p tcp --syn -m connlimit \
    --connlimit-above 50 --connlimit-mask 32 -j DROP

# Limit ICMP ping rate (anti-ping flood)
sudo iptables -A INPUT -p icmp -m limit \
    --limit 1/second --limit-burst 5 -j ACCEPT
sudo iptables -A INPUT -p icmp -j DROP

# Block UDP floods to non-essential ports (keep DNS open if needed)
sudo ufw deny proto udp from any to any port 1:52
sudo ufw deny proto udp from any to any port 54:65535

# Save iptables rules
sudo apt install -y iptables-persistent
sudo netfilter-persistent save

Layer 3: Nginx Rate Limiting and Connection Controls

<code">sudo nano /etc/nginx/nginx.conf

<code">http {
    # ── Rate limiting zones ─────────────────────────────────
    # Zone for general HTTP requests (per IP)
    limit_req_zone $binary_remote_addr zone=general:10m rate=30r/s;

    # Tighter zone for login/auth endpoints
    limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;

    # Zone for search/expensive operations
    limit_req_zone $binary_remote_addr zone=search:10m rate=10r/m;

    # ── Connection limits ──────────────────────────────────
    # Maximum connections per IP
    limit_conn_zone $binary_remote_addr zone=addr:10m;

    # ── Slowloris protection ────────────────────────────────
    client_body_timeout 10s;     # Abort if body not received in 10s
    client_header_timeout 10s;   # Abort if headers not received in 10s
    send_timeout 10s;            # Abort if not sending to client in 10s
    keepalive_timeout 15s;       # Keep connections alive max 15s
    keepalive_requests 100;      # Max requests per keepalive connection

    # ── Request size limits ────────────────────────────────
    client_body_buffer_size 1k;
    client_header_buffer_size 1k;
    large_client_header_buffers 2 1k;
}

<code">sudo nano /etc/nginx/sites-available/yoursite

<code">server {
    listen 443 ssl http2;
    server_name yourdomain.com;

    # General rate limit: 30 req/s, burst 50, then 429
    limit_req zone=general burst=50 nodelay;
    limit_req_status 429;

    # Max connections per IP
    limit_conn addr 20;
    limit_conn_status 429;

    # WordPress login — very tight limit
    location = /wp-login.php {
        limit_req zone=login burst=5;
        proxy_pass http://127.0.0.1:8080;
    }

    # API search endpoint
    location /api/search {
        limit_req zone=search burst=10;
        proxy_pass http://127.0.0.1:3000;
    }

    # Block common bad bots by User-Agent
    if ($http_user_agent ~* (masscan|nikto|sqlmap|nmap|zgrab|nuclei)) {
        return 444;   # Close connection silently (no response)
    }

    # Block requests without Host header (scanner behavior)
    if ($host = "") {
        return 444;
    }
}

Layer 4: Cloudflare for Volumetric Attack Absorption

For attacks exceeding your VPS bandwidth (typically 1–10 Gbps), the VPS itself cannot absorb the traffic. Cloudflare’s free plan absorbs volumetric attacks before they reach your server:

1. Add your domain to Cloudflare → change nameservers to Cloudflare’s
2. Security → Under Attack Mode (temporary — shows JS challenge to all visitors)
3. Security → WAF → Rate Limiting rules (paid plan)
4. Block direct-to-IP access so attackers can’t bypass Cloudflare:

<code"># Allow only Cloudflare IP ranges (they publish these)
# /etc/nginx/snippets/cloudflare-ips.conf
# https://www.cloudflare.com/ips/

# In UFW — block direct IP access, allow only Cloudflare:
for ip in $(curl -s https://www.cloudflare.com/ips-v4 https://www.cloudflare.com/ips-v6); do
    sudo ufw allow from $ip to any port 443 proto tcp
done
sudo ufw delete allow 443   # Remove the open-to-all 443 rule

Layer 5: Automated Detection and Response

<code">sudo nano /usr/local/bin/ddos-detect.sh

<code">#!/bin/bash
# Detect high-connection-count IPs and auto-ban via UFW
THRESHOLD=100    # Ban IPs with more than 100 connections
LOG="/var/log/ddos-detect.log"

while true; do
    # Find IPs exceeding threshold
    ss -nt state established | \
        awk '{print $5}' | \
        grep -oP '[\d.]+(?=:\d+$)' | \
        sort | uniq -c | \
        awk -v t="$THRESHOLD" '$1 > t {print $2, $1}' | \
    while read ip count; do
        # Skip whitelisted IPs
        [[ "$ip" =~ ^(127\.|10\.|192\.168\.) ]] && continue

        # Ban if not already banned
        if ! sudo ufw status | grep -q "$ip"; then
            echo "[$(date)] Banning $ip ($count connections)" >> "$LOG"
            sudo ufw insert 1 deny from "$ip"
        fi
    done

    sleep 30
done

<code">chmod +x /usr/local/bin/ddos-detect.sh

# Run as systemd service
sudo nano /etc/systemd/system/ddos-detect.service

<code">[Unit]
Description=DDoS Detection
After=network.target

[Service]
ExecStart=/usr/local/bin/ddos-detect.sh
Restart=always

[Install]
WantedBy=multi-user.target

<code">sudo systemctl enable ddos-detect
sudo systemctl start ddos-detect

Recovery During an Active Attack

<code"># 1. Identify top source IPs
ss -nt state established | awk '{print $5}' | \
    grep -oP '[\d.]+(?=:\d+$)' | sort | uniq -c | sort -rn | head -20

# 2. Block top attacker IPs immediately
sudo ufw insert 1 deny from 198.51.100.0/24

# 3. Enable Nginx rate limiting if not already active
sudo nginx -t && sudo systemctl reload nginx

# 4. Enable Cloudflare Under Attack Mode if attack is volumetric

# 5. Monitor connections dropping
watch -n 2 "ss -s | grep -E 'estab|TCP'"

Getting Started

The kernel sysctl changes and Nginx rate limiting in this guide activate within minutes on any Ubuntu VPS at VPS.DO and provide substantial protection against application-layer and protocol-level attacks. For sustained volumetric attacks, proxy through Cloudflare’s free plan — the combination of Cloudflare’s network-layer absorption and server-side rate limiting handles the vast majority of attacks targeting typical VPS workloads.

Conclusion

VPS DDoS mitigation is layered: kernel SYN cookies and backlog tuning handle protocol floods, Nginx rate limiting and connection limits handle application-layer floods, and Cloudflare absorbs volumetric attacks that exceed VPS bandwidth. The automated detection script bans IPs with suspicious connection counts before they saturate Nginx workers. No single layer is complete — the combination provides defense-in-depth that keeps services available through common attack scenarios.