Cloudflare WAF Custom Rules to Block Scrapers & L7 Botnets

Construct edge firewall expressions to block automated headless browsers, AI scrapers, and malicious botnet clusters before they touch origin servers.

1. Attack Signature & Impact Analysis

High (Origin Bandwidth Squeeze)

Commercial scrapers and botnets rotate residential proxies to bypass simple IP blacklists, mimicking genuine browser user-agents while stripping cookies.

2. Production Hardening Configuration (`Cloudflare Dashboard -> Security -> WAF -> Custom Rules`)

Deploy this validated configuration to your production server or reverse proxy:

# Cloudflare Firewall Expression: Block automated threats & empty User-Agents
(
  (http.user_agent eq "") or
  (http.user_agent contains "python-requests") or
  (http.user_agent contains "curl") or
  (http.user_agent contains "Go-http-client") or
  (cf.bot_management.score lt 20 and not cf.bot_management.verified_bot)
)

# Action: Managed Challenge (Captchas bots, transparent pass for legitimate users)
# Action for known attack regions: Block or Interactive Challenge

3. Live Simulation & Rule Verification Command

Execute these commands to verify that traffic is actively mitigated without service downtime:

# Test if rule challenges headless requests
curl -I -A "python-requests/2.28.1" https://yourdomain.com
# Response should return HTTP 403 Forbidden or Cloudflare Managed Challenge HTML

4. Common Misconfigurations & Troubleshooting Gotchas

⚠️ Production Caution: Always exempt `cf.bot_management.verified_bot eq true` from blocking rules, otherwise Googlebot, Bingbot, and legitimate search spiders will be blocked, tanking your SEO rankings!

5. Best Paired With (Complementary Defense Recipes)

Security engineers implementing this rule frequently deploy these complementary baselines: