python - Django 404 Email spam

Question

I recently launched a web application based Django, and have been very pleased with its results. I also turned on a feature in Django where you can have emails sent to MANAGERS for 404's by adding the middleware 'django.middleware.common.BrokenLinkEmailsMiddleware'. However, ever since I did that, I'm getting LOTS of spam requests hitting 404s. I'm not sure if they are bots or what but this is the information I'm getting from Django:

Referrer: http://34.212.239.19/index.php
Requested URL: /index.php
User agent: Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)
IP address: 172.31.23.16

Why am I getting requests to URL's that don't exist on my site and is there a way to filter out requests so I don't get emails for them? These URL's have never existed on my site (my site is very recently launched). I'm getting roughly 50-100 emails a day from spam requests to my site.

Answer 1

I can't imagine an automated way of filtering out spam as a non-existent URL is indistinguishable from a spam URL, but you can filter out usual suspects using IGNORABLE_404_URLS:

List of compiled regular expression objects describing URLs that should be ignored when reporting HTTP 404 errors via email (see Error reporting). Regular expressions are matched against request's full paths (including query string, if any). Use this if your site does not provide a commonly requested file such as favicon.ico or robots.txt.

For example:

import re
IGNORABLE_404_URLS = [
    re.compile(r'.(php|cgi)$'),
    re.compile(r'^/phpmyadmin/'),
]