Google is now sending out Google Webmaster Tools alerts for crawl errors with websites. The alerts are organized into two categories:
- Website Error alerts for major site-wide problems
- URL Error anomaly alerts for less critical issues
Examples of major site-wide crawl error issues
- Your DNS server is down or misconfigured.
- Your web server itself is firewalled off.
- Your web server is refusing connections from Googlebot.
- Your web server is overloaded, or down.
- Your site’s robots.txt is inaccessible.
Examples of URL Error Anomaly Alerts
- Server error
- Soft 404
- Access denied
- Not found
- Not followed
The reason why Google can detect and distinguish such crawl errors is due to the fact that the Googlebot regularly crawls websites and can find irregularities in crawl patterns. Systemic errors experienced by the Googlebot can indicate problems by average users.