Google’s Martin Splitt addressed a question about malicious bots affecting site performance, offering tips that every SEO and site owner should implement.
Malicious Bots Are An SEO Problem
Many SEOs often overlook security and bot traffic during site audits. It’s not widely understood that security issues impact site performance and can cause inadequate crawling. Improving core web vitals won’t help if poor security is dragging down site performance.
Every website is under constant attack, and excessive crawling can trigger a “500 server error” response code, signaling an inability to serve web pages and hindering Google’s ability to crawl them.
How To Defend Against Bot Attacks
Someone asked Google how to combat scraper bots affecting their server performance.
This was the question:
“Our website is facing significant disruptions due to targeted scraping by automated software, causing performance issues, increased server load, and potential data security concerns. Despite IP blocking and other measures, the problem persists. What can we do?”
Martin Splitt suggested identifying the source of the attacks and notifying them of the abusive behavior. He also recommended utilizing the firewall capabilities of a CDN (Content Delivery Network).
Martin responded:
“This sounds like a distributed denial-of-service issue if the crawling is so aggressive that it causes performance degradation.
You can try identifying the owner of the network where the traffic is coming from, thanking their hoster, and sending an abuse notification. You can use WHOIS information for that, usually.
Alternatively, CDNs often have features to detect bot traffic and block it. By definition, they distribute traffic away from your server, which is beneficial. Most CDNs recognize legitimate search engine bots and won’t block them, but if that’s a concern, consider asking them before using their services.”
Will Google’s Advice Work?
Identifying the cloud provider or server data center hosting the malicious bots is good advice, but there are many scenarios where this won’t work.
Three Reasons Why Contacting Resource Providers Won’t Work
1. Many Bots Are Hidden
Bots often use VPNs and open source “Tor” networks to hide their source, defeating attempts to identify the cloud services or web host. Hackers also hide behind compromised home and business computers, known as botnets, making identification impossible.
2. Bots Switch IP Addresses
Some bots switch to a different network instantly when their IP is blocked, resuming their attack from another location. An attack originating from a German server might switch to an Asian network provider when blocked.
3. Inefficient Use Of Time
Contacting network providers about abusive users is futile when the traffic source is obscured or coming from hundreds of sources. Site owners may be surprised by the intensity of attacks. Dealing with even a small group of offenders is inefficient, with millions of other bots ready to replace any that are blocked.
And what about botnets made up of thousands of compromised computers worldwide? Notifying all those ISPs is impractical.
Contacting infrastructure providers is not a viable approach to stopping bots that impact site performance. Realistically, it’s a futile and inefficient waste of time.
Use A WAF To Block Bots
Using a Web Application Firewall (WAF) is a good idea, as suggested by Martin Splitt when he recommended a CDN. A CDN sends browsers and crawlers the requested web page from the closest server, speeding up site performance and reducing server resource usage for the site owner.
A CDN also includes a WAF, which automatically blocks malicious bots. Martin’s suggestion to use a CDN is solid, especially due to the additional benefit of improved site performance.
An option Martin didn’t mention is using a WordPress plugin WAF like Wordfence. Wordfence automatically blocks bots based on their behavior. For example, it will create a temporary IP block if a bot requests an excessive number of pages. If the bot switches IPs, it will recognize the behavior and block it again.
An alternative solution is a SaaS platform like Sucuri, which offers a WAF and a CDN to enhance performance. Both Wordfence and Sucuri are reliable WordPress security providers with limited but effective free versions.
See also: WordPress Security: 16 Steps to Secure & Protect Your Site
Listen to the question and answer at the 6:36 mark of the Google SEO Office Hours podcast:
Featured Image by Shutterstock/Krakenimages.com
