In a recent Webmaster Hangout, Google’s John Mueller addressed the issue of preventing Google from crawling and indexing a staging server.
What is a Staging Server?
A staging server, or staging site, is a part of a server or website used to test new versions before they go live. For instance, a publisher might create a subdomain to test a new web design.
Traditionally, the most common method to block Google from indexing a staging site involved creating a robots.txt file to disallow Google from crawling the staging pages.
Google’s John Mueller’s Recommendation
John Mueller suggested that the best method to stop Google from crawling and indexing a staging site is to implement authentication to block access to the web pages.
Mueller’s Explanation:
“There are multiple ways… people do it in multiple ways. I think that the important part is that you don’t link to it. Because if we don’t find it then we can’t crawl it. But sometimes that still happens.”
Mueller’s comment referenced the use of robots.txt to block crawler access. However, if Google finds a link to a blocked page, it will attempt to crawl the URL.
John continued with a more effective solution:
“Ideally, what you would want to do is provide some kind of server-side authentication so that normal users, including GoogleBot, are blocked from accessing the content. This could be done based on IP address, with a cookie, or using standard authentication methods. Anything that requires verification to view the content is the best approach for staging servers…”
Mueller explained that a staging site with authentication eliminates the need to alter settings like robots.txt or noindex meta tags on the live site, reducing the risk of mistakes.
By using password protection, you can prevent Google and others from accessing the staging site without risking errors in settings on the live site.
How to Create a Password Protected Staging Site
There are several methods to create a password-protected staging site:
- Dedicated Server: Use your control panel to create a password-protected subdomain.
- WordPress on Shared Hosting: Use a third-party staging server tool or a plugin designed for staging sites.
External tools can make it straightforward to stage a site in a password-protected environment and to push changes live with minimal effort.
Note: Always back up your website before migrating. Using a reliable backup plugin can make this process easier.
Returning to Mueller’s advice, staging a site behind a password-protected directory or subdomain is sound strategy.
Watch John Mueller’s detailed explanation for context and further insights.
