WordPress has announced a significant change to how it will block search engines from indexing websites. This update shifts from the traditional Robots.txt method to the Robots Meta Tag approach. The change aligns WordPress with the objective of preventing blocked pages from appearing in Google’s search results.
New Robots Meta Tag
WordPress will use the following Robots Meta Tag:
<meta name=’robots’ content=’noindex,nofollow’ />Blocking Google From Indexing
Traditionally, Robots.txt has been used to block the "indexing" of a website. The term "indexing" in this context referred to GoogleBot crawling the site. By blocking GoogleBot via Robots.txt, it was assumed Google would not show these pages in its search results. However, the Robots.txt directive only prevented Google from crawling the page; Google could still add the page URL to its index if discovered.
To effectively prevent a site from appearing in search results, blocking Google from "indexing" pages using Robots.txt wasn’t consistently effective.
WordPress 5.3 and Indexing
WordPress previously adopted the Robots.txt approach, but this is set to change in version 5.3. Currently, when publishers select "discourage search engines from indexing this site," it adds an entry to the site’s Robots.txt file that stops Google from crawling the site. With version 5.3, WordPress will adopt the more reliable Robots Meta Tag method to prevent website indexing.
The new approach will affect the "discourage search engines from indexing this site" setting.
Benefits
This change is a significant improvement, ensuring that blocked web pages will not appear in Google’s search results, giving WordPress publishers more control and security over their site’s visibility.
Previous Methods and Their Limitations
WordPress used Robots.txt to block website indexing because it was the standard practice to keep pages from appearing in Google’s search results. However, the Robots.txt method was unreliable.
"Indexing" has two meanings:
- Indexing as crawling, i.e., Googlebot visiting and downloading web pages.
- Indexing as adding a web page to Google’s database of web pages (The Index).
Blocking Google from "indexing" via Robots.txt prevented Google from seeing the web page, but Google could still add it to its index.
Robots.txt vs. Robots Meta Tag
The intent of Robots.txt was not to keep web pages out of Google’s index. This task is best performed by the Robots Meta Tag. It’s promising to see WordPress adopting the Robots Meta Tag to block web pages from search engines.
WordPress 5.3 is scheduled to be released in November 2019.
Related Reading
- Robots meta tag and X-Robots-Tag HTTP header specifications
- Block search indexing with ‘noindex’
