Blog: SEO
googlebot spider

Google Changes How It Crawls WordPress Sites

Avatar for John Locke

John Locke is a SEO consultant from Sacramento, CA. He helps manufacturing businesses rank higher through his web agency, Lockedown Design & SEO.

Some of you may have woken up this morning to emails from Google Webmaster Tools, telling you Googlebot cannot access CSS and JS files on your site.

If you have a WordPress site with a robots.txt file, this may be the cause.

Up until July 28th, 2015, it was common for Google to crawl WordPress sites with a robots.txt file that blocked certain directories from user-agents (browsers) as a security precaution. Starting today, Google is requesting they be able to crawl CSS and JS files for your WordPress site, so they can fully render the page, and make sure it is functioning correctly.

Not every WordPress site has a robots.txt file. These are located in the root folder of the site or WordPress installation.

Many managed WordPress hosts and some security plugins include a robots.txt. If you have a security plugin installed, you may also have to check to make sure you are not restricting Googlebot in your .htaccess file (also at in the root folder).

Here’s an example of a robots.txt file that restricts access to the wp-includes folder, where many of your CSS and JS files reside.


# Robots file, acceptable before July 28, 2015
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Here is an example of your robots.txt file, with the restriction on the wp-includes folder removed.


# Robots file, acceptable after July 28, 2015
User-agent: *
Disallow: /wp-admin/

The team at Yoast recommend not blocking the wp-admin at all, as WordPress automatically does this on a page level, using the tag which looks like this:


<meta name='robots' content='noindex,follow' />>

The important thing to know is going forward, Google needs to be able to crawl your wp-includes folder and any CSS or JavaScript necessary to render your site. You can test to see if Google can successfully crawl your site in your Google Search Console under Crawl > Fetch As Google.

You can learn more about robots.txt and test at Google’s Search Console Help. You can also test your robots.txt in your Google Search Console (formerly Google Webmaster Tools) under Crawl > robots.txt Tester.

Avatar for John Locke

John Locke is a SEO consultant from Sacramento, CA. He helps manufacturing businesses rank higher through his web agency, Lockedown Design & SEO.

Join the Conversation

Your email address will be kept private. Required fields marked *.