How to Fix the Googlebot CSS and JS Problem in robots.txt

One of the many responsibilities that falls on bloggers and website owners is search engine optimization (SEO) other related chores, such as ensuring search engines can easily read your site. Recently, Google Webmaster Tools sent out notices to thousands of site owners across several platforms informing them that there was a Googlebot CSS and JS problem in their site’s robots.txt file. Fix it, or the Google would be angry. In other terms, Google updated the rules of the game (again) and needed their robots to crawl more items in your site.

Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings.

I’m sure you’re thinking, “What? I have robots on my site? And they crawl?”

Let’s back up. Robots, or bots for short, are what search engines use to get an idea of what your site is about. They crawl (read) the content we see, alt tags and descriptions for images, and more. However, now that more people are relying on mobile devices, Google also needs to analyze the mobile versions of our sites. As website owners we have the ability to indicate to search engines what files to crawl and which ones to not crawl via the robots.txt file.

How to fix the CSS and JS issue with robots.txt the easy way

Google was having problems reading certain pages of information that helps it understand how your site should look like on mobile devices. Remember when we were told that our sites had to be mobile-friendly and our SEO would suffer otherwise? This new request is Google’s way of 1) making sure your site is mobile-friendly and 2) making sure it understands how the mobile-friendly portion of your site should render (appear) across devices.

I use WordPress as my blogging platform, and I use Wordfence as my security plugin. Here’s what my robots.txt file looked like, which caused Google to ask me to update the permissions:

wp-admin is where the CSS and JS files are housed by WordPress. And, seeing as the robots.txt was disallowing bots to crawl it, Google asked me to fix this. However, I don’t want to open the floodgates and allow search engines to search ALL of my admin files, as it could be a security issue. Instead, I just wanted to allow the bots to search the CSS and JS files they needed.

Googlebot Screenshots: Before and After fixing robots.txt

Here’s how Googlebots saw my site on smartphones before I updated my robots.txt file:

How Google sees your site before fixing robots file

I found this by expanding the results in the Fetch as Google tab in my Webmaster Tools account. The JavaScript was missing, and it’s evident in the navigation bar (all the links are expanded and there’s no collapsed “hamburger” menu) and the slider and testimonials did not show up. These are all JavaScript elements.

Here’s how Googlebots saw my site on smartphones after I updated my robots.txt file:

How Google sees your site after fixing robots.txt file

They look exactly the same, which is the goal: to have Googlebots see your site exactly as human eyes would see it.

The Solution

Thanks to the generous WordPress community I was able to find the easiest solution, and I thought I’d share the wealth. By adding three more lines of text allowing the bots to ONLY read the CSS and JavaScript files within my admin area, I am helping to keep my site secure while giving ’em what they want:

How to find your robots.txt file

If you don’t use WordPress SEO by Yoast (highly recommended) you will need to find your robots.txt file in your site’s files in the cPanel File Manager or via FTP access. I prefer to do this the easy (and smart) way so I use the File Editor in my WordPress SEO plugin.

Go to SEO > Tools > File Editor.

Important: Make sure you do NOT touch the .htaccess file or you will face the white screen of death

If you do not have a robots.txt file, create one and paste the code from above.

How to Fix CSS and JS error | Fetch as Google after updating robots.txt file.

Save your changes. You will need to go back to your Webmaster Tools account and follow their instructions:

How to Fix CSS and JS error | Fetch as Google after updating robots.txt file.
How to Fix CSS and JS error | Fetch as Google after updating robots.txt file.

Note that the earliest fetches (before I completed all these steps) were labeled as Partial by Google, while after the fix they considered it Complete* (green text). Be sure to Fetch as Google for both desktop and smartphone to cover all your bases.

This is one of many ways to fix the issue, but I like its simplicity and effectiveness. I’ve learned that it takes a little longer for some sites owners’ robots.txt files to reflect the changes in the Webmaster account. Be patient and follow up with it!

*Note: if you run ads from Adsense or ad networks like BlogHer or GLAM Media, you will not see a “Complete” render as we have no control over the scripts of the ads. We can only set rules for Googlebots on our site, not third-party scripts.

One Response

Leave a Reply

Your email address will not be published. Required fields are marked *

As a fellow creative, I understand all too well the importance of a first impression. Curious about why I love what I do?