One of the many responsibilities that falls on bloggers and website owners is search engine optimization (SEO) other related chores, such as ensuring search engines can easily read your site. Recently, Google Webmaster Tools sent out notices to thousands of site owners across several platforms informing them that there was a Googlebot CSS and JS problem in their site’s robots.txt file. Fix it, or the Google would be angry. In other terms, Google updated the rules of the game (again) and needed their robots to crawl more items in your site.
I’m sure you’re thinking, “What? I have robots on my site? And they crawl?”
Let’s back up. Robots, or bots for short, are what search engines use to get an idea of what your site is about. They crawl (read) the content we see, alt tags and descriptions for images, and more. However, now that more people are relying on mobile devices, Google also needs to analyze the mobile versions of our sites. As website owners we have the ability to indicate to search engines what files to crawl and which ones to not crawl via the robots.txt file.
Google was having problems reading certain pages of information that helps it understand how your site should look like on mobile devices. Remember when we were told that our sites had to be mobile-friendly and our SEO would suffer otherwise? This new request is Google’s way of 1) making sure your site is mobile-friendly and 2) making sure it understands how the mobile-friendly portion of your site should render (appear) across devices.
I use WordPress as my blogging platform, and I use Wordfence as my security plugin. Here’s what my robots.txt file looked like, which caused Google to ask me to update the permissions:
wp-admin is where the CSS and JS files are housed by WordPress. And, seeing as the robots.txt was disallowing bots to crawl it, Google asked me to fix this. However, I don’t want to open the floodgates and allow search engines to search ALL of my admin files, as it could be a security issue. Instead, I just wanted to allow the bots to search the CSS and JS files they needed.
Googlebot Screenshots: Before and After fixing robots.txt
Here’s how Googlebots saw my site on smartphones before I updated my robots.txt file:
Here’s how Googlebots saw my site on smartphones after I updated my robots.txt file:
They look exactly the same, which is the goal: to have Googlebots see your site exactly as human eyes would see it.
How to find your robots.txt file
If you don’t use WordPress SEO by Yoast (highly recommended) you will need to find your robots.txt file in your site’s files in the cPanel File Manager or via FTP access. I prefer to do this the easy (and smart) way so I use the File Editor in my WordPress SEO plugin.
Go to SEO > Tools > File Editor.
Important: Make sure you do NOT touch the .htaccess file or you will face the white screen of death
If you do not have a robots.txt file, create one and paste the code from above.
Save your changes. You will need to go back to your Webmaster Tools account and follow their instructions:
Note that the earliest fetches (before I completed all these steps) were labeled as Partial by Google, while after the fix they considered it Complete* (green text). Be sure to Fetch as Google for both desktop and smartphone to cover all your bases.
This is one of many ways to fix the issue, but I like its simplicity and effectiveness. I’ve learned that it takes a little longer for some sites owners’ robots.txt files to reflect the changes in the Webmaster account. Be patient and follow up with it!
*Note: if you run ads from Adsense or ad networks like BlogHer or GLAM Media, you will not see a “Complete” render as we have no control over the scripts of the ads. We can only set rules for Googlebots on our site, not third-party scripts.