Google's Mobile Armageddon and Your Robots.txt File

If you work on websites, you've probably heard about Google's "Mobile Armageddon". Google is changing its search results to benefit sites that are mobile-friendly.

Google provided a mobile-friendly test that you can use to check whether your site will benefit from these new changes.

I used Google's test for this site, and I was surprised that some of our main pages were being marked as "Not mobile friendly", even though the pages clearly were mobile-friendly.

After digging deeper, I found out it was a simple fix and all I had to do was modify our robots.txt file to give Google access.

In this tutorial, I'll show you how to check your site with Google and - if your site is responsive - how to resolve any issues with the robots.txt.

Background on the mobile armageddon

The robots.txt tells search engines what pages they shouldn't index.

As more people have started using mobile devices, Google's been modifying their search algorithms to make it easier for mobile devices to find mobile friendly sites.

In February, Google announced that on April 21st, they would rely more on their "mobile-friendliness" signal for ranking and that it would have a significant impact on search results.

The April 21st change is what some people on the web have termed as Google's "Mobile Armageddon" since many sites that aren't mobile friendly will likely have a significant loss in traffic over the coming months.

Checking whether Google thinks your site is mobile-friendly

By now, you've probably heard of "responsive design" and know that it helps to make your site mobile-friendly. A responsively designed web site adjusts to the width of the device viewing it.

If the site doesn't adjust to the width of your mobile device and if you have to scroll horizontally to see the whole page, it's not considered mobile-friendly and you'll likely see a loss of ranking as a result.

If you navigate your site with a mobile device and it looks, navigates, and functions great, it's mobile-friendly. However...

It doesn't matter if you see it as mobile-friendly, but rather whether Google sees it as mobile-friendly.

If your site is mobile-friendly for you, go to https://www.google.com/webmasters/tools/mobile-friendly/ and double check that Google agrees with you. You should primarily check your home page and your main traffic generating pages. You might be surprised by the results, as I was.

Google's Mobile Armageddon and Your Robots.txt File

Here's a screenshot of a site that's responsive, but is showing in Google as "Not mobile-friendly".

Thankfully, Google's tester is very informative and points out the exact issue for this particular site. It's that the robots.txt file is blocking Google from accessing the resources it needs.

The robots.txt fix

Here's the robots.txt file for the particular site.

It's a Joomla site, which has much of the resources that Google needs stored in the "media" and "templates" folder. Also, there are some resources in "components", "plugins", and "modules". So here are the lines that I'm going to remove:

Disallow: /components/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/

Update: Instead of removing the above, I can simply add the following rules underneath the rest of the code:

Allow: *.css

Allow: *.js

This will tell Google, and other search engines, to allow indexing of any .css and js files on the site.

You can find the exact resources that are blocked by Google by clicking on the "Show resources" link in Google's tester. Compare the files there to your robots.txt to determine what needs unblocking.

Now Google agrees that we're mobile-friendly

After saving your robots.txt file, go back to Google's tester and run another test.

Awesome! This page is now mobile-friendly to Google as well.


About the author

Nick is the Director of Support at OSTraining and you can find him in almost every area of the site, from answering support requests and account questions to creating tutorials and software.