One of our students was having trouble removing URLs from Google and received this message:
"Your request has been denied because the webmaster of the site hasn't applied the appropriate robots.txt file or meta tags to block us from indexing or archiving this page. Please work with the webmaster of this site or select an alternate removal option from the webpage removal request tool"
So we created this tutorial for him, which shows how to edit Joomla's robot.txt file to block search engines from crawling certain URLs, as well as the whole site if desired.
Access robots.txt in Joomla Root
Access your host's file manager, e.g. cPanel, plesk, etc.
In the root of your Joomla installation you will find a robots.txt file which you need to open and edit.
By default Joomla's robots.txt file should contain these rules for security measures:
How to block
Disallow: /pathto/page.html # blocks just this page
For example, if you want to block www.yoursite.com/clients/testimonials/business.html use:
Disallow: /clients/testimonials/business.html #
Once you are done, save the robots.txt file.
All of our tutorials are published under the Creative Commons Attribution-NonCommercial license. This means: