Robots.txt
Robots.txt is a file used to monitor what visiting bots can see and index from your website. It was created in 1994, and consists of a txt. file placed in the root of the website hierarchy. It uses simple commands to inform visiting bots to which specific areas of your site you’d like them to avoid, such as images, videos, directories and pages such as your terms & conditions.
This is an advantage if you own a number of websites where your terms and conditions are the same text as Google bots may treat these as being from the same website which would affect your search engine results. When used with features on Google Webmaster Tools (look under the Diagnostics tab) you can exercise some level of control over what is listed from your site on search engine results. Robots.txt files cannot completely guarantee privacy, but they are a useful aspect to learn about when discovering how to best optimize your own website.