When creating your website, you will need to communicate certain information to search engines. This information tends to be about the nature of your website, and can also point the search engine to your sitemap. You may also want to keep some of the pages of your website private, hence you will want to instruct a search engine not to display these pages, when someone searches for your website online. When a search engine robot is looking for websites, it will first try to find a /robot.txt file that is associated with that website. This file will contain all of the information you wish to pass on to the search engine robot. This information will mainly consist of what you want the search engine to display, or to conceal, regarding your site, should anyone enter a search for your site into the search engine. Its worth noting, that if malicious software (malware), tries to access your site, it can choose to ignore the /robot.txt file assigned to your website for nefarious means.
A handy guide on robots.txt file can be found at http://www.robotstxt.org/robotstxt.html
If you should have any issues or have any questions regarding this topic please contact us via a support ticket (open a ticket) with our hosting department and we will get back to you as soon as possible
