What is a robots.txt file?

When creating your website you will need to communicate certain information to search engines about the nature of your website (pointing the search engine to your sitemap for instance). You may also want to keep some of the pages of your website private so will want to instruct a search engine not to display these when someone searches for your website online. When a search engine robot is looking for websites it will first try to find a /robot.txt file that is associated with that website. This file will contain all of the information you wish to pass on to the search engine robot. This information will mainly consist of what you want the search engine to display (or not to display) about your site should anyone enter a search for your site into the search engine. Its worth noting that if malicious software (malware) tries to access your site it can choose to ignore the /robot.txt file assigned to your website for nefarious means. 

A handy guide on robots.txt file can be found at http://www.robotstxt.org/robotstxt.html 

If you should have any issues or have any questions regarding this topic please contact us via a support ticket (open a ticket) with our hosting department and we will get back to you as soon as possible

 

 

 

  • searchengine optimisation, seo
  • 0 Users Found This Useful
Was this answer helpful?

Related Articles

Are you ready for Black Friday and Cyber Monday?

Well......it looks like we've arrived at that time of year again! Everybody loves a bargain don't...

Information on SSL

In the world of the world wide web it is extremely important to protect our information and...

What is a Cookie?

A cookie can be explained in a couple of ways......a delicious snack or an essential piece of...

What is Caching?

Caching (pronounced 'cashing') is a temporary storage area for downloaded files, usually files...

5 ways to reduce the size of your website.....

Storage, the never ending problem! No matter how much we think we have, its never quite enough is...

Powered by WHMCompleteSolution