What is a robots.txt file?

When creating your website, you will need to communicate certain information to search engines. This information tends to be about the nature of your website, and can also point the search engine to your sitemap. You may also want to keep some of the pages of your website private, hence you will want to instruct a search engine not to display these pages, when someone searches for your website online. When a search engine robot is looking for websites, it will first try to find a /robot.txt file that is associated with that website. This file will contain all of the information you wish to pass on to the search engine robot. This information will mainly consist of what you want the search engine to display, or to conceal, regarding your site, should anyone enter a search for your site into the search engine. Its worth noting, that if malicious software (malware), tries to access your site, it can choose to ignore the /robot.txt file assigned to your website for nefarious means. 

A handy guide on robots.txt file can be found at http://www.robotstxt.org/robotstxt.html 

If you should have any issues or have any questions regarding this topic please contact us via a support ticket (open a ticket) with our hosting department and we will get back to you as soon as possible

 

 

 

  • searchengine optimisation, seo
  • 0 Users Found This Useful
Was this answer helpful?

Related Articles

Are you ready for Black Friday and Cyber Monday?

Well......it looks like we've arrived at that time of year again! Everybody loves a bargain don't...

Information on SSL

In the world of the world wide web, it is extremely important to protect our information. We need...

What is a Cookie?

A cookie can be explained in a couple of ways. Those being, a delicious snack, or an essential...

What is Caching?

Caching (pronounced 'cashing'), is a temporary storage area for downloaded files. This storage...

5 ways to reduce the size of your website.....

Storage, the never ending problem! No matter how much we think we have, its never quite enough is...

Powered by WHMCompleteSolution