Robots.txt for SEO

Posted by admin on February 25th, 2008 at 02:40pm

Today we will discuss the importance SEO wise of a websites robots.txt file. I am constantly amazed at the number of large and notable sites (including SEO company sites) which have ignored or are unaware of this important SEO tactic.

For those of you who are asking; what the heck is a robots.txt file and what does it have to do with search engine optimization, the answer is that in simple terms this is the traffic-cop of search engine bots and instructs those bots which parts of a website they should and should not follow.

While having a robots.txt file will not have any direct effect on your search engine rankings in the SERP’s (Search Engine Results Pages), the importance of this from a SEO vantage point should be obvious. Helping the search engine robots effectively navigate your site can only aid you in your SEO efforts by getting your webpages properly indexed in Google and the other search engines.

Creating the file is simple. Simply open any text editor like Notepad and copy and paste the following:

User-agent: *
Disallow:

Then save the file as “robots.txt” and upload this file into the root directory of your website.

The asterisk is a wildcard denoting all search engine robots and leaving the Disallow command blank allows them access to any parts of your site. If there are any areas of your website that for whatever reason you do not want the search engine spiders to visit simply place the file or folder name in the Disallow command like so:

User-agent: *
Disallow: /examplefolder/

Or:

User-agent: *
Disallow: /examplefile.html

If you feel that this is all above your head you can also just do a Google search for “robots.txt generator” or check our “SEO Tools” section for both a generator and robots.txt validator to ensure yours is properly formatted.

Under SEO