The importance of robots.txt in Search Engine Optimization

The Robots.txt file guides search engine bots through the pages that should be indexed. The Robots Exclusion Protocol – the rules for controlling how web pages on your site are indexed can be defined in robots.txt file. The file should lie at: http://www.yourdomain.com/robots.txt

The robost.txt file plays a vital role in the SEO process. It is important due to:
If you want to exclude pages that should be accessed ONLY by regesitered users.
If you want to exclude the duplicate content pages.

You may use the google webmaster tool to genearte and analyze the robots.txt file.
You can find out more information about robots.txt at http://www.robotstxt.org