Robots Exclusion Protocol
The Robots Exclusion Protocol is a method that allows Web site administrators to indicate to visiting robots which parts of their site should not be visited by the robot. When a Robot visits a web site, it first checks for the file robots.txt in the root directory; e.g. http://exapmle.com/robots.txt. If it can find this file, it will analyze its contents to see if it may retrieve further documents (files). You can customize the robots.txt file to apply only to specific robots, and to disallow access to specific directories or files. Note that there can only be a single “/robots.txt” on a site.
The Robots META tag
The Robots META tag allows HTML authors to indicate to visiting robots if a document may be indexed, or used to harvest more links. No server administrator action is required. Note that currently only a few robots implement this.
< META name="robots" content="noindex,nofollow" >
Parameters that can be passed
ALL = INDEX, FOLLOW
NONE = NOINDEX, NOFOLLOW
INDEX Index this page.
FOLLOW Follow links from this page.
NOINDEX Don't index this page.
NOFOLLOW Don't follow links from this page.
The Robots META tag name and values are case-insensitive. Note: In early 1997 only a few robots implemented this, but this is expected to change as more public attention is given to controlling indexing robots.