»» Using a Robots Text File

Search engines will search for a special file called robots.txt before spidering your site. The Robots Text File is created specifically to give directions to web crawlers/robots. Place the following two lines in your robots.txt file if you wish to allow search engines to crawl/spider everything on your site:

User-agent: *
Disallow:

The * in the first line specifies that the directions are for all search engines. The second line indicates that nothing is disallowed.

Once you have created your robots.txt file, upload it to your websites main directory where your homepage and other HTML files are located.

to the top

»» Robots.txt Usage Examples

Below are several common examples of how you can use a robots.txt file to set parameters and control how different crawlers/robots access your website.

The following example would allow all crawlers/robots to access all files except for your images file.

User-agent: *
Disallow: /images/

The following would all crawlers/robots to crawl all files except the cgi-bin files, images directory, and your log files.

User-agent: *
Disallow: /cgi-bin/
Disallow: /logs/
Disallow: /images/

Using "Crawl-delay" parameters in the robots.txt file. This parameter indicates the number of seconds for a crawler/spider to delay between requests.

User-agent: Googlebot
Crawl-delay: 20

User-agent: Slurp
Crawl-delay: 20

User-Agent: msnbot
Crawl-Delay: 20

Bad Robots and Email Harvestors

Below are several robots/crawlers that you might want to block.

User-agent: Titan
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: ExtractorPro
User-agent: WebZip
User-agent: larbin
User-agent: b2w/0.1
User-agent: htdig/3.1.5
User-agent: teleport
User-agent: NPBot
User-agent: TurnitinBot
User-agent: dloader(NaverRobot)
User-agent: dloader(Speedy Spider)
User-agent: FunWebProducts
User-agent: WebStripper
User-agent: WebSauger
User-agent: WebCopier

to the top

»» Robots Resources and Tools

• Resources
• The Robots Exclusion Protocol
• Web Robots FAQ
• Using Apache To Stop Bad Robots
• List of Robots
• Database of Web Robots
• Types and Details of Robots
• Articles and Papers

• Tools
• Robots.txt Generator
• Robots.txt File Checker
• SEOChat.com Robots.txt Generator
• Robots Text File Manager

»» About the Author

Sean Odom runs a Houston, Texas based SEO Consulting firm specializing in search
engine marketing and Social Media Optimization.