Blog

Using a Robots Text File

Search engines will search for a special file called robots.txt before spidering your site. The Robots Text File is created specifically to give directions to web crawlers/robots. Place the following two lines in your robots.txt file if you wish to allow search engines to crawl/spider everything on your site:

The * in the first line specifies that the directions are for all search engines. The second line indicates that nothing is disallowed.

Once you have created your robots.txt file, upload it to your websites main directory where your homepage and other HTML files are located.

Robots.txt Usage Examples

Below are several common examples of how you can use a robots.txt file to set parameters and control how different crawlers/robots access your website.

The following example would allow all crawlers/robots to access all files except for your images file.

The following would give Google a direct link to your XML sitemap (more info here), as well as other search engiens, and direct crawlers/robots to crawl all files except the cgi-bin files, images directory, and your log files.

Using “Crawl-delay” parameters in the robots.txt file. This parameter indicates the number of seconds for a crawler/spider to delay between requests.

Bad Robots and Email Harvestors

Below are several robots/crawlers that you might want to block.

Robots Resources and Tools

• The Robots Exclusion Protocol
• Web Robots FAQ
• Using Apache To Stop Bad Robots
• List of Robots
• Database of Web Robots
• Types and Details of Robots
• Articles and Papers

Tools

• Robots.txt Generator
• Robots.txt File Checker
• Internet Marketing Ninjas Robots.txt Generator
• Robots Text File Manager

About The Author:

Sean Odom is the founder of SeOpt Internet Marketing, a Houston based local SEO service provider. Sean is also co-founder of Socialot.com, a SAAS social contact management system for small businesses.