Ever wondered how search engines work? Or who looks for all those results and brings them together so you can search with ease so quickly? Well, that job is done by web robots, programs that traverse the web automatically also known as: Web Wanderers, Crawlers, or Spiders. If you care to get to know more about them, you should link to robotstxt.
org, were you’ll have access to information and to a group of websites listings containing information related to web crawlers. If you simply want to get general knowledge of what a web spider is, how it works and others, you should check the sites “FAQ’s” which provides simple definitions and answers to all those frequently asked questions about web robots, from web users, web authors, and robot implementers. But that’s not it as you can get interesting information such as how you can direct robots that visit your web site, or check out the database of currently known robots, with description and contact details, or even access to articles and papers about WWW robots, or related sites.
robotstxt.org In Their Own Words
“Welcome to www.robotstxt.org, the new home of The Web Robots Pages.
This is the main source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots. “
Why robotstxt.org It Might Be A Killer
The site provides interesting information of web crawlers in a friendly way in order for anyone to understand the sites contents, there’s no need of being an internet expert.
Some Questions About robotstxt.org
The site lacks a lot of look and feel, will this change in order to make users more comfortable and stay longer? What’s the use of having a “Robot Mailing List” if its services are currently out of order? 
