robotstxt.org - The Web Robots Pages

Ever wondered how search engines work? Or who looks for all those results and brings them together so you can search with ease so quickly? Well, that job is done by web robots, programs that traverse the web automatically also known as: Web Wanderers, Crawlers, or Spiders. If you care to get to know more about them, you should link to robotstxt.

org, were you’ll have access to information and to a group of websites listings containing information related to web crawlers. If you simply want to get general knowledge of what a web spider is, how it works and others, you should check the sites “FAQ’s” which provides simple definitions and answers to all those frequently asked questions about web robots, from web users, web authors, and robot implementers. But that’s not it as you can get interesting information such as how you can direct robots that visit your web site, or check out the database of currently known robots, with description and contact details, or even access to articles and papers about WWW robots, or related sites.