Many so called webbots or web spiders are currently used for many different things on the Internet. Examples include search engines that use them to catalog the Internet, email marketing people that search for email addresses and many more. For a description of such robots check out The Web Robots Faq.
Table of contents
* Trapped and/or Banned Robots
* Definition of a bad robot
* How to identify bad robots
* Banning bad robots from the site
* The Robot Trap
* Download the traps
