User Agent Category: Spider/Bot
Any type of robot that does not fit in other categories from the Crawlers group. This include unknown and undocumented visitors that act like spiders, traversing a large number of pages in a small period of time.
| Agent | Description |
|---|---|
| A-Online | A-Online.at robot, now Jet2Web Search. |
| Abcdatos | Used to verify availability of the ABCdatos directory entries, checking HTTP HEAD. |
| ABCsearch | |
| Aberja | Link checker. |
| AbiLogicBot | Checkes links for the AbiLogic web directory |
| aBot | Nameprotect copyright search robot. |
| About | |
| AboutUsBot | The AboutUs:Bot gathers descriptive information about a website from several sources to build a Wiki Page. This pre-built wiki page gives website owners and AboutUs.org contributors a head-start in creating a useful and informative AboutUs.org page. |
| Ack | Ackerm search robot. |
| Acme.Spider | A Java utility class for writing your own robots. |
| Acoi | Picture finder robot. |
| Acoon | |
| Advanced Email Extractor | E-mail collector. |
| Aesop | |
| AgentName | Linkomatic submission verifier. |
| Ah-ha | |
| Ahoy | Research project at the University of Washington, for finding personal Homepages. |
| AIBOT | Real artificial intelligence search engine China. |
| AITCSRobot | Its purpose is to generate a Resource Discovery database. This Robot traverses the net and creates a searchable database of Web pages. It stores the title string of the HTML document and the absolute url. A search engine provides the boolean AND & OR query models with or without filtering the stop list of words. Feature is kept for the Web page owners to add the url to the searchable database. |
| Aladin | |
| Alcohol Search | |
| Alexa | Alexa crawler. |
| Alkaline | Unix/NT internet/intranet Vestris search engine designed at the University of Geneva. |
| Allesklar | |
| Almaden/IBM Planetwide | Restricted to IBM owned or related domains. Set of research technologies that collect, store and analyze massive amounts of unstructured and semi-structured text. It is built on an open, extensible platform that enables the discovery of trends, patterns and relationships from data. |
| Amgen | |
| AMZNKAssocBot | |
| Ananzi | |
| Annomille | Annomille historical oriented robot. |
| Anonymizer | Faked user agent. |
| Anthill | Gather price information automatically from online stores. Research project at the University of Mannheim. |
| AntiBot/AntiSearch | Discontinued robot. |
| Anzwers | Yahoo robot. |
| Aport | |
| Arachmo | Web site file extraction tool. |
| Arachnophilia | Collect approximately 10k html documents for testing automatic abstract generation. |
| Arale | Java multithreaded web spider. Download entire web sites or specific resources from the web. Render dynamic sites to static pages. |
| Araneo | For crawling and indexing web pages written in the international language Esperanto. |
| AraybOt | Agent software of AraykOO! which crawls web sites listed in dmoz.org/Adult/, in order to build a adult search engine. |
| ArchitextSpider/Excite | Generate a Resource Discovery database and statistics. The ArchitextSpider collects information for the Excite and WebCrawler search engines. |
| Aretha | A crude robot built on top of Netscape and Userland Frontier, a scripting system for Macs |
| Argus | Simpy Bookmarklet crawler. |
| Ariadne | Prototype of an environment for testing focused crawling strategies. |
| Arianna | |
| Arks | The Arks robot is used to build the database for the dpsindia/lawvistas.com search service. The robot runs weekly, and visits sites in a random order. |
| Asahina Antenna | ASAHINA Antenna information detecting agent. |
| Ask.24x.Info | Ask 24x Info robot. |
| AskAboutOil | Petroleum related search, using Nutch. |
| ASpider | ASpider is a CGI script that searches the web for keywords given by the user through a form. |
| AspTear | URL fetching program component, Download32.com spider. |
| ASSORT | Associative sort robot. |
| Astra/AstraFind/AstraSpider | Adult search robot. |
| Atlocal | Local business search robot. |
| ATN Worldwide | The ATN robot is used to build the database for the AllThatNet search service operated by All That Net. The robot runs weekly, and visits sites in a random order. |
| Atomz | Robot used for web site search service. Developed for Atomz.com, launched in 1999. |
| atSpider | Ceased email harvester/spambot. |
| AugurFind | Augurnet search robot. |
| AURESYS | The AURESYS is used to build a personnal database for somebody who search information. The database is structured to be analysed. |
| AutoEmailSpider | Auto Email Pro Email harvester. |
| AV Fetch | |
| Avalon | |
| Aztrx | Adult toys and videos. |
| BaBoom | BaBoom Web Portal (ODP) robot. |
| BackRub | |
| BACS | |
| Badongo | |
| BanBots | Perl script-based robot. |
| BarraHome | Barrahome crawler. |
| Batsch | |
| bBot | Mainly intended for site level search, sometimes set loose. |
| BCentral | BCentral crawler. |
| bdcIndexer | Business.com robot. |
| BDFetch | Brandimensions Brand Protection robot. |
| BeautyBot | Robot for Cosmoty, beauty and wellness search engine. |
| BebopBot | A Passion for Jazz music related search robot. |
| BillBot | Carnegie Mellon School robot, link checker. |
| BimBot | Provides converged data and voice services. |
| Bisnisseek | |
| Bitacle | Blog search archive robot. |
| Bjaaland | Crawls sites listed in the ODP. |
| BlackWidow | Started as a research project, used to find links for a random link generator. Also used to research the growth of specific sites. |
| Blaiz-Bee | Blaiz Enterprises RawGrunt search. |
| Blazer | |
| BlogBot | blogdex robot from MIT.edu. |
| Blogpulse | Blog search. |
| BlogSearch | |
| BlogsNowBot | BlogsNow realtime link tracker robot. |
| BlogWatcher | Robot from Okumura Group Tokyo. |
| BlogzIce | |
| Bloodhound | Bloodhound will download a whole web site depending on the number of links to follow specified by the user. |
| BoardReader | BoardReader search image and favicon fetcher. |
| Boito | |
| Borg-Bot | Developmental crawler to feed a search engine. |
| Bot | |
| BoxSeaBot | Nutch-based crawler. This robot is used to find pages for building the BoxSea search engine indices. The robot code uses Nutch. Earlier experimental crawls were done under various user agent names such as NutchCVS. |
| Brismee | |
| BruinBot | Webarchive Project Bruinbot crawler. |
| BSDSeek | Inktomi Hotbot-Lycos NBCi robot. |
| BSpider | BSpider is crawling inside of Japanese domain for indexing. |
| BuildCMS | Market monitoring project. |
