User Agent Category: Search Robot
Search Engine crawlers collect text-based content data or multimedia data (pointers to images, videos or audio files) for indexing purposes. Unlike those other spiders you don't know why they collect your data and what they do with it - and which can be found in the generic category of Spiders - search engine crawlers are in most cases good for you, because they help people find your website pages and bring you more visitors. You can also go to their search engine main pages and look for the data from your site that they indexed.
| Agent | Description |
|---|---|
| AbachoBot | Search engine spider, with localized versions for UK, Germany, France, Italy, Spain, Sweden, Turkey, Switzerland, Austria. Crawler.de robot. |
| Accoona | Accoona's web crawler. |
| ADSA | |
| Agadine | |
| aipBot | Nameprotect copyright search robot. |
| Aleksika | Danmark search engine optimization spider. |
| AltaVista/Scooter/Mercator | Scooter is AltaVista's prime index agent. Mercator is another AltaVista crawler. |
| Amazon | Amazon.com robot for checking affiliate sites. |
| AmfibiBot | Amfibi search robot. |
| amiBot | Amidalla search engine robot. |
| AnsearchBot | |
| AnswerBus | Natural language search using COLLATE technology. |
| AnswerChase | Search tool. |
| AnyApexBot | Crawler for the web directory AnyApex. |
| Apexoo | Search spider. |
| Appie/Walhello | Collect and index web pages for the Walhello search engine. |
| Arikus | inContext search engine software. |
| Ask 24x | DMOZ related robot. |
| askEd | Inferret search robot, using Nutch. |
| AskJeeves | Ask Jeeves/Teoma web indexing robot. |
| ASPSeek | ASPSeek search engine software. |
| Asterias/SingingFish | Media spider. |
| AVSearch | National Research Council Canada or Altavista robot. |
| Axadine | |
| Axmo | Axmo search robot. |
| Baidu | Crawler for the chinese search engine Baidu. |
| BDNcentral | |
| BecomeBot | Shopping related portal and search, with international versions. |
| BigCliqueBOT | BigClique search robot. |
| Biglotron | Search France robot. |
| BigSearch | |
| BilgiBetaBot | Nutch-based search robot for Bilgi.com. |
| BlitzBOT | Localized versions for Germany, Switzerland, Netherlands, Austria. |
| Boitho | Distributed crawler for Boitho. Downloads web pages to build the database used by Boitho.com to search in. To allow volunteers to donate their superfluous bandwidth and idle CPU time, they have developed a distributed crawler, like seti@home and Grub. That way people can install a program on their computers and help them with the crawling. boitho.com-robot is an old version, no longer in use. |
| BTbot | Search engine for bit-torrents, ring-tones for cell phones, friends and extra-terrestrial intelligence. |
| Burf | Burf.com UK Search Engine robot. |
| BurstFind | |
| Catall Spider | |
| Ccubee | Empyreum Ccubee search engine solution. |
| CE-Preload | Cisco Content Engine. |
| CipinetBot | Cipinet search engine crawler. |
| Claymont.com | Claymont Search robot. |
| CNET | |
| Comrite | Nutch-based crawler for search engine, for Oversea web Sites. |
| Content-S-SCEI | Nutch-based search engine crawler. |
| Crawllybot | |
| Cydral | Image and site spider. |
| DatenBot | Security related search. |
| dbDig | Search engine. |
| Ditto | Picture search robot. |
| DoWeb | UK Business directory link checking. |
| Earthcom | |
| Eidetica | Search and text mining spider. |
| Envolk | Search engine Internet Tracking Spider (ITS). |
| EroCrawler | Adult search robot. |
| EuripBot | European Internet Portal robot. |
| EvaalSE | |
| Eventax | Event search. |
| ExaBot/Exalead | Exalead French text/image search robots. Exava's shopping search engine and other shopping search crawlers. |
| ExactSEEK | Jayde Online robots. |
| Execrawl | |
| Factbot | |
| Fast-Search-Engine | |
| Fastbot | |
| FastBug | Geo sync search robot. |
| Favo | |
| Faxobot | FaXo Search robot. |
| FDSE/Fluid Dynamics | Search engine of Fluid Dynamics Software Corporation (FDSE). |
| FindITAnswersbot | Search engine for software developers. |
| FindLinks | NextLinks vocabulary and hyperlink search. |
| Fireball | The Fireball robots gather web documents in German language for the database of the Fireball search service. |
| Flapbot | Flaptor information retrieval solutions robot. |
| Fooky | |
| Forex Trading Network | |
| Francis | |
| Free-Downloads | Shareware directory link checking. |
| FreeFind | Automatic web crawling robot. The spider reads pages from a client website in order to build an index of that website. This website index is used to perform searches for pages which are part of the website. |
| FyberSpider | |
| GaisBot | GAIS (Global Area Information Servers) search engine crawler of the National Chung Cheng University, Taiwan. |
| Galaxy/FusionBot | |
| Gallent Search Spider | The Gallent Search Engine retrieves results that have been saved by the Gallent Search Spider. The Gallent Search Spider uses in-house designed technology that is unique to robot technology. |
| GenieBot/wgao | GenieBot/wgao is a web-indexing robot of GenieKnows Local Search Engine. The GenieBot crawler collects webpages from the web to build an index system to provide service of local search. GenieBot crawler can discover and crawl webpages reachable by following hypertext links. |
| GeonaBot | |
| Gigabot/Gigablast | Gigablast's indexing agents. |
| GoForIt | |
| Goliat | |
| Gonzo | |
| Google Desktop | Desktop search application, to search your computer as easy as searching the web with Google. Provides full text search over your email, files, music, photos, chats, Gmail, web pages that you've viewed, and more. |
| Google-Image | Google's image search engine crawler. |
| Google-Mobile | Google Mobile Search crawler. |
| Googlebot | Google's search engine crawler. For most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to network delays, it's possible that the rate will appear to be slightly higher over short periods. |
| GPU | Distributed search engine crawler. |
| GrigorBot | |
| GurujiBot | Indian search engine robot. |
| Hamahakki | |
| HappyFunBot | |
| Holmes | Search robot or Morfeo/Centrum Search robot. |
| Homerbot | |
| Honda-Search | Honda-Search.com Honda cars related search robot. |
| HooWWWer | Next Generation Information Retrieval robot. |
