User Agent Category: Crawlers
Different types of crawlers, search engine robots, spiders, e-mail harvesters/collectors, link checkers/validators, sitemap builders, download managers or other kind of automated bots, that traverse website pages for different purposes.
| Category | Description |
|---|---|
| Download Manager | Download managers are either standalone applications or browser plugins that transfer locally different elements from the web, such as full HTML pages, images, multimedia etc. |
| E-Mail Collector | E-Mail collectors or harvesters crawl web pages looking for and collecting e-mail addresses. |
| Spider/Bot | Any type of robot that does not fit in other categories from the Crawlers group. This include unknown and undocumented visitors that act like spiders, traversing a large number of pages in a small period of time. |
| Link Checker | Link Checkers traverse hyperlinks and check for broken links or missing pages. Broken links are URLs you have to external unavailable pages in your pages. Missing pages are links you may have to other pages from your own site, which are no longer accessible. Most Link Validators will not visit and crawl your site, unless you specifically requested their services. Link Exchange crawlers make sure reciprocal links are present on the web site. |
| Search Robot | Search Engine crawlers collect text-based content data or multimedia data (pointers to images, videos or audio files) for indexing purposes. Unlike those other spiders you don't know why they collect your data and what they do with it - and which can be found in the generic category of Spiders - search engine crawlers are in most cases good for you, because they help people find your website pages and bring you more visitors. You can also go to their search engine main pages and look for the data from your site that they indexed. |
| Sitemap | Sitemap webmaster tools are either online services, that recursively parse your site to build a sitemap file for search engines, as defined by sitemap.org, or search engine crawlers that read your XML sitemap file to determine which pages you want indexed. Most sitemap online services will act only on your specific request and will crawl your site only when you requested them to build a sitemap file. |
