Crawler
A program search engines use to crawl the web. Bots visit webpages to collect information and add or update a search engine’s index.
A program search engines use to crawl the web. Bots visit webpages to collect information and add or update a search engine’s index.