|
A piece of software, such as a browser or spider, that interprets the content on a web server and presents it to the user as a web page. Examples include Internet Explorer, Opera, Netscape and various search engine spiders.
Examples: MS Internet Explorer, Netscape Navigator, Googlebot, Slurp, T-Rex
Also called bots or spiders; programs that follows links to visit web sites on behalf of search engines. Crawlers then process and index the code and content of a web page according to an algorithm and store the pages in the search engine's database. Googlebot is the crawler that travels the web finding and indexing pages for the Google search engine.
Googlebot is the Google's spider or crawler or bot; it travels the web finding and indexing pages for the Google search engine. Googlebot leaves it's identity on your web server's log file. Webmasters should study their server log files closely to see the googlebot's visit is successful and is able to crawl the whole site.
| 
