|
A small program, often written in Java, which usually runs in a web browser, as part of a web page. It is possible that the use of such a program may cause spiders and robots to stop indexing a page.
Abbreviation for robot (also called a spider). It refers to software programs that scan the web. Bots vary in purpose from indexing web pages for search engines to harvesting e-mail addresses for spammers.
Pages that are included in a search engine's index. An important step in search engine optimisation in insuring your web sites pages are included, indexed, in the search engine databases. Search engines index pages through the spiders or robots.
Robots.txt is a file on a web sites root directory which spiders are supposed to read to determine which parts of a website they may or may not visit.
Also known as spider, bot or crawler. The part of a search engine that locates and indexes every page on the Web. Successful search engine optimisation depends on robots finding many or all a Web site's pages.
| 
