How Search Engines
Search engines use software robots to survey the Web and build
their databases. Web documents are retrieved and indexed. When you enter a
query at a search engine website, your input is checked against the search
engine's keyword indices. The best matches are then returned to you as hits.
There are two primary methods of text searching--keyword and
This is the most common form of text search on the Web. Most
search engines do their text query and retrieval using keywords.
Unless the author of the Web document specifies the keywords for
her document (this is possible by using meta
tags in the latest version of HTML), it's up to the search engine to
determine them. Essentially, this means that search engines pull out and index
words that are believed to be significant. Words that are mentioned towards the
top of a document and words that are repeated several times throughout the
document are more likely to be deemed important.
Some sites index every word on every page. Others index only
part of the document. For example, Lycos indexes the title, headings,
subheadings and the hyperlinks to other sites, along with the first 20 lines of
Full-text indexing systems generally pick up every word in the
text except commonly occurring stop words such as "a," "an," "the," "is," "and,"
"or," and "www." AltaVista claims to index all words, even the articles, "a,"
"an," and "the." Some of the search engines discriminate upper case from lower
case; others store all words without reference to capitalization.