Why is part of the Web Invisible?

Recent estimates support the notion that a relatively small fraction of the information available on the Web is accessible through search engines. Furthermore, many search engines return irrelevant results.

Databases typically disclose their contents only in response to a query. For databases accessible through the Web this means that their contents are ephemeral parts of the Web only being disclosed through a query. This makes it virtually impossible for search engines to index their contents.

Researchers Steve Lawrence and Lee Giles have shown that search engines are not indexing even half of the information available on the Web. Their latest research is published in Nature, Vol. 400, July 1999, pp. 107-109. They summarize the results of their research in Accessibility and Distribution of Information on the Web, http://www.wwwmetrics.com.

Getting at the Invisible Web

While search engines are useful, there are a number of other types of resources that need to be considered when using the Internet for research.

These make up what some have called the invisible Web or the hidden Internet. The primary resources of the invisible Web are called specialized databases. Virtual libraries, meta-search tools that include lists of databases, and directories are often the best sources to use when looking for specialized databases.