The Invisible Web A presentation at Computers in Libraries 2000, March 2000
Karen Hartman Ernest Ackermann  


Why is part of the Web Invisible?

Recent estimates support the notion that a relatively small fraction of the information available on the Web is accessible through search engines. Furthermore, many search engines return irrelevant results.

Databases typically disclose their contents only in response to a query. For databases accessible through the Web this means that their contents are ephemeral parts of the Web only being disclosed through a query. This makes it virtually impossible for search engines to index their contents.

Researchers Steve Lawrence and Lee Giles have shown that search engines are not indexing even half of the information available on the Web. Their latest research is published in Nature, Vol. 400, July 1999, pp. 107-109. They summarize the results of their research in Accessibility and Distribution of Information on the Web,

Getting at the Invisible Web

While search engines are useful, there are a number of other types of resources that need to be considered when using the Internet for research.

These make up what some have called the invisible Web or the hidden Internet. The primary resources of the invisible Web are called specialized databases. Virtual libraries, meta-search tools that include lists of databases, and directories are often the best sources to use when looking for specialized databases.


URL and Description

Beaucoup lists over 2,500 specialized databases and directories and also serves as a meta-search tool.
Internet Public Library (IPL) Reference Center
The IPL is a virtual library that provides a good starting point for finding reference works, subject guides, and specialized databases.
This meta-search tool contains links to hundreds of specialized databases.
The Invisible Web
Produced by Intelliseek, Inc., the Invisible Web is a well-organized, comprehensive directory to thousands of specialized databases
LibrarySpot collects links to quality reference resources and provides links to more than 2,500 libraries around the world.
Librarians' Index to the Internet
A virtual library that is both searchable and browsable, this is an excellent source for specialized databases.
The Scout Report
The Scout Report is a good way to keep up with new search tools, especially specialized databases. You can view its weekly report and its archive of previous Scout Reports on the Web. You can also have the report delivered to you via email by subscribing through a listserv. Send an email message to Type subscribe to scout-report in the body of the message.

Karen Hartman
Ernest Ackermann
Presented at Computers in Libraries 2000 March 2000