A Very Large Scale Library for the History of the Web

William Y. Arms
Professor of Computer Science
Cornell University

As digital libraries become ubiquitous, scholarship is changing. We
can foresee a day when humanities and social science scholars carry
out library research with computer programs that act as their agents.
This talks describes the implementation of the Cornell Web Library,
which has been designed for this new style of social science
research. The library is currently loading some 10 billion Web pages,
about 240 Terabytes. They are drawn from the historical collections
of the Internet Archive. The talk will discuss the tools needed to
carry out research on such a large corpus; it will also discuss the
capabilities and limitations of today's high performance computing
for very large-scale digital libraries.

Web Site:
http://www.infosci.cornell.edu/SIN/WebLib/

William Y. Arms is professor of computer science at Cornell University.
His career includes appointments at the British Open University, Dartmouth
College, and Carnegie Mellon University, where as Vice President for
Computing he led campus-wide networking and distributed computing,
educational computing and libraries. At Cornell, he was the first director
of the Information Science program. He has more than thirty years
experience applying computing to academic activities, notably educational
computing, computer networks, and digital libraries. He was one of the
founders of D-Lib Magazine in 1995 and Editor-in-Chief from 1998-2001. His
book Digital Libraries was published by MIT Press in 2000.

http://www.cs.cornell.edu/wya/