Research?

Teaching

Downloads

Classes

Publications

arXiv:1212.6177

Summary

Ainsworth, Scott G., Alsum, Ahmed, SalahEldeen, Hany, Weigle, Michele C. and Nelson, Michael L., "How much of the web is archived? (Long version of JCDL'11 paper)," arXiv, December 2012.

Abstract

Although the Internet Archive's Wayback Machine is the largest and most well-known web archive, there have been a number of public web archives that have emerged in the last several years. With varying resources, audiences and collection development policies, these archives have varying levels of overlap with each other. While individual archives can be measured in terms of number of URIs, number of copies per URI, and intersection with other archives, to date there has been no answer to the question "How much of the Web is archived?" We study the question by approximating the Web using sample URIs from DMOZ, Delicious, Bitly, and search engine indexes; and, counting the number of copies of the sample URIs exist in various public web archives. Each sample set provides its own bias. The results from our sample sets indicate that range from 35%-90% of the Web has at least one archived copy, 17%-49% has between 2-5 copies, 1%-8% has 6-10 copies, and 8%-63% has more than 10 copies in public web archives. The number of URI copies varies as a function of time, but no more than 31.3% of URIs are archived more than once per month.

Bibtex entry

@ARTICLE { arXiv:1212.6177,
    ABSTRACT = { Although the Internet Archive's Wayback Machine is the largest and most well-known web archive, there have been a number of public web archives that have emerged in the last several years. With varying resources, audiences and collection development policies, these archives have varying levels of overlap with each other. While individual archives can be measured in terms of number of URIs, number of copies per URI, and intersection with other archives, to date there has been no answer to the question "How much of the Web is archived?" We study the question by approximating the Web using sample URIs from DMOZ, Delicious, Bitly, and search engine indexes; and, counting the number of copies of the sample URIs exist in various public web archives. Each sample set provides its own bias. The results from our sample sets indicate that range from 35%-90% of the Web has at least one archived copy, 17%-49% has between 2-5 copies, 1%-8% has 6-10 copies, and 8%-63% has more than 10 copies in public web archives. The number of URI copies varies as a function of time, but no more than 31.3% of URIs are archived more than once per month. },
    ARCHIVEPREFIX = { arXiv },
    ARXIVID = { 1212.6177 },
    AUTHOR = { Ainsworth, Scott G. and Alsum, Ahmed and SalahEldeen, Hany and Weigle, Michele C. and Nelson, Michael L. },
    BOOKTITLE = { Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL'11) },
    JOURNAL = { arXiv },
    KEYWORDS = { Digital Libraries,Information Retrieval },
    EPRINT = { 1212.6177 },
    MONTH = { dec },
    TITLE = { How much of the web is archived? (Long version of JCDL'11 paper) },
    YEAR = { 2012 },
    PUBDATE = { 201212 },
    PDF = { ainsworth-jcdl11-arxiv.pdf },
    SLIDES = { http://www.slideshare.net/aalsum/jcdl-howmuchisarchived },
    URL = { http://arxiv.org/abs/1212.6177 },
}

History Print Recent Changes Search

Page last modified on March 01, 2014, at 11:00 AM