Warcbase

Warcbase is an open-source platform for managing web archives built on Hadoop and HBase. The platform provides a flexible data model for storing and managing raw content as well as metadata and extracted knowledge. Tight integration with Hadoop provides powerful tools for analytics and data processing via Spark.

Bad news: Warcbase is defunct and no longer under active development!

Good news: In June 2017, the University of Waterloo and York University were awarded a grant from the Andrew W. Mellon Foundation to build the next generation of tools that will make historical internet content accessible to scholars. Warcbase serves as the foundation for the ArchivesUnleashed Toolkit!

If you're interested in reading about the development of Warcbase, check out this article:

Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work has been supported in part by the U.S. National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, the Ontario Ministry of Research and Innovation's Early Researcher Award program, and the Mellon Foundation (via Columbia University). Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.