helgeho/internetarchive-transfer-scripts

Stars
9
Rank 1,939,727 (Top 39 %)
Language
Python
License
GNU Affero Genera...
Created almost 9 years ago
Updated over 6 years ago

helgeho/internetarchive-transfer-scripts

helgeho

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Scripts to transfer archive.org collections, using https://github.com/jjjake/internetarchive

ArchiveSpark

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

Web2Warc

An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)

HadoopConcatGz

A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz

HadoopWebGraph

A Hadoop input format to use gaphs in WebGraph's BV format with Hadoop and Spark.

Exspec

Don't write specs anymore, just save 'em while testing your code interactively. Specs will become a byproduct.

IABooksOnArchiveSpark

Analyze digitized books from the Internet Archive remotely with ArchiveSpark

Micrawler

Create and cite micro Web archives with semantics as temporal representations of objects / entities / concepts on the Web

ArchiveSpark-server

A server application that provides a Web service API for ArchiveSpark to be used by third-party applications to integrate temporal Web archive data with a flexible, easy-to-use interface.

FEL4ArchiveSpark

Yahoo's Fast Entity Linker for ArchiveSpark

MHLonArchiveSpark

Work with Medical Heritage Library collections using ArchiveSpark

ArchiveSpark-Zeppelin-Docker

ArchiveSpark with Zeppelin as ready-to-use Docker image

ArchiveSpark-AUT-bridge

The compatibility layer between ArchiveSpark and The Archives Unleashed Toolkit (AUT)

WarcPartitioner

Partition (W)ARC Files by MIME Type and Year

ArchiveSpark-docker

ArchiveSpark on Docker

ArchiveSpark2Triples

Convert web archives to RDF triples with ArchiveSpark

Jupyter Notebook

MapReduceLecture

A lecture on MapReduce with example code

ArchivePig

An Apache Pig framework that facilitates access to Web Archives, enables easy data extraction as well as derivation.