helgeho/HadoopWebGraph helgeho/HadoopWebGraph

  • Stars
    star
    7
  • Rank 2,294,772 (Top 46 %)
  • Language
    Java
  • License
    GNU General Publi...
  • Created over 7 years ago
  • Updated over 7 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
helgeho/HadoopWebGraph
github

helgeho

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A Hadoop input format to use gaphs in WebGraph's BV format with Hadoop and Spark.

More Repositories

1

ArchiveSpark

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
Scala
143
star
2

Web2Warc

An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
Scala
24
star
3

HadoopConcatGz

A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Java
9
star
4

internetarchive-transfer-scripts

Scripts to transfer archive.org collections, using https://github.com/jjjake/internetarchive
Python
9
star
5

Exspec

Don't write specs anymore, just save 'em while testing your code interactively. Specs will become a byproduct.
Ruby
5
star
6

IABooksOnArchiveSpark

Analyze digitized books from the Internet Archive remotely with ArchiveSpark
Scala
5
star
7

Micrawler

Create and cite micro Web archives with semantics as temporal representations of objects / entities / concepts on the Web
JavaScript
4
star
8

ArchiveSpark-server

A server application that provides a Web service API for ArchiveSpark to be used by third-party applications to integrate temporal Web archive data with a flexible, easy-to-use interface.
Scala
3
star
9

FEL4ArchiveSpark

Yahoo's Fast Entity Linker for ArchiveSpark
Scala
3
star
10

MHLonArchiveSpark

Work with Medical Heritage Library collections using ArchiveSpark
Scala
2
star
11

ArchiveSpark-Zeppelin-Docker

ArchiveSpark with Zeppelin as ready-to-use Docker image
Shell
2
star
12

ArchiveSpark-AUT-bridge

The compatibility layer between ArchiveSpark and The Archives Unleashed Toolkit (AUT)
Scala
2
star
13

WarcPartitioner

Partition (W)ARC Files by MIME Type and Year
Java
1
star
14

ArchiveSpark-docker

ArchiveSpark on Docker
Python
1
star
15

ArchiveSpark2Triples

Convert web archives to RDF triples with ArchiveSpark
Jupyter Notebook
1
star
16

MapReduceLecture

A lecture on MapReduce with example code
Java
1
star
17

ArchivePig

An Apache Pig framework that facilitates access to Web Archives, enables easy data extraction as well as derivation.
Java
1
star