LinkedInAttic/datafu

Stars
585
Rank 76,419 (Top 2 %)
Language
Java
Created about 13 years ago
Updated over 10 years ago

LinkedInAttic/datafu

LinkedInAttic

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Hadoop library for large-scale data processing, now an Apache Incubator project

Apache DataFu

Follow @apachedatafu

Apache DataFu is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.

It consists of two libraries:

Apache DataFu Pig: a collection of user-defined functions for Apache Pig
Apache DataFu Hourglass: an incremental processing framework for Apache Hadoop in MapReduce

DataFu is currently undergoing incubation with Apache. A mirror of the official git repository can be found on GitHub at https://github.com/apache/incubator-datafu.

For more information please visit the website:

http://datafu.incubator.apache.org/

If you'd like to jump in and get started, check out the corresponding guides for each library:

Blog Posts

Presentations

Videos

Introduction to Apache DataFu @ ApacheCon 2014

Other Resources

An interesting example of using Quantile from DataFu can be found in the Hadoop Real-World Solutions Cookbook.

From Around the Web

Papers

Hourglass: a Library for Incremental Processing on Hadoop (IEEE BigData 2013)

Getting Help

Please visit the website:

http://datafu.incubator.apache.org/

hopscotch

A framework to make it easy for developers to add product tours to their pages.

LayoutKit

LayoutKit is a fast view layout library for iOS, macOS, and tvOS.

camus

LinkedIn's previous generation Kafka to HDFS pipeline.

indextank-engine

Indexing engine for IndexTank

LIExposeController

Expose style navigation for iOS apps

Selene

iOS library which schedules the execution of tasks on a background fetch

cleo

A flexible, partial, out-of-order and real-time typeahead search library

sensei

distributed realtime searchable database

inject

AMD and CJS dependency management in the browser

indextank-service

The API, BackOffice, Storefront, and Nebulizer for IndexTank

venus.js

where bugs go to die

Fiber

Lightweight JavaScript prototypal inheritance model

sepia

Sepia is a VCR-like module for node.js that records HTTP interactions, then plays them back exactly like the first time they were invoked

JTune

A high precision Java CMS optimizer

scanns

A scalable nearest neighbor search library in Apache Spark

Cubert

Fast and efficient batch computation engine for complex analysis and reporting of massive datasets on Hadoop

naarad

Naarad is a highly configurable system analysis tool that parses and plots timeseries data for better visual correlation. Naarad was built to help in performance analysis and investigations.

simoorg

Failure inducer framework

white-elephant

Hadoop log aggregator and dashboard

nginx-config-builder

A python library for building nginx configuration files programatically

Zopkio

A Functional and Performance Test Framework for Distributed Systems

fossor

A plugin-oriented tool for automating the investigation of broken hosts and services.

api-get-started

LinkedIn REST API Getting Started Tutorial

dustjs-helpers

Helpers for dustjs-linkedin

archetype

Archetype is a Compass/Sass based framework for authoring configurable, composable UI components and patterns.

Isaac

This library parses data from JSON objects into NSObject models without needing to write parsing code for each model.

linkedin-utils

Base utilities shared by all linkedin open source projects

lafayette

Lafayette is a system to store various email abuse reports sent in ARF.

rest.li-api-hub

API Hub is a web UI for browsing and searching a catalog of Rest.li APIs.

jaqen

Jaqen - Simple DNS rebinding

Backbone.TableView

Backbone View to render collections as tables

linkedin-zookeeper

This project provides utilities and wrappers around ZooKeeper

sometime

A BurpSuite plugin to detect Same Origin Method Execution vulnerabilities

RookBoom

A web application for creating meetings.

datacl

A collection of efficient utilities for a data scientist.

mobster

Mobster is a tool that can help you get deeper understanding into the performance of mobile web applications on real mobile devices

vagrant-autodns

Vagrant plugin for automagically managing guest DNS

dmarc-msys

This set of scripts in Lua implements DMARC policy checking and reporting for the Message Systems MTA products, a popular extendable commercial MTA.

talkin

TalkIn is an interface providing safe and easy unidirectional cross-document communication.

play-testng-plugin

TestNG runner for the Play Framework 2.4

sin

insframe

Central hub for distributing web apps to multiple browsers on multiple environments

Tachyon-iOS

Tachyon provides configurable UI components for iOS that are commonly used in calendar features and applications.

postcss-lang-optimizer

PostCSS plugin to extract language specific CSS rulesets to their own CSS files to optimize stylesheet delivery.

bowser

Extensible language parser with Python-like syntax. Written in Java and antlr.

adfullssl

AdFullSsl is a tool that can automatically detect SSL non-compliant ads and fix them

dustjs-filters-secure

extend dustjs-linkedin to enhance the filters methods

gradle-plugin-insight

Automatic, effortless, accurate documentation for any Gradle plugin

timingz.js

Measure code execution in the browser and derive statistical data

Idiomatic-JSLint

streaming

custom-gradle-plugin-portal

An example implementation of a gradle plugin portal.

sbt-restli

A collection of sbt plugins providing build integration for the rest.li REST framework

PTYHooks

MTBT

inject-bower

Please use linkedin/inject

rest.li-skeleton.g8

Rest.li tool for generating skeleton rest.li projects.

naarad-examples

Example logs and configs for naarad

html5-presentation

Code for the "Building a Performant HTML5 App" presentation at http://www.meetup.com/SF-Web-Performance-Group/events/71651452/