• Stars
    star
    149
  • Rank 248,619 (Top 5 %)
  • Language
    Java
  • License
    Other
  • Created almost 12 years ago
  • Updated over 8 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An implementation of a real-world map-reduce workflow in each major framework.

Realistic Hadoop Data Processing Examples

This code is to accompany my blog post on map reduce frameworks

The point of the code in this repository is to provide an implementation for a business question (listed below) in each of the major Map Reduce frameworks.

Each implementation will get it's own subdirectory with it's own build and running instructions. Each framework will also get an accompanying test, and an in-depth walkthrough about implementation details.

The following implementations are complete:

The problem

The Data

We have two datasets: customers, and transactions.

Customer Fields:

Transaction Fields:

  • transaction-id (1)
  • product-id (1)
  • user-id (1)
  • purchase-amount (19.99)
  • product-description (a rubber chicken)

These two datasets are stored in tab-delimited files somewhere on HDFS.

The Question

For each product, we want to know the number of locations in which that product was purchased.

That's it!

In the real world, we might have other questions, like the number of purchases per location for each product.