Discover tmalaska/CleanUpEmptyFilesTool Open Source project

Stars
3
Rank 3,963,521 (Top 79 %)
Language
Scala
License
Apache License 2.0
Created over 8 years ago
Updated over 8 years ago

tmalaska

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

This tool is designed to look through your HDFS folders to ether identify files with no data in them or delete files with no data in them.

SparkOnKudu

Based off the design of SparkOnHBase. This Repo will support Spark, Spark Streaming, and Spark SQL integration with Kudu.

Scala

SparkStreaming.Sessionization

NRT Sessionization with Spark Streaming landing on HDFS and putting live stats in HBase

Scala

SparkUnitTestingExamples

This project is a collection of Spark Unit Tests Examples to help new Spark users have good examples on how to unit start their code for Spark Core, Spark SQL, and Spark Streaming

Scala

Spark.TableStatsExample

Simple Spark example of generating table stats for use of data quality checks

Scala

HBase-ToHDFS

Reads a HBase table and writes the out as Text, Seq, Avro, or Parquet

Java

SparkOnHBase

Scala

SparkOnALog

Examples of Integrating Spark Streaming, Flume, and HBase to solve Streaming problems

Java

CopybookInputFormat

Using JRecord to build a mapred and mapreduce inputformat for HDFS, MAPREDUCE, PIG, HIVE, Spark, ...

Java

HBase.MCC

HBase.MCC (HBase Multi Cluster Client). The goal is to support aways up solutions with HBase through multiple clusters

Java

Hive.Generate.DDL

Generation tool that generates DDLs and simple data load scripts.

Java

hadcom.utils

Advanced common functionality for hadoop

Java

Taxi360

Simple Example of HBase, SolR, and Kudu for Entity 360 using NY taxi data

Scala

teraSort-Compressed

The rules of tera sort say you can't compress the input and output. Well those rules are out of touch with how real use cases on hadoop.

Java

FileIngestor

A simple program to put files from a directory into HDFS with the added functionality and defining how that action will happen

Java

Spark..Unique.Seq.Generator

This is an example of how to make Unique Sequences in a distributed way with Spark (No dups, No Skips)

Java

Spark.GraphX.Examples

Just some example of using GraphX

Scala

Flume.NettyAvroAsyncRpcClient

This is a layer on top of the Flume NettyAvroRpcClient that allows for multiple connects to a server.

Java

MRSmallFileCombiner

Tool to read many small files in HDFS with MR while control allowing the caller to define the number of mappers.

Java

Spark.ProdictBehaviorBasedOnPastActives

This is an example of how to do window analysis with Spark

Scala

HBase.GetTopNRecords

This is a simple example to show how a single HBase "get" can retrieve the top N {items,amount} in the order of amount decresing

Java

HBaseMassiveBulkLoadUtils

This is a tool for testing and managing many repeatedly and large bulk loads on HBase

Java

AppTrans

Examples for training

Scala

EdgeNodeGraphUi

Connecting the power of the D3 graphing library to CDH (HDFS, HBase and Impala)

Java

HBase-FastTableCopy

This will contain implementations that will copy records from a table with less regions then the final table.

Java

spark.mergesort.example

An example of how to do a merge sort

Scala

FairSchedulerPlus

A upgrade Extended FairScheduler that takes Sub-Groups into account.

Java

MapReduce.Unique.Seq.Generator

This is a single map reduce job that will append a unique sequence number to the front of every row in a source file.

Java

FixedLengthInputFormat

This is a FixedLengthInputFormat for Hadoop map reduce.

Java

SparkStreamingSeqSink

Support to write Seq Files with Spark Streaming with similar functionality as Flume HDFS Sink with Seq Files

Scala

IngestProcessStoreInNRT

This is a demo/training application. Used to show how easy it is to do operations like ingestion, aggregation, and change data capture. Using tools like Kafka, Spark Streaming, Flume, Kudu, SolR, HBase, and HDFS

Scala

tmalaska/CleanUpEmptyFilesTool

tmalaska

Reviews

Repository Details

More Repositories