Data-mining-the-water-table

We have taken the data from Driven Data ‘Pump it up: Data Mining the Water Table’ challenge where we have to predict status_group which is functional, non-functional and functional needs repair. The dataset has features such as the location of the pump, water quality, source type, extraction technique used, and population demographics of pump location. The training set has 59,401 rows and 40 features including an output column. Out of the 40 features in the data, we have 31 categorical variables, 7 numerical variables, and 2 date variables.

Predicting-click-through-rate-of-ads

Taken the dataset from KDD 2012 cup which is of around 10gb. We have placed the training file on hadoop cluster. Initially, we have used pig to transform the data. Later, we have used Spark MLlib for dimensionality reduction and model building. Finally, evaluated the models using some evaluation metrics.