ismaildawoodjee/spark-big-data

Stars
1
Language
Scala
Created over 2 years ago
Updated over 2 years ago

ismaildawoodjee/spark-big-data

ismaildawoodjee

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Spark with Scala. Big data project to analyze 35 GB Parquet data (~400 GB as decompressed CSV) and extract business insights from it

GreatEx

A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.

aws-data-pipeline

A batch processing data pipeline, using AWS resources (S3, EMR, Redshift, EC2, IAM), provisioned via Terraform, and orchestrated from locally hosted Airflow containers. The end product is a Superset dashboard and a Postgres database, hosted on an EC2 instance at this address (powered down):

Forecasting-Intermittent-Demand

Data Mining Cup 2020

Jupyter Notebook

ADExploitation-Obsidian-CheatSheet

Obsidian-friendly cheatsheets for Active Directory enumeration and exploitation

Great-Expectations-for-CSV

Ensuring data quality in an e-commerce data set using Great Expectations.

Jupyter Notebook

Python-Spellchecker-GUI

A purely Pythonic spellchecker with an integrated user-friendly GUI, built from an economics textbook corpus.

Superset-and-Metabase

product-catalogue

Web scraping task to obtain industrial equipment data and produce a product catalogue out of it. From this page:

NocoDB-and-Baserow

nifi-data-pipeline

📦 Containerizing everything from the book "Data Engineering with Python"