trevransom/sparkify-airflow-etl

Stars
1
Language
Python
Created almost 4 years ago
Updated over 3 years ago

trevransom/sparkify-airflow-etl

trevransom

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

A high grade Airflow data pipeline that is dynamic and built from reusable tasks, can be monitored, and allows easy backfills.

bees

Created a Plotly chart for information on bee colonies dying.

de_project_1

Setting up an ETL pipeline

spark_aws_data_lake

Spark and AWS Data Lake with ETL pipeline

wait-for-it

Open Source project for CSCI 450

imdb_analysis

Developed a script to address the question "Does having a movie available in more translations increase its average rating?"

aws_sparkify_etl

AWS Redshift data warehouse and S3 solution to provide an analytical database for Sparkify from JSON logs

spotify_etl

Setting up a Spotify ETL process to save recently played tracks into an SQLite database

cassandra_etl

Modeled data by creating tables in Apache Cassandra to run queries. Configured ETL pipeline that transfers data from a set of CSV files within a directory to create a streamlined CSV file to model and insert data into Apache Cassandra tables.

Jupyter Notebook

sparkify_etl

Sparkify, a music streaming startup, wanted to collect logs they have on user activity and song data and centralize them in a database in order to run analytics. This Postgres database, set up with a star schema, will help them to easily access their data in an intuitive fashion and start getting rich insights into their user base.

Jupyter Notebook