mehroosali/bigquery-sparksql-batch-etl

Stars
2
Language
Python
Created almost 3 years ago
Updated almost 3 years ago

mehroosali/bigquery-sparksql-batch-etl

mehroosali

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Batch ETL pipeline project on GCP to load and transform daily flight data using Spark to update tables in BigQuery. The pipeline is automated using Airflow.

databricks-F1-Project

A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.

Jupyter Notebook

s3-redshift-batch-etl-pipeline

Built functional python ETL script with functions that initialized spark clusters using pyspark library to extract songs stored in S3 bucket. Partitioned songs data by year and artist_id and compressed in parquet output files to increase load performance. Used the overwrite mode in spark to ensure every new run of ELT script is overwritten in the data lake to avoid duplicates. Orchestrated ELT data pipeline that extracts from S3, loads in redshift for transformation and loads output back to S3. Used hooks in airflow to make connection credentials configurable in order to separate access rights from code base for security. Used operators to execute loading and transformation scripts for redshift with airflow DAG.

Ebay-DB-design

Ebay database design project for the class CS6360 (Database Design) at the University of Texas at Dallas.

Twitter-Sentiment-Analysis

personal project to pull live Twitter data using Nifi getTwitter processor and pushes to Kafka topic which is then consumed by a Spark Streaming application where basic sentiment analysis is performed and the final result is stored in elastic search for visualization using Kibana.

Information-Retrieval-Search-Engine

Search Engine project for Information Retrieval class.

tash-shell

tash-shell project using C for class CS5348 (Operating Systems) at the University of Texas at Dallas.

assembly_file_statistics

Assembly project to compute file statistics using MIPS for class CS5330 (Computer Architecture) at the University of Texas at Dallas.