chayansraj/Data-Science-Apache-Spark-Databricks-ETL-Project chayansraj/Data-Science-Apache-Spark-Databricks-ETL-Project

Twitter logo Share LinkedIn logo Share Facebook logo Share
chayansraj/Data-Science-Apache-Spark-Databricks-ETL-Project
github

chayansraj

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Using Databricks community edition to build multiple end-to-end ETL pipelines using PySpark for different file formats such as CSV, Parquet, Delta table. Predictive modeling is performed using different machine learning algorithms.

More Repositories

1

Data-Pipeline-with-dbt-using-Airflow-on-GCP

This project demonstrates how to build and automate an ETL pipeline using DAGs in Airflow and load the transformed data to Bigquery. There are different tools that have been used in this project such as Astro, DBT, GCP, Airflow, Metabase.
Python
18
star
2

Python-ETL-pipeline-using-Airflow-on-AWS

This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.
Python
13
star
3

Youtube-video-data-analytics-using-AWS

This project aims to leverage Amazon Web Services to create trending Youtube videos analytics service. Project contains different data engineering, data analysis and data science parts.
Python
7
star
4

LINHAC-2022-Data-Science-Student-Competition

Linkรถping Hockey Analytics Conference - LINHAC 2022 | Given the event data, generate findings/patterns related to sequences of events leading up to a particular outcome.
Jupyter Notebook
7
star
5

Microsoft-Azure-Medallion-Data-pipeline

In this project we are going to create an end-to-end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
Jupyter Notebook
4
star
6

End-to-end-machine-learning

The idea of this project is to apply statistical methods learned in university lectures to find patterns in the data and use machine learning to solve a supervised classification problem
Jupyter Notebook
3
star
7

Snowflake-data-ingestion-hands-on-tutorial

This repo covers the two most widely used and recommended file based data ingestion approaches: COPY INTO and Snowpipe.
2
star
8

Vision-Transformer-Research

The purpose of this research project is to compare traditional CNNs to vision transformers, can transformers give a higher AUC when classifying Atypical Femoral Fracture / Normal Femoral Fracture?
Jupyter Notebook
2
star
9

Neural-Networks-and-Learning-Systems

Solving problems using different machine learning algorithms. Machine learning, classification, pattern recognition and high-dimensional data analysis.
Jupyter Notebook
2
star
10

Advanced-Regression-Techniques-for-Ames-housing-data-prediction

Prediction of Ames house prices using advanced regression techniques and ML algorithms.
R
2
star
11

GPT-and-LangChain-for-Data-analysis

Jupyter Notebook
1
star
12

Prediction-of-used-car-prices-using-various-regression-techniques

It is a work on a regression problem in which our objective is to predict the prices of used cars given a number of features/predictors about them
Jupyter Notebook
1
star
13

dbt_learn_fundamentals

dbt is a SQL-first transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation.
1
star