Stock Market Kafka Real Time Data Engineering Project
Introduction
In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.
We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.
Architecture
Technology Used
- Programming Language - Python
- Amazon Web Service (AWS)
- S3 (Simple Storage Service)
- Athena
- Glue Crawler
- Glue Catalog
- EC2
- Apache Kafka
Dataset Used
You can use any dataset, we are mainly interested in operation side of Data Engineering (building data pipeline)
Here is the dataset used in the video - https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/indexProcessed.csv
Complete Video Tutorial
Video Link - https://www.youtube.com/embed/KerNf0NANMo