yennanliu/spark_emr_dev

Stars
3
Rank 3,963,521 (Top 79 %)
Language
Scala
Created almost 5 years ago
Updated over 1 year ago

yennanliu/spark_emr_dev

yennanliu

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Collection of code for submitting Spark/Hadoop/Hive/Pig tasks to EMR (AWS Elastic MapReduce) | #DE

CS_basics

My CS learning : algorithm, data structure, and system design | #SE

NYC_Taxi_Pipeline

Design/Implement stream/batch architecture on NYC taxi data | #DE

DE-100-days

data engineering 100 days 🤖 🧲 🦾 | #DE

NYC_Taxi_Trip_Duration

Develop ML models predict taxi trip duration in NYC. Ranked : Top 6% | RMSLE : 0.377 (Kaggle) | #DS

Jupyter Notebook

knowledge_base_repo

Resources for software/backend/data learning | #SE | #DE | #DS

utility_Python

Collection of Python utility scripts & OOP basic demo | #SE

analysis

Repo for practical data science problems approaches, including notebook demo and working scripts | #DS | #analysis

Jupyter Notebook

spark-etl-pipeline

Various data stream/batch process demo with Apache Scala Spark 🚀

web_scraping

Collect/process data via various data sources : website / js website / API. Run scrapping pipeline via Celery, and Travis cron task. Dump the scraped data to slack

Jupyter Notebook

AirflowJob

Airflow POC demo : 1) env set up 2) airflow DAG 3) Spark/ML pipeline | #DE

spotify_recSys_challenge2018

airflow-ubuntu-dev

deploy airflow to heroku demo

data_infra_repo

Collections of POC/dev data infrastructure. | #SE

mlflow-heroku-dev

deploy mlflow to heroku demo

YelpReviews

Build an end to end data application with Yelp review dataset. (data collect -> DB config -> data ETL -> data dashboard (analysis/ML)

SpringPlayground

Java Spring (Boot/Cloud..) backend playground | #SE

KafkaSparkPoc

Build Kafka <-> Spark, Kafka <-> Kafka .. stream processing POC | #DE

utility_Scala

Scala programming language basic & functional programming (fp) & and other utility cases | #SE

movie_recommendation

Build recommendation algorithms via movie data that can be potentially applied to customized content such as music, and ecommerce

Jupyter Notebook

ScalaPlayground

Scala POCs : API, backend, tools | #SE

FinatraHelloWorld

Build REST API via Scala via Finatra framework

KafkaHelloWorld

Kafka application/infra POC | #SE

spotify_recommend_playlist

Generate suggested song playlists via "Tinder like process". Leverage Spotify recommendation API and ML

dj-playground

Django playground : web app, backend | #SE

utility_shell

Collection of shell/Bash scripts for various using cases | #SE

SGTaxiMap

SG taxi real-time heat map with gov.SG API and python flask, JS leaflet library. Docker/Travis CI integrated

til

Today I Learned

spark-scala-word-count

Run a simple spark word count job via 1) scala sbt spark 2) docker

KKBox_Music_Recommendation

Jupyter Notebook

RabbitMQHaloWorld

// RabbitMQ hello world project

repos

data-platform-poc

JS_Playground

Javascript code playground

Kaggle.com_mini_project

public data competitions & projects on the https://www.kaggle.com/

Jupyter Notebook

LambdaHelloWorld

AWS Lambda demo projects

Redshift-poc

HousePricePredAPI

ML api predict house price wrapped in Docker and deployed to AWS ECS/Fargate | #DE |#ML

JavaHelloWorld

Java basics project : data structure, algorithm, syntax, concept

CDKPoc

https://cdkworkshop.com/

IreHouse

yennj12_blog_V2

My current blog (via fastpages template)

Jupyter Notebook

ScalaCoursera

My learning on Functional Programming Coursea Scala Course | #SE

GitCommitQ

ETL collect Github commits data (no API account required)