josephmachado/data_engineering_best_practices_log

Stars
1
Language
Python
Created 9 months ago
Updated 9 months ago

josephmachado/data_engineering_best_practices_log

josephmachado

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Code to demonstrate data engineering metadata & logging best practices

beginner_de_project

Beginner data engineering project - batch edition

data_engineering_project_template

A template repository to create a data project with IAC, CI/CD, Data migrations, & testing

simple_dbt_project

Code for dbt tutorial

bitcoinMonitor

Near real time ETL to populate a dashboard.

online_store

End to end data engineering project

beginner_de_project_stream

Simple stream processing pipeline

spark_submit_airflow

Simple repo to demonstrate how to submit a spark job to EMR from Airflow

analytical_dp_with_sql

Code for my "Analytical Data Processing in SQL" book.

socialetl

Project for "Data pipeline design patterns" blog.

e2e_datapipeline_test

Example repo to create end to end tests for data pipeline.

local_dev

Local development environment for python data projects, with Docker

data_test_ci

Repository showing how to automate data testing as part of CI

change_data_capture

Repo for CDC with debezium blog post

idempotent-data-pipeline

Making data pipelines idempotent

dbt_development

Repo to explain development, CI/CD cycle in dbt

data-engineering-interview-series

WIP repository for Data Engineering Interview Series

Jupyter Notebook

trigger_spark_with_lambda

Simple example showing how to trigger a spark job with AWS Lambda

adv_data_transformation_in_sql

Code for "Advanced data transformations in SQL" free live workshop

data_engineering_best_practices

spark_submit_airflow-

Simple repo to demonstrate how to submit a spark job

python_essentials_for_data_engineers

recipes

personal how-tos for common DE tasks

josephmachado

unit_test_dbt

unit test example in DBT

sde_superset_demo

Apache Superset Demp

docker_for_data_engineers

how-to-slash-dbt-cost-w-duckdb-