• Stars
    star
    1
  • Language
    Scala
  • Created over 1 year ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This project shows a real world use case with ETL batch pipeline using Cloud Storage, Scala Spark with Dataproc Serverless and BigQuery orchestrated by Cloud Composer / Airflow

More Repositories

1

asgarde

Asgarde allows simplifying error handling with Apache Beam Java, with less code, more concise and expressive code.
Java
71
star
2

bigtesty

BigTesty is a framework that allows to create Integration Tests with BigQuery on a real and short lived Infrastructure.
Python
52
star
3

pasgarde

Asgarde allows simplifying error handling with Apache Beam Python, with less code, more concise and expressive code.
Python
31
star
4

midgard

Midgard is a wrapper on Beam Kotlin, allowing more concise and expressive code. It removes Beam boilerplate code and proposes more Functional Programming style
Kotlin
17
star
5

teams-league-cloud-workflows-elt

This project shows a real world use case with ELT pipeline using Cloud Storage, BigQuery and Cloud Workflows
HCL
7
star
6

dataflow-python-ci-cd

Project showing a CI CD pipeline for Dataflow Python with Flex Template and Cloud Build
Python
7
star
7

teams-league-python-ddd-beam-summit

Python
6
star
8

dataflow-java-ci-cd

Project showing a CI CD pipeline for Dataflow Java with Flex Template and Cloud Build
Java
5
star
9

teams-league-airflow-cloudrun-etl

This project shows a real world use case with ETL batch pipeline using Cloud Storage, Cloud Run Service and BigQuery orchestrated by Cloud Composer / Airflow
Python
4
star
10

teams-league-airflow-elt

The goal of this article is showing a real world use case for ELT batch pipeline, with Cloud Storage, BigQuery, Apache Airflow and Cloud Composer : The Extract part is managed in Cloud Storage The Load part is managed from Cloud Storage to BigQuery The Transform part is managed by a BigQuery SQL query Everything is orchestrated by Airflow
Python
4
star
11

world-cup-qatar-elt-dbt

This repo shows a real world use case with DBT, BigQuery and Google Cloud. The raw and input data are represented by the Qatar Fifa World Cup Players stats, some transformations are applied with the ELT pattern and DBT to apply aggregation and business transformations.
Shell
4
star
12

teams-league-cloudrun-service-fastapi

Project showing a complete use case with a Cloud Run Service written with a Python module and multiple files. The deployment of service is done with FastApi and Uvicorn.
Python
3
star
13

teams-league-python-dlq-asgarde-beam-summit

Python
3
star
14

world-cup-qatar-team-stats-kotlin-midgard

This application shows a full Apache Beam pipeline with Kotlin and Midgard library. The use case works on the last Qatar FIFA world cup data and calculate players statistics per team. This application will be presented at Beam Summit 2023 in New York
Kotlin
3
star
15

sa-custom-roles-gcp-terraform

This project shows a complete use case with the least privilege principle on Google Cloud using modular Terraform, Terragrunt and Cloud Build
HCL
3
star
16

datasets-tables-bq-one-module-terraform

This project how to create BigQuery Datasets and tables with Terraform and elegant Json configuration. This example use a single Terraform module to create datasets and tables. The deployment of IAC part is done with Cloud Build.
HCL
2
star
17

teams-league-cloud-workflows-etl-dataflow

This project shows a real world use case with ETL pipeline using Cloud Storage, Dataflow, BigQuery and Cloud Workflows
HCL
2
star
18

datasets-tables-bq-multi-modules-terraform

This project how to create BigQuery Datasets and tables with Terraform and elegant Json configuration. This example use two Terraform modules to create datasets and tables. The deployment of IAC part is done with Cloud Build.
HCL
2
star
19

teams-league-spark-scala-dataproc-serverless

This project shows a complete, concrete and a real world use case with a Spark Scala job run with Dataproc Serverless on Google Cloud.
Scala
2
star
20

tosun-si

TOSUN SI repository
Java
2
star
21

teams-league-java-ddd-beam-summit

Java
2
star
22

world-cup-qatar-event-driven-serverless-archi-workflows

Project showing a use case with a full Event Driven and Serverless Architecture with Cloud Functions, Cloud Run services and Cloud Workflows
Python
2
star
23

teams-league-cloudrun-job

Project showing a complete use case with a Cloud Run Job written with a Python module and multiple files.
Python
1
star
24

bees-demo-fp

This project shows some legacy code and patterns and a refactoring for each of them with Functional Programming, lambda and functions composition
Java
1
star
25

teams-league-airflow-beam-bq

Python
1
star
26

syomi

Scala
1
star
27

tp-teams-handling

TypeScript
1
star
28

java8-example

Java
1
star
29

datasets-tables-bq-pulumi

This project shows how to create BigQuery Datasets with tables using Pulumi and elegant/scalable Json configuration. This use cas was previously created with Terraform and the goal is to rewrite it with Pulumi and the Python SDK.
Python
1
star
30

teams-league-java-dlq-asgarde-beam-summit

Project presented at Beam Summit 2022 that shows a complete use case with a Dead Letter Queue on errors with Beam Java and Asgarde library.
Java
1
star
31

event-arc-trigger-function

This project shows a complete use case with an event driven Cloud Function written in Python and triggered with Event Arc
HCL
1
star
32

world-cup-qatar-team-stats-java

This application shows a full Apache Beam pipeline with Java. The use case works on the last Qatar FIFA world cup data and calculate players statistics per team. This application will be presented at Beam Summit 2023 in New York. A Kotlin version of this use case is also used in another repo.
Java
1
star
33

airflow-composer-error-handling

This project shows how to apply error handling in Airflow DAGs and Cloud Composer in Google Cloud. Instead of repeat a technical code of error handling in each DAG, the principle is to use a callback and a failure interceptor.
Python
1
star
34

world-cup-qatar-event-driven-serverless-archi

This project show a complete and real world use case with Event Driven and Serverless architecture on Google Cloud
Python
1
star