kafka-in-production
Curious to know how big companies are operating their kafka fleet in production? This might be the repo for you:
- What are the issues encountered when running kafka in production? ๐
- How other organisations attempt to solve the issues? ๐ ๏ธ
- Why certain approaches are adopted over others?
โ๏ธ - What can we learn for our own use case?
Table of Contents
- Adobe
- Agoda
- Airbnb
- Apple
- AppsFlyer
- Bloomberg
- Bolt
- Booking.com
- Brex
- Cloudflare
- Coinbase
- Criteo
- Datadog
- Deliveroo
- GoTo
- Grab
- Honeycomb
- Hubspot
- Lyft
- Morgan Stanley
- Netflix
- Riskified
- Robinhood
- Shopify
- Slack
- Stripe
- Uber
- Wise
- Wix
- Yelp
- Zalando
- Zopa Bank
Adobe
- How Adobe Experience Platform Pipeline Became the Cornerstone of In-Flight Processing for Adobe -
2019- ๐ - Moving Beyond Newtonian Reductionism in the Management of Large-Scale Distributed Systems, Part 2 -
2019-๐ - Adobe Experience Platformโs Streaming Sources and Destinations Overview and Architecture -
2019-๐ - Wins from Effective Kafka Monitoring at Adobe: Stability, Performance, and Cost Savings -
2019-๐ - Creating Adobe Experience Platform Pipeline with Kafka -
2018-๐
Agoda
- How Agoda manages 1.5 Trillion Events per day on Kafka -
2021-๐ - Adding Time Lag to Monitor Kafka Consumer -
2021- ๐ - How our data scientists' petabytes of data is ingested into Hadoop (from Kafka) -
2021-๐
Airbnb
- Migrating Kafka transparently between Zookeeper clusters -
2021-๐
Apple
- Balance Kafka Cluster with Zero Data Movement -
2023-๐๏ธ - Experiences Operating Apache Kafkaยฎ at Scale -
2019-๐๏ธ - Kafka as a Service A Tale of Security and Multi Tenancy -
2018-๐๏ธ
AppsFlyer
- Four Crucial Steps to Take Before Changing Kafka Partition Key at Scale -
2023-๐ - Kafka Lag Monitoring For Human Beings -
2020-๐๏ธ - Apache Kafka Lag Monitoring at AppsFlyer -
2020-๐ - Managing your Kafka in an explosive growth environment -
2019-๐๏ธ
Bloomberg
- Fully-Managed, Multi-Tenant Kafka Clusters: Tips, Tricks, and Tools -
2022-๐๏ธ
Bolt
- Using Apache Kafka and ksqlDB for Data Replication at Bolt -
2021-๐๏ธ - How Bolt Has Adopted Change Data Capture with Confluent Platform -
2020-๐ - Kewei Shang -
2020-๐
Booking.com
- Data Streaming Ecosystem Management at Booking.com -
2018-๐
Brex
- Transactional Events Publishing At Brex -
2022- ๐
Cloudflare
- Intelligent, automatic restarts for unhealthy Kafka consumers -
2023-๐ - Using Apache Kafka to process 1 trillion inter-service messages -
2022-๐
Coinbase
- Kafka infrastructure renovation at Coinbase -
2022-๐ - How we scaled data streaming at Coinbase using AWS MSK -
2021-๐
Criteo
- Managing Kafka and Data Streams at Criteo -
2023- ๐ - Upgrading Kafka on a large infra, or: when moving at scale requires careful planning -
2019-๐ - How Criteo is managing one of the largest Kafka Infrastructure in Europe -
2019-๐
Datadog
- Running Production Kafka Clusters in Kubernetes -
2019- ๐๏ธ
Deliveroo
GoTo
- Sink Kafka Messages to ClickHouse Using 'ClickHouse Kafka Ingestor' -
2022-๐ - When Kafka Went Offshore -
2021- ๐ - Enhancing Zigguratโ-โThe Backbone Of Gojek's Kafka Ecosystem -
2021-๐ - Handling Dead Letters in a Streaming System -
2020-๐ - How Kafka Solved a Culture Problem at Gojek -
2019-๐ - Fronting : An Armoured Car for Kafka Ingestion -
2018-๐ - Sakaar: Taking Kafka data to cloud storage at GO-JEK -
2018- ๐
Grab
- Zero trust with Kafka -
2022-๐ - How Kafka Connect helps move data seamlessly -
2022-๐ - Exposing a Kafka Cluster via a VPC Endpoint Service -
2022-๐ - Detect Fraud Successfully with GrabDefence! -
2021-๐๏ธ - Optimally Scaling Kafka Consumer Applications -
2020- ๐
Hubspot
Honeycomb
- Scaling Telemetry Systems with Streaming -
2023-๐๏ธ - Lessons Learned From the Migration to Confluent Kafka -
2021-๐ - Scaling Kafka at Honeycomb -
2021-๐ - Bitten by a Kafka Bug - Postmortem -
2019-๐
- Load-balanced Brooklin Mirror Maker: Replicating large-scale Kafka clusters at LinkedIn -
2022-๐ - TopicGC: How LinkedIn cleans up unused metadata for its Kafka clusters -
2022-๐ - How LinkedIn customizes Apache Kafka for 7 trillion messages per day -
2019-๐ - URP? Excuse You! The Three Metrics You Have to Know -
2018-๐๏ธ - Test Strategy for Samza/Kafka Services -
2017-๐ - Kafka Ecosystem at LinkedIn -
2016-๐ - Kafkaesque Days at LinkedIn โ Part 1 -
2016-๐ - How Weโre Improving and Advancing Kafka at LinkedIn -
2015- ๐
Lyft
- Building an Adaptive, Multi-Tenant Stream Bus with Kafka and Golang -
2020-๐ - Can Kafka Handle a Lyft Ride? -
2020-๐๏ธ - Operating Apache Kafka Clusters 24/7 Without A Global Ops Team -
2019- ๐ - Bulletproof Apache Kafkaยฎ with Fault Tree Analysis -
2019-๐๏ธ - Production Ready Kafka on Kubernetes -
2019-๐๏ธ
Morgan Stanley
Netflix
- Featuring Apache Kafka in the Netflix Studio and Finance World -
2020- ๐ - Inca โ Message Tracing and Loss Detection For Streaming Data @Netflix -
2019-๐ - Evolution of the Netflix Data Pipeline -
2016-๐ - Kafka Inside Keystone Pipeline -
2016-๐
- Lessons Learned from Running Apache Kafka at Scale at Pinterest -
2021-๐ - How Pinterest runs Kafka at scale -
2018-๐ - Open sourcing DoctorKafka: Kafka cluster healing and workload balancing -
2017-๐
Riskified
- How to Manage Schemas and Handle Standardization -
2023-๐ - How to Roll Your Kafka Cluster With Zero Downtime and No Data Loss -
2023-๐ - Know Your Limits: Cluster Benchmarks -
2022-๐ - Letโs Make Your CFO Happy; A Practical Guide for Kafka Cost Reduction -
2022-๐๏ธ - From AWS CloudFormation to Terraform: Migrating Apache Kafka -
2021-๐
Robinhood
- Tackling Kafka, with a Small Team -
2019- ๐๏ธ
Shopify
- Capturing Every Change From Shopifyโs Sharded Monolith -
2021-๐ - Running Apache Kafka on Kubernetes at Shopify -
2018- ๐ - Kafka Producer Pipeline for Ruby on Rails -
2014-๐
Slack
Stripe
- 6 Nines: How Stripe keeps Kafka highly-available across the globe -
2022- ๐๏ธ
Uber
- Securing Kafkaยฎ Infrastructure at Uber -
2022- ๐ - Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot -
2021-๐ - Introducing uGroup: Uberโs Consumer Management Framework -
2021-๐ - Disaster Recovery for Multi-Region Kafka at Uber -
2020-๐ - Kafka Cluster Federation at Uber -
2019- ๐๏ธ - Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka -
2018-๐ - Introducing Chaperone: How Uber Engineering Audits Apache Kafka End-to-End -
2016-๐ - uReplicator: Uber Engineeringโs Robust Apache Kafka Replicator -
2016-๐
Wise
- Streaming Infrastructure at Wise -
2023-๐๏ธ - Rack awareness in Kafka Streams -
2022-๐ - Teamwork: Implementing a Kafka retry strategy at Wise -
2021- ๐ - Running Kafka in Kubernetes, Part 1: Why we migrated our Kafka clusters to Kubernetes. -
2021-๐ - Running Kafka in Kubernetes, Part 2: How we migrated our Kafka clusters to Kubernetes. -
2021-๐ - Securing Kafka with SPIFFE at TransferWise - Jonathan Oddy, Levani Kokhreidze -
2020-๐๏ธ - Achieving high availability with stateful Kafka Streams applications -
2018- ๐
Wix
- 4 Steps for Kafka Rebalance - Notes From the Field -
2021-๐ - Wixโs Journey Into Data Streams -
2021-๐ - Building a High-level SDK for Kafka: Greyhound Unleashed -
2020-๐
Yelp
- Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 1 - Architecture) -
2021- ๐ - Streams and Monk โ How Yelp is Approaching Kafka in 2020 -
2020-๐ - Billions of Messages a Day โ Yelpโs Real-time Data Pipeline -
2017- ๐๏ธ
Zalando
- Rock Solid Kafka and ZooKeeper Ops on AWS -
2018- ๐ - Many-to-Many Relationships Using Kafka -
2018- ๐ - Event First Development - Moving Towards Kafka Pipeline Applications -
2017- ๐ - Reattaching Kafka EBS in AWS -
2017-๐ - Real-time Ranking with Apache Kafkaโs Streams API -
2017-๐ - Running Kafka Streams applications in AWS -
2017-๐ - A Recipe for Kafka Lag Monitoring -
2017-๐ - Surviving Data Loss -
2017- ๐
Zopa Bank
- Highly Available Kafka Consumers and Kafka Streams on Kubernetes -
2023-๐๏ธ