madd86/awesome-system-design madd86/awesome-system-design

  • Stars
    star
    9,405
  • Rank 3,800 (Top 0.08 %)
  • Language
  • License
    Creative Commons ...
  • Created over 4 years ago
  • Updated 5 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
madd86/awesome-system-design
github

madd86

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.

Awesome

If you appreciate the content ๐Ÿ“–, support projects visibility, give ๐Ÿ‘| โญ| ๐Ÿ‘

A curated list of awesome System Designing articles, videos and resources for distributed computing, AKA Big Data.

Whether you're preparing for an interview or you want to design a distributed/microservice oriented application, this list will definitely help you achieve that.

Attention: Stars on GitHub does not reflect usage or popularity for every item here listed.

Inspired By Awesome-BigData

Started By Gabriel Leon de Mattos

Contents

Articles

Books

Videos

Tools

Bonus

Articles

Introduction / Interviews

Advanced

Books

  • Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services - [Paid ๐Ÿ’ต] - Book that talks about disitributed systems as well as lightly demonstrating some code of what it looks like.

  • Designing Data Intensive Applications - [Paid ๐Ÿ’ต] - Goes in depth to explain various resources we use when working with distributed systems, as well as how it came to be and what problems it aims to solve.

  • The System Design Manual - [Paid ๐Ÿ’ต] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.

  • Building Microservices - [Free ๐Ÿ‘] - Awesome book that talks about designing sytem architecture with microservices in depth, includes most relevant topics in this regard.

  • Monolith to Microservices - [Free ๐Ÿ‘] - Written by the same author as the one above, this book will cover the migration from Monolith to Microservices, it's recommended you start by the previous book.

  • Distributed Systems (3rd Edition) - [Free ๐Ÿ‘] - Great overview of and in-depth introduction to distributed systems. Recommended for intermediate level readers.

Videos

A collection of videos based on distributed systems.

Introduction / Interviews

Advanced

Tools

  • A collection of most commonly used tools for distributed systems

Relational Database Management System

  • MariaDB - MariaDB is a fork of MySQL server.

  • MySQL - Widely used relational database.

  • PostgresSQL - Relational database that has been gaining popularity.

  • SQLite - Another widely used database that is built into all mobile phones and most computers.

  • Sql Server - Widely used relational database.

NoSQL

Cache (Key-Value)

  • Apache Ignite - [3.3k โญ] - In memory caching with ACID properties.

  • Couchbase - Inspired by memcached, adding features such as replication and persistance.

  • Oracle Coherence - [126 โญ] - High scaling, low latency in-memory caching.

  • Memcached - [10.2k โญ] - One of the first in-memory caching database, high performing and multi-threaded.

  • Redis - [44k โญ] - Widely used in-memory caching database with many added features such as persistent storage and supporting strings, lists, sets, hashses, streams, bitmaps, etc.

Store (Key-Value)

  • Apple FoundationDB - [10k โญ] - Multi-model (many data types in a single database), ACID key-value store. Easily scalable and fault tolerant.

  • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.

Document Store

  • CouchDB - [4.6k โญ] - ACID compliant NoSQL document-store DB, provides a RESTful HTTP API for reading and updating database documents.

  • MongoDB - One of the most popular 'NoSQL' database for general purpose.

  • RethinkDB - [23.8k โญ] - Document-store DB.

  • ElasticSearch - [49.9k โญ] - Widely popular 'NoSQL' database for fast and scalable search engines.

  • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.

Wide Column Store

  • Amazon DynamoDB - Key-Value and Document database, highly performant, scalable and secure.

  • Google Bigtable - Scalable and performant 'NoSQL' database for large analytical and operational workload.

  • Cassandra - Facebook-born project very fast, easily scalable, with option to include consistency with each operation.

  • Scylla - [4.9k โญ] - 'NoSQL' data store using seastar framework, compatible with Cassandra.

  • HBase - [3.6k โญ] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project and runs on top of HDFS or Alluxio. (See Hadoop Related)

  • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.

Graph

  • Amazon Neptune - Fast, reliable and fully managed graph database service.

  • ArangoDB - [10k โญ] - Flexible database for documents, key-value, graphs. Uses its own query language, AQL.

  • Neo4j - [7.9k โญ] - Good support for a graph db, ACID compliant and flexible.

  • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.

Distributed File Systems

  • HDFS - Hadoop File System is a a widely popular choice among its big data competitors, providing high throughput access.

  • Lustre - File system for computer clusters.

  • CephFS - Unified, distributed storage system.

  • GlusterFS - Scale-out NAS file system.

  • MooseFS - POSIX-compliant distributed file system.

  • XtreemFS - Fault tolerant file system.

Resource Management

  • Kubernetes - Highly popular way to deploy, manage and automatically scale a cluster of containers on bare-metal or virtual servers.

Stream Processing

  • Apache Samza - Build stateful applications that process data in real time from multiple sources, including Kafka. Easy and inexpensive multi-subscriber model, can eliminate backpressure and has reliable persistency with low latency.

  • Apache Flink - Based on the concept of streams and transofrmations. Uses maven, handles batch tasks as data streams with finite boundaries. Low latency, high throughput.

  • Amazon Kinesis Streams - Durable, scalable, real-tme service. Collects gigabytes of data per second from hundreds of thousands of sources, including database event streams, website clickstreams, financial transactions, etc.

  • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.

Message Broker

  • Amazon MQ - Open source message broker from Amazon.

  • Apache ActiveMQ - It's a multi-protocol, java based messaging server.

  • Apache Kafka - Widely popular message broker with low latency for data streaming.

  • RabbitMQ - Widely popular lightweight message broker written in erlang that also supports multiple messaging protocols.

  • IronMQ - Very fast and highly scalable messaging broker. (not open source)

  • Apache Pulsar - Created by yahoo, also highly scalable, low latency, geo-replication and multi-tenacy.

  • Kestrel - Written in Scala and speaks the memcached protocol. It works much like Kafka.

  • Azure Service Bus - A fully managed enterprise integration message broker.

Load Balancers

Open Source Software

  • SeeSaw - [5.1k โญ] - Used by Google, developed in Go, linux-based virtual load balancer server.

  • HAProxy - Widely popular option, provides high-availability, proxy, TCP/HTTP load balancing. Used by Reddit, Imgur, MaxCDN, GitHub, AirBNB.

  • Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.

  • Neutrino - Used by eBay, built with Scala and Netty. Supports round-robin and least-connection algorithms.

  • Nginx - Wait, isn't Nginx a web server? Yes, the open source does support basic level of content switching and request routing. Plus edition supports load balancing, WAF, monitoring, etc.

  • Openresty - Nginx + Lua, perfect combination.

Hardware

  • F5 - Robust hardware load balancer option, supporting multiple protocols (IP, TCP, FTP, UDP, HTTP).

  • TP-Link - Cheaper alternative that works as a load balancer.

  • Barracuda - One of the top choices for load balancing when it comes to in-house servers. Top security measures built in, comprehensive reports and monitoring outbound traffic for data loss prevention.

Cloud

  • Amazon Elastic Load Balancing - Popular choice for amazon customers, supports lambda functions, highly scalable.

  • Google Load Balancing - Popular choice for google customers, comes with auto-scaling feature, very fast, has intergrated CDN.

  • Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.

  • DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.

  • Azure Load Balancing - Popular choice for Microsoft's Azure customers. Supports internal and external traffics, ipv6, monitorining and the standard load balancing set of features.

Hadoop Ecosystem

Dashboard

  • Ambari - Dashboard that integrates most of hadoop related technologies for easy management and executions.

Data Ingestion

  • Sqoop - Efficiently transfer data between Hadoop and structured datastores such as relational databases.

  • Flume - Distributed, highly available and efficient in collecting, aggregating and moving large amounts of log data.

  • Apache Kafka - Widely popular message broker with low latency for data streaming.

Workflow Scheduler

  • Oozie - Create workflows in xml to execute jobs (from other hadoop-ecosystem applications) in steps, allows for parallel execution as well.

Query

  • Hive - Query hadoop stored data in SQL.
  • Pig - Scriping language that looks like SQL to query hadoop data.

Processing

  • Tez - Solves a similar problem to Spark and MapReduce, it's more efficient than MapReduce because it calculates the most efficient way of doing it.
  • Map Reduce - MapReduce, as the name implies, maps data and reduce the results.
  • Spark - Powerful data processing to not only process data like Tez (and MapReduce), it can process streams of data in real time, apply regression analysis algorithms in ML and much more.
  • Apex - *Retired project, it's a YARN-native platform that unifies stream and batch processing.

DB

  • HBase - [3.6k โญ] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project.

Resource Management

  • YARN - 'Yet Another Resource Negotiator', works like a kernel to manage computer resources across the clusters.
  • MESOS - Works like a Linux Kernel by managing CPU, memory, storage and other resources across the cluster.

REST Framework

  • Gin - [40.6k โญ] - Blazingly fast microservice framework using Golang, high throughput capacity.

  • Phoenix - [15.5k โญ] - Distributed processing, easily scalable, support for channels and live chat. This framework - written in Elixir, uses BEAM and Erlang, very efficient for large scale systems and supports high throughput.

  • Express.js - [49.6k โญ] - Fast node.js rest api that can perform well under many scenarios.

  • Rails - [46.2k โญ] - Written in Ruby, Rails delivers quick apis from prototype to production in an efficient manner.

  • Play Framework - [11.6k โญ] - Very fast, high throughput framework written in Scala/Java that is RESTful by default.

  • Flask - [51.6k โญ] - A lightweight Python Microframework for fast prototyping and production.

  • FastAPI - [22.7k โญ] - A lightweight Python Microframework inspired in Flask but more modern, using Python async.

  • Django REST - [18.4k โญ] - Written in Python, Django Rest is a powerful and flexible REST API. The efficiency and time to market resembles Rails.

  • ASP.NET Core MVC - A rich framework for building web apps and APIs using the Model-View-Controller design pattern in C# or F#. Number 6 on TechEmpower Composite Benchmarks for web frameworks.

  • Fastify - [15.4k โญ] - A Node.js web framework highly focused on providing the best developer experience with the least overhead and a powerful plugin architecture.

More Repositories

1

compress_image

A Flutter plugin to compress images
Dart
2
star
2

AmazonScrapper

Blazing fast amazon scrapper written in Golang
Go
2
star