System Design Bytes for Enthusiasts
I complied and collected some of the articles as a part of doing CS654A course at IIT Kanpur for a fantastic course on Advanced Software Architecture taught by Prof.Dr.T.V.Prabbhakar.
Modified from Zach system design repository. Added more links and topics to cover on both PS/DS & System Design Interviews. We will keep updating this posting from time to time. Some more awesome resource
SYSTEM DESIGN IS NOT JUST FOR INTERVIEWS, IT CAN BE HOW YOU MANAGE YOUR LIFE. GIVE IT A THOUGHT
Disclaimer/Acknowledgements
I didn't write the articles myself. I love expanding my understanding of system design complexities and thus read many books, research papers, and articles that enhance my knowledge. I take the opportunity to thank everyone who found this repository useful. I would like to thank all the authors and developers who wrote the articles aggregated in this repository.
More Content Comming !
We are working to create good content to fulfil the needs of system-design noobs. Please feel free to share any resource or material at [email protected].
SWEBOOK & PDFs
- http://swebokwiki.org
- https://cs.fit.edu/~kgallagher/Schtick/Serious/SWEBOKv3.pdf
- System Design Basics by Aditya Malshikhare
All-Time Favourites
How to prepare system design questions?
System design is a very broad topic. Even a software engineer with many years of working experience at a top IT company may not be an expert on system design. If you want to become an expert, you need to read many books, articles, and solve real large scale system design problems.
In order to practice designing a few systems, for the love or learning or preparing for interviews, you can use this practice platform.
This repository only teaches you how to handle the system design interview with a systematic approach in a concise way. You can dive into each topic if you have time. Of course, welcome to add your thoughts!.
Table of Contents
- System Design Interview Tips & PS/DS
- Basic Knowledge about System Design
- Company Engineering Blogs
- Products and Systems
- Hot Questions and Reference
- Good Books
- Object Oriented Design
- Distributed System Design
- Projects to Try
[β¬] System Design Interview Tips & PS/DS
Clarify the constraints and identify the user cases
Spend a few minutes questioning the interviewer and agreeing on the scope of the system. Remember to make sure you know all the requirements the interviewer didn't tell you about in the beginning.
User cases indicate the main functions of the system, and constraints list the scale of the system such as requests per second, requests types, data written per second, data read per second.
In a real world scenario, beyond a white-board interview setting one must understand the functional & non-functional requirements of an
engineering project.
- Requirements Engineering
- How to manage Modern Software Projects ?
- A Design Doc : Online Judge (https://www.youtube.com/watch?v=_eRTl_YxgUw)
- Important Data Engineering
High-level architecture design
Sketch the important components and the connections between them, but don't go into some details. Usually, a scalable system includes webserver (load balancer), service (service partition), database (primary/secondary database cluster plug cache).
Component design
For each component, you need to write the specific APIs for each component. You may need to finish the detailed OOD design for a particular function. You may also need to design the database schema for the database.
Revist Basic PS/DS & Machine Coding Questions
Basics of OOPs, SOLID Concepts, algorithms and data structures will help a lot in clearing most tof the onsite interviews at large tech companies. Having a habit of solving problems on LeetCode or CodeFoces will definitely help in thinking faster and better in terms of approach to problem solving.
- What is PS/DS?
- What is Machine Coding?
- How to practice for Machine Coding?
- Sample Machine Coding Problems
- LeetCode Problems
- Try the CodeChef Medium Problems
- Try this as well
[β¬ ] Basic Knowledge about System Design:
Here are some articles about system design related topics.
- Chaos Engineering
- The New Stack Blogs
- An Illustrated Proof of the CAP Theorem
- Educative.io & Other Lists
- Node Clean Architecture
- Services Engineering Resources
- How to Rock a Systems Design Interview
- System Interview
- SE Daily : Fiverr Engineering with Gil Sheinfeld
- Scalability for Dummies
- Scalable Web Architecture and Distributed Systems
- Numbers Everyone Should Know
- Fallacies of distributed systems
- Designing Tinder
- Scalable System Design Patterns
- Introduction to Architecting Systems for Scale
- Transactions Across Datacenters
- A Plain English Introduction to CAP Theorem
- The CAP FAQ
- Paxos Made Simple
- Simple : Consistent Hashing
- More On : Consistent Hashing
- NOSQL Patterns
- Emerging Architectures
- Scalability, Availability & Stability Patterns
- Simple Publisher-Subscriber design for Koinex : PushMan
- Web Architecture 101
- Testing in Production: the hard parts
- Distributed Tracing β weβve been doing it wrong
- Distributed Consistent Hashing
- System Design White Board
- Bloom Filter : A Probabilistic Data Structure
- Basics : ML Architecture
- Security By Design
- How Netlifyβs deploying and routing infrastructure works
- Basic Overview of Blockchain
- Architecting a Machine Learning Pipeline
- ML Reference Architecture
- Reducing Databse Costs
- Patterns : Thinking About Microservices
- The Reactive Monolith - How to Move from CRUD to Event Sourcing
- Terraform Software
- FreeCodeCamp : DevOps Course
- Some Linux Know How is necessary
- Architecture of Open Source Systems
- Some Quickstart ML know-how for system design : What to use? Where to use it?
- Quick : Observability and Microservices
- System Design Interviews
- Designing Microservices
- Crack the System Design Interview - FreeCodeCamp
- Intro to Distributed Logging
- Usenix -- Gmail Overview
- Jeff Dean On Large-Scale Deep Learning At Google
- Infinite Memory Transformer: Attending to Arbitrarily Long Contexts Without Increasing Computation Burden Researchers from Instituto de Telecomunicaçáes.
Of course, if you want to dive into system related topics, here is a good collection of reading list about services-engineering, and a good collection of material about distributed systems.
[β¬] Off-topic Blogs:
- Architecture of Machine Learning Systems
- Engineering Dependability And Fault Tolerance In A Distributed System - @@Ably
- Postman API Testing
- Microservices and the Economics of Small Things
- Service Mesh Ultimate Guide - Second Edition: Next Generation Microservices Development
- Adoption of Cloud Native Architecture, Part 3: Service Orchestration and Service Mesh
[β¬ ] Company Engineering Blogs:
If you are going to have an onsite with a company, you should read their engineering blog.
- InfoQ
- High Scalability
- Complex Systems: Microservices and Humans
- The GitHub Blog
- Engineering at Quora
- Yelp Engineering Blog
- Twitter Engineering
- Facebook Engineering
- Yammer Engineering
- Etsy Code as Craft
- Foursquare Engineering Blog
- Airbnb Engineering
- WebEngage Engineering Blog
- LinkedIn Engineering
- The Netflix Tech Blog
- BankSimple Simple Blog
- Square The Corner
- SoundCloud Backstage Blog
- Flickr Code
- Instagram Engineering
- Dropbox Tech Blog
- Cloudera Developer Blog
- Bandcamp Tech
- Oyster Tech Blog
- THE REDDIT BLOG
- Groupon Engineering Blog
- Songkick Technology Blog
- Google Research Blog
- Pinterest Engineering Blog
- Twilio Engineering Blog
- Bitly Engineering Blog
- Uber Engineering Blog
- Godaddy Engineering
- Splunk Blog
- Coursera Engineering Blog
- PayPal Engineering Blog
- Nextdoor Engineering Blog
- Booking.com Development Blog
- Scalyr Engineering Blog
- Architect.io Blog
- Ably Engineering Blog
- DoorDash Engineering Blog
[β¬] Products and Systems:
The following papers/articles/slides can help you to understand the general design idea of different real products and systems.
- MapReduce: Simplied Data Processing on Large Clusters
- Bigtable: A Distributed Storage System for Structured Data
- The Google File System
- The Chubby lock service for loosely-coupled distributed systems
- Dynamo: Amazon's Highly Available Key-value Store
- Frontend in React
- State Management using Redux
- Angular Get Started
- First DJango App
- Build a CRUD Todo app with Django and React/Redux
- The MERN Stack Tutorial β Building A React CRUD Application From Start To Finish
- Introduction to Memcached
- Cassandra Introduction Features
- Introduction to HBase
- Introduction to MongoDB
- Introduction to Redis
- Storm
- Introduction to Zookeeper
- Kafka
- YouTube Architecture
- Scaling Pinterest
- Google Architecture
- Scaling Twitter
- The WhatsApp Architecture
- Flickr Architecture
- Amazon Architecture
- Stack Overflow Architecture
- Pinterest Architecture
- Tumblr Architecture
- Instagram Architecture
- TripAdvisor Architecture
- Scaling Mailbox
- Salesforce Architecture
- ESPN Architecture
- Uber Architecture
- DropBox Design
- Splunk Architecture
- Good Parts of AWS
- Azure DataCenter Architecture
- Evolution Of Search Engines Architecture - Highscalability
Trends
- Emerging Trends in Software Architecture
- Software Architecture and Design InfoQ Trends ReportβApril 2020
[β¬] Hot Questions and Reference:
There are some good references for each question. The references here are slides and articles.
Design a CDN network
Reference:
Design a Google document system
Reference:
Design a random ID generation system
Reference:
Design a key-value database
Reference:
Design the Facebook news feed function
Reference:
- Deisgn a News Feed System
- What are best practices for building something like a News Feed?
- What are the scaling issues to keep in mind while developing a social network feed?
- Activity Feeds Architecture
Design the Facebook timeline function
Reference:
Design a function to return the top k requests during past time interval
Reference:
- Efficient Computation of Frequent and Top-k Elements in Data Streams
- An Optimal Strategy for Monitoring Top-k Queries in Streaming Windows
Design an online multiplayer card game
Reference:
- How to Create an Asynchronous Multiplayer Game
- How to Create an Asynchronous Multiplayer Game Part 2: Saving the Game State to Online Database
- How to Create an Asynchronous Multiplayer Game Part 3: Loading Games from the Database
- How to Create an Asynchronous Multiplayer Game Part 4: Matchmaking
- Real Time Multiplayer in HTML5
Design a graph search function
Reference:
- Building out the infrastructure for Graph Search
- Indexing and ranking in Graph Search
- The natural language interface of Graph Search and Erlang at Facebook
Design a picture sharing system
Reference:
Design a search engine
Reference:
Design a recommendation system
Reference:
Design a tinyurl system
Reference:
Design a garbage collection system
Reference:
Design a scalable web crawling system
Reference:
Design the Facebook chat function
Reference:
Design a micro blogging site
Reference:
Design hotel booking platform
Reference:
Design a trending topic system
Reference:
- Implementing Real-Time Trending Topics With a Distributed Rolling Count Algorithm in Storm
- Early detection of Twitter trends explained
Design a cache system
Reference:
Design LeetCode
Design a Payment System
Basic SRE Reference:
Design a React Native Application Reference :
Numbers from concerte implementations References :
[β¬ ] Good Books:
- Big Data: Principles and best practices of scalable realtime data systems
- Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data
- Building Microservices: Designing Fine-Grained Systems
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
- Site Reliability Engineering - How Google Runs Production Systems (English, Paperback, Beyer Betsy)
[β¬] Object Oriented Design:
Tips for OOD Interview
Clarify the scenario, write out user cases
Use case is a description of sequences of events that, taken together, lead to a system doing something useful. Who is going to use it and how they are going to use it. The system may be very simple or very complicated.
Special system requirements such as multi-threading, read or write oriented.
Define objects
Map identity to class: one scenario for one class, each core object in this scenario for one class.
Consider the relationships among classes: certain class must have unique instance, one object has many other objects (composition), one object is another object (inheritance).
Identify attributes for each class: change noun to variable and action to methods.
Use design patterns such that it can be reused in multiple applications.
Useful Websites
- 101 Design Patterns & Tips for Developers
- A Solid Guide to SOLID Principles
- SOLID Design Principles Explained
- SOLID Principles made easy
- Becoming a better developer by using the SOLID design principles by Katerina Trajchevska
- Common System Design Interview Questions
[β¬] Distributed System Design :
Distributed System Design
- Basics of Lamport Clocks
- Distributed Consensus : RAFT
- Distributed Consensus : PAXOS
- Two Generals Problem
[β¬] Projects to try :
Misc
- Scale like a Pro
- GoLang to Node.Js
- Documenting Software Architecture Decisions
- Choosing the correct storage solutions
More Info
Feel free to contact Sumit Lahiri