Personal R&D Lab

Making magic happen

🪴 Project Activity

👩‍🚀 Introduction

A personal research and development (R&D) lab that facilitates the sharing of knowledge.

🚀 Architecture

The diagram illustrates the repository's architecture, which is considered overly complex. It is essential to thoroughly understand the tradeoffs associated with before onboarding any technology into your project.

(The diagram here may take a moment to load. Please wait patiently.)

📦 Setup

make kubernetes-set-up

make kubernetes-clean

🧱 Tech Stack

Web

React - Web framework
Redux - State container
React Query - Hooks for fetching, caching and updating asynchronous data
redux-observable - Side effects
RxJS - Asynchronous programming with observable streams
graphql-tag - GraphQL query parsing
Bulma - CSS framework
PurgeCSS - Unused CSS removing
Jest - Unit testing, snapshot testing
React Testing Library - React component testing
Storybook - Visual testing
- Chromatic - Storybook reviewing
rxjs/testing - Marble testing
Cypress - End-to-end testing
Lighthouse CI - Performance, accessibility, search engine optimization (SEO), progressive web app (PWA) analysis
Sentry - Error tracking
Report URI - Security reporting
Google Tag Manager - Tag management
Google Analytics - Web analytics
FullStory - Experience analytics, session replay, heatmaps
Namecheap - Domain
Cloudflare - CDN, DNS, DDoS protection
HTTP/3 Check - HTTP/3 checking
hstspreload.org - HSTS checking
Mozilla Observatory - Security monitoring
UptimeRobot - Uptime monitoring

Mobile - iOS

SwiftUI - UI framework
XCTest - Unit testing, performance testing
Slather - Code coverage reports generating

Mobile - Android

AndroidX - Android Jetpack
JUnit - Unit testing, instrumented testing

Mobile - React Native

Expo - Universal native apps making platform
React Native - Mobile application framework
UI Kitten - UI library
React Native Testing Library - React Native component testing

Desktop - Qt

Qt Quick - Cross-platform application development framework
QML - Qt modeling language

API Server - Go

Gin - Web framework
gRPC - Remote procedure call (RPC) framework
graphql-go - GraphQL
jwt-go - JSON Web Token (JWT)
gin-contrib/cors - Cross-Origin Resource Sharing (CORS)
opa - Open Policy Agent
dgo - Dgraph client
minio-go - MinIO client
go-redis - Redis client
pgx - PostgreSQL driver
Resty - HTTP client
Squirrel - SQL query builder
apm-agent-go - Application performance monitoring (APM) agent
OpenTelemetry Go - OpenTelemetry
Prometheus Go - Prometheus
Testify - Unit testing
GoDotEnv - Environment variables loading
jsonparser - JSON parser
zerolog - Logging

API Server - Python

Flask - Web framework
Flask-APScheduler - APScheduler
Flask-CORS - Cross-Origin Resource Sharing (CORS)
Flask-Sock - WebSocket
confluent-kafka - Kafka client
Gunicorn - Python web server gateway interface (WSGI) HTTP server
asyncpg - PostgreSQL client
Tenacity - General-purpose retrying library
pytest - Unit testing
pydantic - Data validation
pypdf - PDF library
Poetry - Python package management
- Poe the Poet - Poetry task runner

API Server - Node.js

Express - Web framework
GraphQL.js, express-graphql - GraphQL
- graphql-ws, graphql-subscriptions - GraphQL subscriptions
- graphql-upload - GraphQL upload
- graphql-shield - GraphQL permissions
- graphql-depth-limit - GraphQL depth limit
- graphql-query-complexity - GraphQL query complexity analysis
DataLoader - Batching and caching
Knex.js - SQL query builder
node-postgres - PostgreSQL client
ioredis - Redis client
rate-limiter-flexible - Rate limiting
expressjs/cors - Cross-Origin Resource Sharing (CORS)
csurf - CSRF protection
jsonwebtoken, express-jwt - JSON Web Token (JWT)
bcrypt - Password hashing
axios - HTTP client
Helmet - HTTP header Content-Security-Policy, Referrer-Policy, Strict-Transport-Security, X-Content-Type-Options, X-DNS-Prefetch-Control, X-Download-Options, X-Frame-Options, X-Permitted-Cross-Domain-Policies, X-XSS-Protection
Report To - HTTP header Report-To
Network Error Logging - HTTP header NEL
express-request-id - HTTP header X-Request-ID
response-time - HTTP header X-Response-Time
connect-timeout - Request timeout
Terminus - Health check and graceful shutdown
pino - Logging
dotenv-flow - Environment variables loading
Stryker - Mutation testing
SuperTest - HTTP testing
autocannon - HTTP benchmarking
Clinic.js - Performance profiling
Node.js - JavaScript runtime
npm - JavaScript package management

Database

Trino - Distributed SQL query engine
PostgreSQL - Object-relational database
- Postgres Operator - PostgreSQL high-availability (HA) template
- Postgres Operator - PostgreSQL cluster provisioning
- pgAdmin - PostgreSQL management tool
MySQL - Relational database
- MariaDB - Fork of MySQL
Hydra - Column-oriented SQL database
ClickHouse - Column-oriented SQL database
YugabyteDB - Distributed SQL database
TimescaleDB - Time-series SQL database
InfluxDB - Time-series database
- InfluxDB Enterprise - Distributed time-series database
Prometheus - Time-series database
Loki - Log aggregation system
Apache Cassandra - Distributed wide-column NoSQL database
Qdrant - Distributed vector database
Chroma - Distributed vector database
Dgraph - Distributed graph database
Elasticsearch - Distributed document-oriented search engine
- Kibana - Elasticsearch visualization
Redis - Distributed in-memory key–value database
- KeyDB - Multithreaded fork of Redis
MinIO - Object storage
Apache ZooKeeper - Distributed coordination system
Apache Hadoop - Software utility collection
- Apache Hadoop HDFS (Distributed File System) - Distributed file system
- Apache Hadoop YARN (Yet Another Resource Negotiator) - Resource management and job scheduling framework
- Apache Hadoop MapReduce - Data processing framework
Apache Hive - Distributed data warehousing and SQL-like query language system built on top of Apache Hadoop
Apache Kafka - Distributed event streaming platform
- Schema registries
  - Confluent Schema Registry - Schema Registry
  - Apicurio Registry - Schema Registry
- Connectors
  - Debezium - Distributed change-data-capture (CDC) platform
  - kafka-connect-elasticsearch - Elasticsearch sink connector
  - confluentinc-kafka-connect-jdbc - JDBC source and sink connector
  - debezium-connector-postgres - PostgreSQL CDC source connector
  - http-connector-for-apache-kafka - HTTP sink connector
  - kafka-connect-avro-converter - Confluent Avro converter
  - apicurio-registry-distro-connect-converter - Apicurio Avro converter
- Management tools
  - Redpanda Console - Kafka management tool
  - AKHQ - Kafka management tool
  - UI for Apache Kafka - Kafka management tool
Delta Lake - Data lakehouse
golang-migrate/migrate - Database migrations

Data Engineering, Data Science, Machine Learning

NumPy - Scientific computing library
pandas - Data analysis library
AWS SDK for pandas - pandas on AWS
Modin - pandas workflows scaling
Ray - Distributed computing framework
Jupyter Notebook - Web-based interactive computing platform
- nb-clean - Jupyter notebook cleaning
Prefect - Orchestration platform
Apache Airflow - Orchestration platform
Temporal - Orchestration platform
Apache Spark - Data processing framework
- Spark ML - Spark machine learning
- pyspark - Spark API library
- Delight - Spark UI and history server
Apache Flink - Data processing framework
- flink-streaming-java - Flink
- flink-connector-twitter - Flink Twitter connector
- flink-connector-jdbc - Flink JDBC Connector
- flink-connector-redis - Flink Redis connector
Grafana - Data visualization
Metabase - Data visualization
Apache Superset - Data visualization
Tableau - Data visualization
PyTorch - Machine learning framework
- PyTorch Geometric - PyTorch geometric deep learning extension
LangChang - Large language model (LLM) framework
GPT4All - Large language models
OGB - Open graph benchmark
Rasa - Machine learning framework for automated text and voice-based conversations
CML - Continuous machine learning
DVC - Data version control
Feast - Feature store
Kubeflow - Machine learning platform
SkyPilot - Sky computing
MLflow - Machine learning experiment tracking
Weights & Biases - Machine learning experiment tracking
Databricks - Unified data analytics platform

Computer Vision

Open3D - 3D data processing

Cloud Native

Hasura - GraphQL Engine
- hasura-metric-adapter - Hasura GraphQL Engine metric adapter
Ory Hydra - OAuth 2.0 and OpenID Connect server
Terraform - Infrastructure as code (IaC)
TorchServe - PyTorch models serving
Linkerd - Service mesh
Caddy - Web server, reverse proxy, load balancer
Traefik - Web server, reverse proxy, load balancer
nginx - Web server, reverse proxy, load balancer
Open Policy Agent (OPA) - Policy-based control
OPAL - Open-policy administration layer
Elastic APM - Application performance monitoring
OpenTelemetry - Observability framework
Jaeger - Distributed tracing system
Netdata - Distributed monitoring platform
Telegraf - Plugin-driven server agent
Thanos - Highly available Prometheus setup with long term storage capabilities
Vector - Log collector
Fluent Bit - Log collector
Pixie - Observability tool for Kubernetes applications
Docker - Container
Skaffold - Continuous development for Kubernetes applications
Multipass - VM management
Locust - Load testing
Cloudflare Tunnel - Tunneling
Kubernetes - Container-orchestration system
Vertical Pod Autoscaler - Kubernetes vertical pod autoscaler
K3s - Lightweight Kubernetes
containerd - Container runtime

Cloud

Amazon Web Services
- Amazon Athena - Serverless query service
- Amazon CloudTrail - Data governance, data compliance, data auditing
- Amazon EBS - Block storage
- Amazon EC2 - Cloud computing
- Amazon ECR - Container registry
- Amazon EKS - Kubernetes
- Amazon EMR - Big data platform
- Amazon EventBridge - Serverless event bus
- Amazon MSK - Kafka
- Amazon RDS - Relational database service
- Amazon S3 - Object storage
- Amazon SageMaker - Machine learning platform
- Amazon SQS - Queue
- AWS CloudFormation - Infrastructure as code (IaC)
- AWS CodeCommit - Version control
- AWS Glue - Serverless data integration
  - AWS Glue Crawler - Data source discovery
  - AWS Glue Data Catalog - Data catalog
- AWS Glue DataBrew - Data cleaning
- AWS IAM - Identity and access management
- AWS IoT Core - Internet of Things (IoT)
- AWS Secrets Manager - Password management
Google Cloud
- BigQuery - Data warehouse
  - BigQuery ML - BigQuery machine learning
- Dataprep - Data cleaning
- Looker Studio - Data visualization
- Vertex AI - Machine learning platform
Karpenter - Kubernetes node autoscaler
Komiser - Cloud cost monitoring

Ops

Argo CD - Declarative GitOps CD for Kubernetes
Rancher - Kubernetes container management platform
Goldilocks - Kubernetes resource requests recommendation
Polaris - Kubernetes best practices validation
Sloop - Kubernetes history visualization
OpenCost - Kubernetes cost monitoring
Kubecost - Kubernetes cost monitoring
Diun - Container image update notifier
Vagrant - Development environments building and distributing
Ansible - IT automation system
Discord - ChatOps
Opsgenie - Incident management platform
GitHub Actions - Continuous integration

Embedded, IoT, Hardware

VHDL - Very High Speed Integrated Circuits Program (VHSIC) hardware description language
pySerial - Serial communication library
NI VeriStand - Real-time testing and simulation
- niveristand - NI VeriStand API library
- npTDMS - TDMS files reading and writing
- PyVISA - Virtual instrument software architecture (VISA) API library
MATLAB - Programming and numeric computing platform
- Simulink - Simulation and model-based designing
  - Simscape - Model and simulate multidomain physical systems
Yocto Project - Linux distribution creating
ROS - Robot operating system
FreeRTOS - Real-time operating system
RTI Connext - Real-time, distributed systems framework
- RTI Connext DDS - Data distribution service (DDS)
Rclone - Sync program
restic - Backup program
OpenSCAD - 3D CAD Modeller
Arduino Uno - Microcontroller board
BeagleBone Black - Microcontroller board
Raspberry Pi 4 Model B - Single-board computer (SBC)
Jetson Nano - Single-board computer (SBC)
Jetson TX2 - Single-board computer (SBC)
CubeSat - Miniaturized satellite
QGroundControl - Ground control station (GCS) for unmanned aerial vehicles (UAVs)

Ethereum

Solidity - Contract-oriented programming language
solc-js - JavaScript bindings for the Solidity compiler

Code

ansible-lint - Ansible linter
CMakeLint - CMake linter
commitlint - Commit message linter
Stylelint - CSS linter
hadolint - Dockerfile linter
golangci-lint - Go linter
ESLint - JavaScript linter
Ktlint - Kotlin linter and code formatter
Kubeconform - Kubernetes manifest linter
markdownlint-cli2 - Markdown linter
MISS_HIT - MATLAB code formatter
Buf - Protocol Buffers linter
Ruff, autoflake, isort - Python linter
qmllint - QML linter
RuboCop - Ruby linter and code formatter
Scalafmt - Scala code formatter
Scalafix - Scala linter
ShellCheck - Shell linter
solhint - Solidity linter
SQLFluff - SQL linter and code formatter
terraform - Terraform code formatter
VHDL Style Guide (VSG) - VHDL code formatter
yamllint - YAML linter
Prettier - Code formatter
ClangFormat - C/C++ code formatter
gofmt - Go code formatter
Black - Python code formatter
opa - Rego code formatter
@prettier/plugin-xml - XML formatter
Mypy - Python static type checker
tsc - TypeScript static type checker
detekt - Kotlin static type checker
GitHub - Version control
SonarCloud, Codacy, Code Climate - Code reviews and analytics
Codecov - Code coverage reports
Depfu - Dependency monitoring
FOSSA - License compliance

Security

CodeQL - Variant analysis
Prowler - Cloud security assessments
Gitleaks - Git secret scanning
GitGuardian - Git secret scanning

Design

Fusion 360 - Industrial design
Blender - 3D graphic design
Figma - UX design

Physics

SimScale - Computational fluid dynamics (CFD), finite element analysis (FEA), thermal simulation
Qiskit - Quantum computing

Bots

Renovate - Dependency updating
CodeReview BOT - Code reviewing
Mergify - Automatically merging
Stale - Stale issues and pull requests closing
ImgBot - Image compression
semantic-release - Version management and package publishing

Testing

Unit testing
Snapshot testing
Visual testing
Instrumented testing
Smoke testing
Sanity testing
Compatibility testing
Integration testing
End-to-end testing
Contract testing
Mutation testing
Performance testing
Duck testing
Bus testing
Load testing
Durability testing
Fault injection testing
Parallel testing
Acceptance testing
Model-in-the-loop (MIL) testing
Hardware-in-the-loop (HIL) testing
Environmental testing
- Vibration testing
- Shock testing
  - Drop testing
- Temperature testing
- Humidity testing
- Altitude testing
- Icing testing
- Rain testing
- Fungus testing
- Salt fog testing
- Lightning testing
Structural testing
Dyno testing
Wind tunnel testing
Ground testing
Flight testing

Communication Standards and Protocols

User Datagram Protocol (UDP)
Transmission Control Protocol (TCP)
The Internet Protocol (IP)
- IPv4
- IPv6
Hypertext Transfer Protocol (HTTP)
- Hypertext Transfer Protocol Secure (HTTPS)
- HTTP/1.1
- HTTP/2
- HTTP/3
WebSocket
Web Real-Time Communication (WebRTC)
Remote Procedure Call (RPC)
- RPC frameworks
  - Apache Avro
  - Apache Thrift
  - gRPC Remote Procedure Calls (gRPC)
- Binary data serialization formats
  - Avro
  - Thrift
  - Protocol Buffers (Protobuf)
File transfer protocols
- File Transfer Protocol (FTP)
  - Secure File Transfer Protocol (SFTP)
- Server Message Block (SMB)
- Web Distributed Authoring and Versioning (WebDAV)
Email protocols
- Post Office Protocol (POP)
- Simple Mail Transfer Protocol (SMTP)
- Internet Message Access Protocol (IMAP)
Serial protocols
- Inter-Integrated Circuit (I²C)
- Serial Peripheral Interface (SPI)
- Controller Area Network (CAN)
Time protocols
- Network Time Protocol (NTP)
- Precision Time Protocol (PTP)
- IRIG-B Time Protocol
MQTT

Guidelines

Avionics Systems
- DO-178C - Software considerations in airborne systems and equipment certification
- DO-254 - Design assurance guidance for airborne electronic hardware
- ARP4754A - Guidelines for development of civil aircraft and systems
- DO-160G - Environmental conditions and test procedures for airborne equipment
- DO-331 - Model-based development and verification supplement to DO-178C and DO-278A
- DO-330 - Software tool qualification considerations
Drone Systems
- Pixhawk standards - Hardware specifications and guidelines for drone systems development
Automotive Systems
- ISO 26262 - Road vehicles — functional safety
- AUTOSAR - Automotive open system architecture
Industrial Robot Systems
- ISO 10218 - Robots and robotic devices — safety requirements for industrial robots
- ANSI/RIA R15.06-2012 - Industrial robots and robot systems — safety requirements