To view as a website, see: http://zhangyushao.site/awesome-database-testing/.
Awesome-Database-Testing
This is a curitated list of resources on database testing.
What covered: papers, blogs, notes, tools and etc. for whoever wants to test a database mangement system (DBMS).
What is not: using SQLs, learning database basic knowledge, implementing a database system.
Note: this is NOT an exhaustive list of materials, you may want to find more accroding to references of each item.
High-level Views
Here we put materials with general discussions.
Research Papers
NOTE: We only include peer-reviewed and published papers here, the same also applies to other entries. You may want to find the authors' free-version of the papers on their personal page.
-
Understanding the query optimization Query Optimization in Database Systems [1984]
-
Understanding the query optimization - a more recent view An Overview of Query Optimization [1998]
-
What are the problems from the indutrial view Testing SQL Server's Query Optimizer : Challenges , Techniques and Experiences
-
Optimizer Evaluation How good are query optimizers, really?
Others
- What are the problems from the academic view Is Query Optimization a “Solved” Problem?
Input Generation
Research Papers
- Query generation with template substitution Generating Thousand Benchmark Queries in Seconds
- Language for customized data generator Flexible Database Generators
- RAGS Microsoft SQL Server Massive Stochastic Testing of SQL
- Generate data regarding the query constraints QAGen: Generating Query-Aware Test Databases
- How could symbolic execution help Qex: Symbolic SQL Query Explorer
- Data generation as a search problem Search-based test data generation for SQL queries
- For more valid inputs! SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback [CCS 2021]
Tools
- Randgen MySQL version (not maintained) doc, MariaDB version, PingCAP version
- pquery - stress testing MySQL
- Sysbench - MySQL benchmarks
- SQLsmith - fuzzing style tool
- Squirrel - tool for CCS 2021 paper
- mtr_to_sql.sh@Percona-QA - a script to extract mysql-test cases as a sql file, with engine replacement and shuffles according to your need
Engine Correctness
Here we mainly refer to the case where the engine returns wrong query result.
Research Papers
What You Really Want to Know About
- RAGS with system-differential testing Massive Stochastic Testing of SQL
- Data generation with contraints sovling Query-Aware Test Generation Using a Relational Constraint Solver
- Differential testing for the optimizing rules A framework for testing query transformation rules
- Criteria on integrtity testing The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints
What You Want to Know About
- Parallelism on the old machines Quickly generating billion-record synthetic databases
- Verified database system Toward a Verified Relational Database Management System
- Reply the transactions Debugging transactions and tracking their provenance with reenactment
What's New
- Search-based test data generation for SQL queries [2018]
- Automated verification of query equivalence using satisfiability modulo theories [2019]
- Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction [2020]
- Testing query execution engines with mutations [2020]
Tools
- SQLancer@ETH_ZURICH
- go-sqlancer@PingCAP
- pquery@Percona-QA - a tool providing the ability to combine correctness and crash recovery testing on both single-node and clustered instances
Performance
Here we focus on detecting and debugging performance issues.
Research Papers
What You Really Want to Know About
- Very first system Efficient testing of high performance transaction processing systems
- Is the exact estimation possible Exact Cardinality Query Optimization for Optimizer Testing
- How to identify issues on plan generation On the stability of plan costs and the costs of plan stability
- What's wrong with the models Testing cardinality estimation models in SQL server
What You Want to Know About
- Generate and compare the plans Testing the accuracy of query optimizers
What's New
Tools
- TPC Benchmarking
- APOLLO@gatech
Industry Practice
We list idea and tools which are adopted in the industry.
Tools
- Chaos Mesh from PingCAP
Contributors
Contributions are Super Welcome!
What you can contribute:
- papers (only peer-reviewed papers will be accepted to this repo)
- talks/slides/articles
- tools
Please also add yourself to Contributors in the PR.