SQLancer (Synthesized Query Lancer) is a tool to automatically test Database Management Systems (DBMS) in order to find logic bugs in their implementation. We refer to logic bugs as those bugs that cause the DBMS to fetch an incorrect result set (e.g., by omitting a record).
SQLancer operates in the following two phases:
sudo apt install mavenon Ubuntu)
The following commands clone SQLancer, create a JAR, and start SQLancer to test SQLite using Non-optimizing Reference Engine Construction (NoREC):
git clone https://github.com/sqlancer/sqlancer cd sqlancer mvn package -DskipTests cd target java -jar sqlancer-*.jar --num-threads 4 sqlite3 --oracle NoREC
If the execution prints progress information every five seconds, then the tool works as expected. Note that SQLancer might find bugs in SQLite. Before reporting these, be sure to check that they can still be reproduced when using the latest development version. The shortcut CTRL+C can be used to terminate SQLancer manually. If SQLancer does not find any bugs, it executes infinitely. The option
--num-tries can be used to control after how many bugs SQLancer terminates. Alternatively, the option
--timeout-seconds can be used to specify the maximum duration that SQLancer is allowed to run.
If you launch SQLancer without parameters, available options and commands are displayed. Note that general options that are supported by all DBMS-testing implementations (e.g.,
--num-threads) need to precede the name of DBMS to be tested (e.g.,
sqlite3). Options that are supported only for specific DBMS (e.g.,
--test-rtree for SQLite3), or options for which each testing implementation provides different values (e.g.
--oracle NoREC) need to go after the DBMS name.
|Pivoted Query Synthesis (PQS)||PQS is the first technique that we designed and implemented. It randomly selects a row, called a pivot row, for which a query is generated that is guaranteed to fetch the row. If the row is not contained in the result set, a bug has been detected. It is fully described here. PQS is the most powerful technique, but also requires more implementation effort than the other two techniques. It is currently unmaintained.|
|Non-optimizing Reference Engine Construction (NoREC)||NoREC aims to find optimization bugs. It is described here. It translates a query that is potentially optimized by the DBMS to one for which hardly any optimizations are applicable, and compares the two result sets. A mismatch between the result sets indicates a bug in the DBMS.|
|Ternary Logic Partitioning (TLP)||TLP partitions a query into three partitioning queries, whose results are composed and compare to the original query's result set. A mismatch in the result sets indicates a bug in the DBMS. In contrast to NoREC and PQS, it can detect bugs in advanced features such as aggregate functions.|
Please find the
.bib entries here.
Since SQL dialects differ widely, each DBMS to be tested requires a separate implementation.
|SQLite||Working||Untyped||This implementation is currently affected by a significant performance regression that still needs to be investigated|
|MySQL||Working||Untyped||Running this implementation likely uncovers additional, unreported bugs.|
|Citus (PostgreSQL Extension)||Working||Typed||This implementation extends the PostgreSQL implementation of SQLancer, and was contributed by the Citus team.|
|MariaDB||Preliminary||Untyped||The implementation of this DBMS is very preliminary, since we stopped extending it after all but one of our bug reports were addressed. Running it likely uncovers additional, unreported bugs.|
|ClickHouse||Preliminary||Untyped, Generic||Implementing the different table engines was not convenient, which is why only a very preliminary implementation exists.|
|TDEngine||Removed||Untyped||We removed the TDEngine implementation since all but one of our bug reports were still unaddressed five months after we reported them.|
|YugabyteDB||Working||Typed (YSQL), Untyped (YCQL)||YSQL implementation based on Postgres code. YCQL implementation is primitive for now and uses Cassandra JDBC driver as a proxy interface.|
|QuestDB||Working||Untyped, Generic||The implementation of QuestDB is still WIP, current version covers very basic data types, operations and SQL keywords.|
SQLancer stores logs in the
target/logs subdirectory. By default, the option
--log-each-select is enabled, which results in every SQL statement that is sent to the DBMS being logged. The corresponding file names are postfixed with
-cur.log. In addition, if SQLancer detects a logic bug, it creates a file with the extension
.log, in which the statements to reproduce the bug are logged.
After finding a bug, it is useful to produce a minimal test case before reporting the bug, to save the DBMS developers' time and effort. For many test cases, C-Reduce does a great job. In addition, we have been working on a SQL-specific reducer, which we plan to release soon.
We would appreciate it if you mention SQLancer when you report bugs found by it. We would also be excited to know if you are using SQLancer to find bugs, or if you have extended it to test another DBMS (also if you do not plan to contribute it to this project). SQLancer has found over 400 bugs in widely-used DBMS, which are listed here.
Official release are available on: