This is a simple benchmark for various asynchronous Python MySQL client libraries.

The client libraries tested here are:

• Twisted's adbapi which uses the MySQL client library written in C.
• the pure Python txMySQL asynchronous client library
• my own adb.py interface using tornado and adisp

The adbapi and txMySQL libraries assume they're running on the Twisted framework. The benchmarks allow you to run different reactor implementations:

• Twisted's default reactor
• the Tornado-based reactor for Twisted

The adb.py library is designed to run directly on top of Tornado and uses adisp.py to make asynchronous programming a lot easier.

Here is how you write asynchronous database code with the adb and adisp modules:

from adisp import process from adb import Database

def __init__(self):
    self.adb = Database(driver="psycopg2",
                        host='DATABASE_HOST',
                        database='DATABASE_DB',
                        user='DATABASE_USER',
                        password='DATABASE_PASSWD',
                        num_threads=3,
                        tx_connection_pool_size=2,
                        queue_timeout=0.001)


@process
def someFunctionInvokedFromIOLoop(self):
    # Drop the table if it exists
    yield self.adb.runOperation("drop table if exists benchmark")
# Create a table to insert the data into
    yield self.adb.runOperation("""
      create table benchmark (
        userid int not null primary key,
        data VARCHAR(100)
      );
    """)
    rows_to_insert = 100000
    # Insert some rows in parallel
    start_time = time.time()
    stmts = []
    for i in xrange(rows_to_insert):
        stmts.append(
            ("insert into benchmark (userid, data) values (%s, %s)",
             (i, i)))
    numrows = yield map(self.adb.runOperation, stmts)
    end_time = time.time()

    rows = yield self.adb.runQuery("select count(*) from benchmark")
    print 'inserted %s records, time taken = %s seconds' % \
        (rows, end_time - start_time)

You can also use transactions:

@process
def transactions(self):
    txId = yield self.adb.beginTransaction()
    yield self.adb.runOperation(
        "insert into mytable (userid, data) values (%s, %s)",
        (1, "test"),
        txId)
    yield self.adb.commitTransaction(txId)

To rollback a transaction, use rollbackTransaction(txId) instead of commitTransaction().

The command line options for the benchmark are the following:

--db The database to use --dbhost Database host --dbpasswd Database user's password --dbuser Database user to use --pool_size Database connection pool size --use_adb Use adb.py database module --use_adbapi Use twisted's adbapi module --use_tornado Use tornado twisted reactor instead of twisted's reactor --use_txmysql Use txMySQL database module. Only works with Twisted and MySQL

Below are some benchmark results. The benchmark program runs on my Hackintosh Core i7-2600K 3.4GHz (4 cores 8 threads) running MacOS X 10.6.7. The database servers were run on the same machine. The benchmark connects to the databases using TCP sockets instead of Unix domain sockets.

The MySQL server version used was 5.1, while Postgres version was 9.0.4.

The pool size specified was --pool_size=10

Here are the times in seconds reported by the program:

MySQL: Tornado Twisted adb 11.79 N/A adbapi 18.99 19.74 txMySQL 45.19 43.80 (numbers from a previous version of this benchmark)

Postgres: Tornado Twisted adb 9.03 N/A adbapi 16.49 17.00

Conclusion

The new adb.py module running on top of Tornado is 40% faster than Twisted's adbapi. As a bonus, in this benchmark the memory usage of adb was 140MB versus the 300MB used by Twisted's adbapi.

When used with adisp, programming asynchronous database operations with adb and Tornado is a real breeze, since you no longer need to worry about creating callbacks to handle the results from the database.

References

The txMySQL implementation tested against is available in the main trunk at:

hybridlogic/txMySQL

The Tornado-based reactor implementation used is available in the main trunk at:

facebook/tornado