SQLite on the Web: absurd-sql

Love it or hate it, the capabilities of your modern web browser continuously grow in strange and wild ways. The ability for web apps to work offline requires a persistent local storage solution and for many, IndexedDB is the only choice as it works across most browsers and provides a database-like interface. However, as [James Long] found, IndexedDB is painfully slow on chrome and limited in querying ability. He set out to bring a tool he was familiar with, SQLite, and bring it to the web browser as absurd-sql.

Why absurd? Partially because most browsers (not chrome) implement IndexedDB on top of SQLite. So for many browsers, it is just SQLite on top of IndexedDB on top of SQLite. Luckily for [James] there already was a project known as sql.js that uses emscripten to compile the C-based SQLite into WebAssembly. However, sql.js uses an in-memory storage backing and all data is lost when refreshing the page. [James] tweaked SQLite’s method of reading and writing blocks. Instead of being memory backed, he added a layer to read and write blocks from IndexedDB. This means that only sections of the database need to be read in, bringing in huge performance gains.

That brings us to the other reason why it’s absurd. On chrome (as well as Firefox), absurd-sql beats IndexedDB on almost every benchmark. A query like SELECT SUM(*) FROM kv led to stunning results.

So what’s the downside? Other than a somewhat large WebAssembly file that needs to get downloaded (409KB) and cached, there really isn’t. Of course, it’s not all roses when it comes to web development. Native SQLite runs 2-3 times faster than absurd-sql, which demonstrates how slow IndexedDB really is.

There are other storage standards on the horizon for web browsers, but locking becomes an issue. SQLite expects synchronous reads and writes because it’s just simple C. IndexedDB and other storage solutions are asynchronous as the event loop of Javascript lends itself well to that model. Absurd-sql gets around that by creating a SharedArrayBuffer that is shared with a worker process. The atomics API is used to communicate with the buffer. In particular, atomics.wait() allows the worker to block main thread execution until the read or write has finished. From the perspective of SQLite, the operations are synchronous. IndexedDB provides transactions so multiple connections can happen (for example multiple tabs open). Multiple readonly transactions can occur in parallel but only one readwrite transaction can be in flight.

Why not pull up your browser and start playing around with it? You’re already doomed to learn WebAssembly anyway.