todb

An in process node.js document store. This is a work in progress.

This project originally started as an experiment to create a binary dependency free index for the search-index search engine and kind of grew, there are other databases out there written in node but they either have binary dependencies, require the keys to be in memory or the entire database. The goal of todb is to have a reasonably performant database and storage engine written in pure javascript and stores indexes on disk.

Why todb? Well if you squint just right, and have had a few drinks, then it kind of looks like todo - as in I'm doing this whenever the mood strikes me and it's likely always going to have things todo. Not to be confused with toodb which is a similar database that I'm writing in rust.

It is modeled loosely off of LevelDB.

Writes are guaranteed so in that way it's kind of ACID compliant.

Data is stored in an append only table format. Indexes are stored in index files as inverted indexes, these are SSTables.

ToDB has support for secondary indexes and basic querying on secondary indexes.

It's node6 only right now. I'll re-add grunt and babel when this latest rebuild is finished, it's just not a priority for me atm.

Performance:

The current version has indexes in memory and content on disk so it's quite a bit faster than the final version which will have most indexes on disk (with some in memory):

On a modern SSD I get the following results:

-In 3 seconds it can write ~37,000 records. -In 3 seconds it can read ~124,000 records.

Currently a write operation requires two write operations on disk, and a get requires just one read. When indexes are persisted on disk a get will require two read events, the performance of write and read will therefore be similar.

Examples:

Using callbacks:

//creates a new database directory called testDB with ap primary key of email
new DB( 'testDB' , ( err , db ) => {
    //creates a new table called people with a primary key of email, at this stage there is no support for autogenerated PK's like an incremented id, or a uuid.
    db.table("people" , { id : 'email' } ,( err , table) => {
         //we've just created secondary indexes for the name field.
        table.createIndex("name" , ( err ) => {
            //create a new record
            table.put( { email : 'sarah@madeupcompany.com' , name : 'Sarah Smith' , age : 34 } , ( err ) => {
                table.where( 'name' , 'Sarah Smith' , ( err , data ) => {
                    console.log( data );
                } );
             } );
        } );
    });
} );

Using promises:

//creates a new database directory called testDB with a primary key of email
new DB( 'testDB' ).then( db => db.table( 'people' , { id : 'email' }  ) )
    .then( table => table.createIndex( 'name' ) )
    .then( table => table.put(  { email : 'sarah@madeupcompany.com' , name : 'Sarah Smith' , age : 34 } ) )
    .then( table => table.where(  'name' , 'Sarah Smith' ) )
    .then( results => console.log(  results ) );

Using the Query interface (coming at some point)

Future options:

//I'd like to use features like proxies and decorators so you could have something like:
@persistent
class Person {
    constructor( email , name , age ) {
        @primaryKey
        this.email = email;

        @indexed
        this.name = name;

        this.age = age;

        @foreignRelationship(Person)
        this.friends = [ ];
    }
}
//And treat the object like you would any other in JS, the proxy would mean that the underlying dataset would update with every change.