README
NestDB
The Node.js Embedded JavaScript Database
Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, written in 100% JavaScript, no binary dependency. Originally forked from NeDB. API is similar to MongoDB's and it's plenty fast.
IMPORTANT NOTE: Please don't submit issues for questions regarding your code. Only actual bugs or feature requests will be answered, all others will be closed without comment. Also, please follow the bug reporting guidelines and check the change log before submitting an already fixed bug :)
Installation
Node.js version via NPM
$ npm install nestdb --save
Node.js version via Yarn
$ yarn add nestdb
Browser version via Bower
$ bower install nestdb --save # Main files under "./browser-version/out/"
Compatibility with NeDB
NestDB was originally forked from NeDB.
NestDB v2.x
will maintain backward compatibility with NeDB v1.x
to make for seamless transitions migrating from NeDB to NestDB.
When we eventually release NestDB v3.x
, it will not be backward compatible with NeDB v1.x
.
API
It is similar to MongoDB's API (the most used operations). The NestDB API is a superset of NeDB's API.
- Creating/loading a datastore
- Persistence
- Inserting documents
- Finding documents
- Counting documents
- Updating documents
- Removing documents
- Indexing
- Destroying a datastore
- Extending with plugins
- Using a custom storage engine
Creating/loading a datastore
You can use NestDB as an in-memory only datastore or as a persistent datastore. One datastore is the equivalent of a MongoDB collection. The constructor is used as follows new Datastore(options)
where options
is an object with the following fields:
filename
(optional): path to the file where the data is persisted. If left blank, the datastore is automatically considered in-memory only. It cannot end with a~
which is used in the temporary files NestDB uses to perform crash-safe writes.inMemoryOnly
(optional, defaults tofalse
): as the name implies.timestampData
(optional, defaults tofalse
): timestamp the insertion and last update of all documents, with the fieldscreatedAt
andupdatedAt
. User-specified values override automatic generation, usually useful for testing.autoload
(optional, defaults tofalse
): if used, the datastore will automatically be loaded from the datafile upon creation (you don't need to callload
). Any command issued before load is finished is buffered and will be executed when load is done.onload
(optional): if you use autoloading, this is the handler called after theload
. It takes oneerror
argument. If you use autoloading without specifying this handler, and an error happens during load, an error will be thrown.idGenerator
(optional): if set, this function will be used for generating IDs. It takes no arguments and should return a unique string.afterSerialization
(optional): hook you can use to transform data after it was serialized and before it is written to disk. Can be used for example to encrypt data before writing datastore to disk. This function takes a string as parameter (one line of an NestDB data file) and outputs the transformed string, which must absolutely not contain a\n
character (or data will be lost).beforeDeserialization
(optional): inverse ofafterSerialization
. Make sure to include both and not just one or you risk data loss. For the same reason, make sure both functions are inverses of one another. Some failsafe mechanisms are in place to prevent data loss if you misuse the serialization hooks: NestDB checks that never one is declared without the other, and checks that they are reverse of one another by testing on random strings of various lengths. In addition, if too much data is detected as corrupt, NestDB will refuse to start as it could mean you're not using the deserialization hook corresponding to the serialization hook used before (see below).corruptAlertThreshold
(optional): between 0 (0%) and 1 (100%), defaults to 0.1 (10%). NestDB will refuse to start if more than this percentage of the datafile is corrupt. 0 means you don't tolerate any corruption, 1 means you don't care.compareStrings
(optional):function compareStrings(a, b)
should compare stringsa
andb
and must return-1
,0
or1
. If specified, it overrides default string comparison (===
), which is not well adapted to non-US characters such as accented or diacritical letters. Using the nativeString.prototype.localeCompare
will be the right choice most of the time.storage
(optional): A custom storage engine for the database files. Must implement at least the handful of methods exported by the standard "storage" module included in NestDB, as detailed below in the Using a custom storage engine section.
If you use a persistent datastore without the autoload
option, you need to call load
manually.
This function fetches the data from datafile and prepares the datastore. Do NOT forget it! If you use a
persistent datastore, no command (e.g. insert
, find
, update
, remove
) will be executed before load
is called, so make sure to either call it yourself or use the autoload
option.
Also, if load
fails, all commands registered to the executor afterwards will not be executed. They will be registered and executed, in sequence, only after a successful load
.
Once the datastore is fully loaded, it also emits a "loaded"
event that you can add a listener for.
Whenever any datastore is fully loaded, the Datastore
object itself emits a global "created"
event that you can add a listener for.
Type 1: In-memory only datastore (no need to load)
var Datastore = require('nestdb')
, db = new Datastore();
// You can issue commands right away
Type 2: Persistent datastore with manual loading
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile' });
db.load(function (err) { // Callback is optional
// Now commands will be executed
});
Type 3: Persistent datastore with automatic loading
var Datastore = require('nestdb')
, db = new Datastore({
filename: 'path/to/datafile'
, autoload: true
, onload: function (err) {
if (err) {
console.error('Failed to load the datastore:', err);
} else {
console.log('Loaded the datastore!');
}
}
});
// You can issue commands right away because of NestDB's internal queueing
// You can also synchronously add an event listener for the 'loaded' event
// If not added synchronously, you will probably miss the event
// This event will not be emitted if an error occurs during loading
db.once('loaded', function () {
console.log('Loaded the datastore!');
});
Type 4: Persistent datastore for a Node WebKit app
// For a Node WebKit app called 'nwtest'
// For example on Linux, the datafile will be ~/.config/nwtest/nestdb-data/something.db
var Datastore = require('nestdb')
, path = require('path')
, db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'something.db') });
Loading a datastore with an event listener
var Datastore = require('nestdb')
, db = new Datastore({ filename: 'path/to/datafile', autoload: true });
// You can issue commands right away because of NestDB's internal queueing
// You can also synchronously add an event listener for the 'loaded' event
// If not added synchronously, you will probably miss the event
// This event will not be emitted if an error occurs during loading
db.once('loaded', function () {
console.log('Loaded the datastore!');
});
Listening for the global "created" event
var Datastore = require('nestdb')
, db;
Datastore.on('created', function (dbRef) {
// dbRef === db
console.log('Created or loaded a datastore: ' + (dbRef.filename || 'in-memory'));
});
db = new Datastore({ filename: 'path/to/datafile', autoload: true });
Loading multiple datastores
// Of course you can create multiple datastores if you need several collections.
// In this case, it is best to use autoload for all -- unlike this example!
db = {};
db.users = new Datastore('path/to/users.db');
db.robots = new Datastore('path/to/robots.db');
// You need to load each datastore (here we do it manually)
db.users.load();
db.robots.load();
Persistence
Under the hood, NestDB's persistence uses an append-only format, meaning that all updates and deletes actually result in lines added at the end of the datafile, for performance reasons. The datastore is automatically compacted (i.e. put back in the one-line-per-document format) every time you load each datastore within your application.
You can manually call the compaction function with yourDatastore.persistence.compactDatafile(cb)
, where callback is optional and get passed an error if any. It queues a compaction of the datafile in the executor, to be executed sequentially after all pending operations. The datastore will also fire a "compacted"
event whenever a compaction process is finished.
You can also set automatic compaction at regular intervals with yourDatastore.persistence.setAutocompactionInterval(interval)
, interval
in milliseconds (a minimum of 5 seconds is enforced), and stop automatic compaction with yourDatastore.persistence.stopAutocompaction()
.
Keep in mind that compaction takes a bit of time (not too much: 130ms for 50k records on a typical development machine) and no other operation can happen when it does, so most projects actually don't need to use it.
Compaction will also immediately remove any documents whose data line has become corrupted, assuming that the total percentage of all corrupted documents in that datastore still falls below the specified corruptAlertThreshold
option's value.
Durability works similarly to major databases: compaction forces the OS to physically flush data to disk, while appends to the data file do not (the OS is responsible for flushing the data). That guarantees that a server crash can never cause complete data loss, while preserving performance. The worst that can happen is a crash between two syncs, causing a loss of all data between the two syncs. Usually syncs are 30 seconds appart so that's at most 30 seconds of data. This post by Antirez on Redis persistence explains this in more details, NestDB being very close to Redis AOF persistence with appendfsync
option set to no
.
Inserting documents
The supported native types are String
, Number
, Boolean
, Date
and null
. You can also use arrays and subdocuments (objects). If a field is undefined
, it will not be saved (this is different from MongoDB which transforms undefined
in null
, something I find counter-intuitive).
If the document does not contain an _id
field, NestDB will automatically generate one for you (a 16-character alphanumerical string). The _id
of a document, once set, cannot be modified.
Field names cannot begin by '