Mongolian DeadBeef

Mongolian DeadBeef is an awesome Mongo DB node.js driver that attempts to closely approximate the mongodb shell.

Introduction

Mongolian DeadBeef and its documentation is super under construction! Go check out examples/mongolian_trainer.js and the rest of the source!

Unlike other MongoDB node.js drivers, Mongolian DeadBeef is built from the ground up for node.js, using node-buffalo for BSON/message serialization.

v0.1.15 Upgrade notes

0.1.15 uses node-buffalo instead of mongodb-native for serialization, this means a few incompatibilities:

• The helper methods on ObjectId are removed, use the ObjectId constructor to parse hex strings
• Code type is removed, use vanilla function instances instead
• DBRef is not supported
• Error messages may be different

Installation

DISCLAIMER: The API is experimental (but stabilizing). I will be adding, removing, and changing the API in the interest of a solid API. Use at your own risk

You can either clone the source and install with npm link, or install the latest published version from npm with npm install mongolian.

Running Tests

Run the tests with npm test.

Motivation

Not a fan of existing asynchronous mongodb apis for node.js, I set out to write my own. To avoid completely reinventing the wheel, much of the Mongolian DeadBeef API is inspired by the mongodb shell.

High level principles:

• Less is more
  • Nothing is added without careful consideration
  • Remove everything but the essentials
  • Each refactor should remove as much unnecessary lines of code as possible
• Fail early and often
  • If I can easily detect a programmer error, an exception will be thrown

Notes:

• mongodb is pretty simple, much of its functionality is defined as queries on special databases
  • This allows for lots of code reuse
• Avoid callbacks unless they are absolutely necessary

Basics

Most of the work in MongolianDeadBeef doesn't occur until a query is actually made. This means that simple operations are fast and synchronous. Currently there is one connection per server.

Examples

var Mongolian = require("mongolian")

// Create a server instance with default host and port
var server = new Mongolian

// Get database
var db = server.db("awesome_blog")

// Get some collections
var posts = db.collection("posts")
var comments = db.collection("comments")

// Insert some data
posts.insert({
    pageId: "hallo",
    title: "Hallo",
    created: new Date,
    body: "Welcome to my new blog!"
})

// Get a single document
posts.findOne({ pageId: "hallo" }, function(err, post) {
    ...
})

// Document cursors
posts.find().limit(5).sort({ created: 1 }).toArray(function (err, array) {
    // do something with the array
})
posts.find({ title: /^hal/ }).forEach(function (post) {
    // do something with a single post
}, function(err) {
    // handle errors/completion
})

Connections and Authentication

// Create a server with a specific host/port
var server = new Mongolian("mongo.example.com:12345")


// Authenticate a database
db.auth(username, password)


// Supported connection url format: [mongo://][username:password@]hostname[:port][/databasename]
// Use uri-encoding for special characters in the username/password/database name

// Database/auth shorthand (equivalent to calling db() and auth() on the resulting server)
var db = new Mongolian("mongo://username:[email protected]:12345/database")

// Connecting to replicasets:
var server = new Monglian(
    "server1.local",
    "server2.local",
    "server3.local:27018"
)

Logging

By default, Mongolian logs to console.log, but you can override this by specifying your own log object (any object that provides debug, info, warn, and error methods):

var server = new Mongolian({
    log: {
        debug: function(message) { ... },
        info: function(message) { ... },
        warn: function(message) { ... },
        error: function(message) { ... }
    }
})

var server = new Mongolian('server1.local', 'server2.local', {
    log: { ... }
})

BSON Data Types

Mongolian DeadBeef uses node-buffalo's BSON serialization code. Most BSON types map directly to JavaScript types, here are the ones that don't:

var Long =      require('mongolian').Long       // goog.math.Long - http://closure-library.googlecode.com/svn/docs/class_goog_math_Long.html
var ObjectId =  require('mongolian').ObjectId   // new ObjectId(byteBuffer or hexString)
var Timestamp = require('mongolian').Timestamp  // == Long
var DBRef =     require('mongolian').DBRef      // not supported yet

GridFS

The Mongo shell doesn't support gridfs, so Mongolian DeadBeef provides a custom Stream-based GridFS implementation. It consists of two main classes, MongolianGridFS and MongolianGridFile. You can get a MongolianGridFS object from a database with the gridfs([gridfs name]) function.

// Get a GridFS from a database
var gridfs = db.gridfs() // name defaults to 'fs'

// Writing to GridFS consists of creating a GridFS file:
var file = gridfs.create({
    filename:"License",
    contentType:"text/plain"
})
// And getting writable Stream (see http://nodejs.org/docs/v0.4/api/streams.html#writable_Stream )
var stream = file.writeStream()

// You can then pipe a local file to that stream easily with:
fs.createReadStream('LICENSE').pipe(stream)

// Reading a file from GridFS is similar:
gridfs.findOne("License", function (err, file) {
    if (!err && file) {
        // Get the read stream:
        var stream = file.readStream()

        // You could then pipe the file out to a http response, for example:
        stream.pipe(httpResponse)
    }
})

// You can access metadata fields from the file object:
file.length // might be a Long
file.chunkSize
file.md5
file.filename
file.contentType // mime-type
file.uploadDate
// These two are optional and may not be defined:
file.metadata
file.aliases

// If you make any changes, save them:
file.save()

Mongodb Shell Command Support

Nearly all commands are identical in syntax to the mongodb shell. However, asynchronous commands that go to the server will have an optional node.js style callback parameter.

Currently most commands starting with get are named without the get. Some of the getters are implemented as values instead of functions.

• Bold functions are supported
• Italicized functions are supported with different syntax
• Everything else is currently unsupported

There will likely be methods below that are never supported by Mongolian DeadBeef, since I'm targetting a slightly different use case.

Databases

From http://api.mongodb.org/js/1.8.1/symbols/src/shell_db.js.html

• db.addUser(username, password[, readOnly=false][, callback])
• db.auth(username, password)
• db.cloneDatabase(fromhost)
• db.commandHelp(name) returns the help for the command
• db.copyDatabase(fromdb, todb, fromhost)
• db.createCollection(name, { size : ..., capped : ..., max : ... } )
• db.currentOp() displays the current operation in the db
• db.dropDatabase() - see callback note below
• db.eval(func[, arg1, arg2, ...][, callback]) run code server-side - see callback note below
• db.getCollection(cname) implemented as db.collection(cname)
• db.getCollectionNames() implemented as db.collectionNames(callback)
• db.getLastError() - just returns the err msg string
• db.getLastErrorObj() implemented as db.lastError(callback) - return full status object
• db.getMongo() get the server connection object implemented as db.server
• db.getMongo().setSlaveOk() allow this connection to read from the nonmaster member of a replica pair
• db.getName() implemented as db.name
• db.getPrevError() (deprecated?)
• db.getProfilingStatus() - returns if profiling is on and slow threshold
• db.getReplicationInfo()
• db.getSiblingDB(name) get the db at the same server as this one
• db.isMaster() check replica primary status
• db.killOp(opid) kills the current operation in the db
• db.listCommands() lists all the db commands
• db.printCollectionStats()
• db.printReplicationInfo()
• db.printSlaveReplicationInfo()
• db.printShardingStatus()
• db.removeUser(username[, callback]) - see callback note below
• db.repairDatabase()
• db.resetError()
• db.runCommand(cmdObj[, callback]) run a database command. if cmdObj is a string, turns it into { cmdObj : 1 }
• db.serverStatus()
• db.setProfilingLevel(level,) 0=off 1=slow 2=all
• db.shutdownServer()
• db.stats()
• db.version() current version of the server

Collections

From http://api.mongodb.org/js/1.8.1/symbols/src/shell_collection.js.html

• collection.find().help() - show DBCursor help
• collection.count(callback)
• collection.dataSize()
• collection.distinct(key[, query], callback) - eg. collection.distinct( 'x' )
• collection.drop([callback]) drop the collection - see callback note below
• collection.dropIndex(name[, callback]) - see callback note below
• collection.dropIndexes()
• collection.ensureIndex(keypattern[,options][, callback]) - options is an object with these possible fields: name, unique, dropDups - see callback note below
• collection.reIndex()
• collection.find([query],[fields]) - query is an optional query filter. fields is optional set of fields to return. e.g. collection.find( {x:77} , {name:1, x:1} ) - returns a cursor object
• collection.find(...).count()
• collection.find(...).limit(n)
• collection.find(...).skip(n)
• collection.find(...).sort(...)
• collection.findOne([query][callback])
• collection.findAndModify( { update : ... , remove : bool [, query: {}, sort: {}, 'new': false] } ) ex: finds document with comment value 0, increase its 'count' field by 1, and return the updated document. collection.findAndModify( {query: {comment:'0'}, update : {"$inc":{"count":1}}, 'new': true}, function (err, doc) { console.log(doc) })
• collection.getDB() get DB object associated with collection implemented as collection.db
• collection.getIndexes() implemented as collection.indexes(callback)
• collection.group( { key : ..., initial: ..., reduce : ...[, cond: ...] } )
• collection.mapReduce( mapFunction , reduceFunction , [optional params][, callback])
• collection.remove(query[, callback]) - see callback note below
• collection.renameCollection( newName , [dropTarget] ) renames the collection.
• collection.runCommand( name , [options][, callback]) runs a db command with the given name where the first param is the collection name
• collection.save(obj[, callback]) - see callback note below
• collection.stats()
• collection.storageSize() - includes free space allocated to this collection
• collection.totalIndexSize() - size in bytes of all the indexes
• collection.totalSize() - storage allocated for all data and indexes
• collection.update(query, object[, upsert_bool, multi_bool][, callback]) - see callback note below
• collection.validate() - SLOW
• collection.getShardVersion() - only for use with sharding

Cursors

From http://api.mongodb.org/js/1.8.1/symbols/src/shell_query.js.html

• cursor.sort( {...} )
• cursor.limit( n )
• cursor.skip( n )
• cursor.count() - total # of objects matching query, ignores skip,limit
• cursor.size() - total # of objects cursor would return, honors skip,limit
• cursor.explain([verbose])
• cursor.hint(...)
• cursor.showDiskLoc() - adds a $diskLoc field to each returned object
• cursor.toArray(callback) - unique to Mongolian DeadBeef
• cursor.forEach(func, callback) - calls func for each document, and callback upon completion or error
• cursor.print() - output to console in full pretty format
• cursor.map( func ) - map documents before they're returned in next, toArray, forEach
• cursor.hasNext()
• cursor.next([callback]) - returns the next document or null if there are no more

Callbacks

Callbacks take the standard node.js format: function(error, value)

Mongodb handles write operations (insert, update, save, drop, etc.) asynchronously. If you pass a callback into one of these methods, this is equivalent to immediately calling db.lastError(callback) on the same server/connection. Passing a null value will not send a getLastError command to the server.

Currently there is no way to specify the write concern on these inlined callbacks.

Todo

• Connection pooling
• Various utility methods
• More unit tests
• Documentation
• Cleanup

Contributing

Try it out and send me feedback! That's the best help I could use right now. Unit tests are good, too.

License

Mongolian DeadBeef is open source software under the zlib license.