Monk and the command buffering pattern

In this article we will show the command buffering pattern and how the open source library Monk makes use of it. With a comparison between MongoDB driver and Monk, we will see how this pattern improves developer experience.

Command buffering pattern

The command buffering pattern buffer operations. What this means is that operations are not executed immediately. They get queued. At a later time, the operations contained in this queue gets executed.

This pattern has shown to be useful in many open source libraries such us Monk or Mongoose.

Monk library

Monk is a tiny wrapper on top of the official MongoDB driver. The benefits it offers over the official MongoDB driver are:

  • friendlier API
  • promise based API
  • command buffering until connection to database is established

Some of you might have heard of Mongoose. Mongoose offers the buffering command feature too. The reason why we chose Monk is because is a small library and code can be read within minutes.

Command buffering pattern: MongoDB driver vs Monk

To understand when to use the command buffering pattern and the problem that it solves, let’s compare how the MongoDB driver library works without this pattern and how Monk uses it to offer a friendlier API.

Here we show how to insert a document using MongoDB driver API:

// Example using MongoDB driver library
const MongoClient = require('mongodb').MongoClient;

// Use connect method to connect to the server
MongoClient.connect(url, function(err, client) {
  const db = client.db(dbName);

  const collection = db.collection('articles')

  collection.insert({
    title: "Hello World"
    ...
    tags: ['article']
    }, (err, result) => {

      // Close connection
      client.close();

    }
  );
});
// End of MongoDB driver library

As we see, MongoClient invokes the connect method. Using the MongoDB driver library, we call the connect method and no operations can be done until the connection to mongodb instance is established.

Once the connection to MongoDB instance is established, the callback gets executed.

Let’s show the same operation using the Monk library:

// Example using Monk library
const db = require('monk')('localhost/mydb')
const collection = db.get('articles')

collection.insert({
      title: "Hello World"
      ...
      tags: ['article']
    })
    .then( () => {

      // Close connection
      db.close()

    })

Monk is a wrapper over the MongoDB driver API. Using the Monk library we do not call the connect method. Despite of that, Monk allows you to get access to the collection and insert documents even before the connection has been established.

Open QA

You: Okay ,I got that Monk is a wrapper library that uses the MongoDB driver. MongoDB driver was using a connect method to estabalish the connection to my mongodb instance. When is Monk establishing the connection to my mongo instance? I have not called the connect method at all.

Me: The connection starts being established since the moment you passed the host and the database name to Monk.

// Example using Monk library
const db = require('monk')('localhost/mydb')

You: I have another question. Monk is not offering hooks on any connect event. Despite of that, I can start querying my mongodb instance. How can Monk guantee me that when I start querying collections and inserting data, the connection has been established?

Me: Good question. Monk is making the illusion that the connection has been established and the resources are there to be consume. BUT as we said, it’s a mere illusion. This trick is accomplished by buffering commands.

Commands are piled up into a queue structure. Once the connection is established, the queue with your commands are executed.

You: Sweet!

Monk command buffering pattern implementation

The code is easy to follow and small enough to read in couple of minutes. You can take a look at the source code.

For the lazy ones I have collected the pieces that implements the command buffering pattern from Monk (I also omit code non-related to it).

If you follow the comments in the code, you should not have any issue understanding this pattern.


/* *********************************************
 *
 *  STEP 0) STATE STATES YOUR SYSTEM CAN BE IN
 ********************************************** */
var STATE = {
  CLOSED: 'closed',
  OPENING: 'opening',
  OPEN: 'open'
}

/**
 * Monk constructor.
 *
 * @param {Array|String} uri replica sets can be an array or
 * comma-separated
 * @param {Object|Function} opts or connect callback
 * @param {Function} fn connect callback
 * @return {Promise} resolve when the connection is opened
 */

function Manager (uri, opts, fn) {

  /*
   * OMITTED CODE
   */

  this._state = STATE.OPENING

  /* *********************************************
   *
   *  STEP 1) CREATE QUEUE STRUCTURE
   *
   *  This queue will keep **resolve** function from different Promises.
   *
   ********************************************** */
  this._queue = []

  /* *********************************************
   *
   *  STEP 2) REGISTER LISTENER TO *OPEN* DB CONNECTION HOOK
   *
   *  Whenever the connection to the database has been established,
   *    this hook is executed. 
   *
   *  Promises get resolved.
   *
   ********************************************* */
  this.on('open', function (db) {
    monkDebug('connection opened')
    monkDebug('emptying queries queue (%s to go)', this._queue.length)
    this._queue.forEach(function (cb) {
      cb(db)
    })
  }.bind(this))


  /*
   * OMITTED CODE
   */

  this.open = this.open.bind(this)
  this.close = this.close.bind(this)
  this.executeWhenOpened = this.executeWhenOpened.bind(this)

  /*
   * OMITTED CODE
   */
}

/**
 * Execute when connection opened.
 * @private
 */

/* *********************************************
 *
 *  STEP 3) METHOD TO CONTROL COMMAND EXECUTION
 *
 *  When the connection to the database has 
 *   not been established (STATE.OPENING or STATE.CLOSED),
 *   
 *  Monk returns a promise. This promise is pushed to the queue.
 *   This is KEY. This *resolve* method functions will get call
 *   whenever the connection to the db has been established.
 *
 *  If the connection has already been established, Monk returns
 *  a resolved promise.
 * 
 ********************************************* */

Manager.prototype.executeWhenOpened = function () {
  switch (this._state) {
    case STATE.OPEN:
      return Promise.resolve(this._db)
    case STATE.OPENING:
      return new Promise(function (resolve) {
        this._queue.push(resolve)
      }.bind(this))
    case STATE.CLOSED:
    default:
      return new Promise(function (resolve) {
        this._queue.push(resolve)
        this.open(this._connectionURI, this._connectionOptions)
      }.bind(this))
  }
}

/**
 * Create a collection.
 *
 * @param {String} name - name of the mongo collection
 * @param {Object} [creationOptions] - options used when creating the collection
 * @param {Object} [options] - options to pass to the collection
 * @return {Collection} collection to query against
 */

/* *********************************************
 *
 *  STEP 4) PREPEND ALL COMMANDS WITH executeWhenOpened METHOD
 *
 *  Whenever we call any method in Monk, it will execute 
 *   the **executeWhenOpened** method.
 *
 *  At some point later in time (check STEP 2) the promises will get resolved.
 *    Once that happen the **createCollection** is executed.
 *
 *  This behaviour is extended to any other command Monk exposes
 * 
 ********************************************* */
Manager.prototype.create = function (name, creationOptions, options) {
  this.executeWhenOpened().then(function (db) {
    db.createCollection(name, creationOptions)
  }).catch(function (err) {
    this.emit('error', err)
  })

  if ((options || {}).cache === false || !this.collections[name]) {
    this.collections[name] = new Collection(this, name, options)
  }

  return this.collections[name]
}

Conclusion

The command buffering pattern is a well known pattern used in many open source projects. Just to name a few: Monk and Mongoose.

Front-end engineers make use of this pattern when they want to hide when a system has established connection or delay operations until a flag is raised without making clients wait for it.

This is one of those patterns that should be in all front-end engineers toolbox. Next time you need to delay operations without throwing errors or polling, give a try to this pattern.

See you in the next click.