The Seven Testing Modes of Meteor

Did you know there are 7 unique modalities to testing Meteor code? And when fully denormalized, there are actually 12 modes plus 4 additional pseudo-unit types (if you really want to get fancy!). Find out what they all are and which of the frameworks currently support them.

Meteor is a special framework that provides everything you need to write a web application under one umbrella, from the database, to the server and to the client. It's easy to forget sometimes how much auto-wiring is happening and how easy that makes our lives as coders.

All this auto-wiring becomes very apparent when we want to test our Meteor applications. One of the key concepts in testing is isolating the system under test or the SUT. An SUT can be a unit, an integration of units, or the whole end-to-end app. Focussing the SUT to a specific level requires support from testing frameworks. In writing this book, I've come to understand the intricacies Meteor has at the various levels and I was surprised to see so many! My starting point was to understand the different workflows that developers use when writing Meteor code.

Some Meteor developers write all their code in the main app. Others write all their code in packages and have very little (if any) code in the main app. And there are those that are somewhere between the aforementioned. From a testing standpoint, this creates two distinct modes of testing: app-level testing and package-level testing. Let's see what type of tests exist under each level through some examples.

App-Level Testing

App-Level End-to-End

Test Mode #1

When the SUT focus is end-to-end, it's trivial to write a test. You fire up your app and poke the UI with something like Zombie or Selenium. Consider this snip of code that does exactly that: (source)

webdriverio
  .remote(options)
  .init()
  .url('http://localhost:3000')
    .title(function(err, res) {
    // assert on the title
  })
  .end();

Test Mode #2

If you're testing the API and not the UI, then you can write a test in any framework that has a DDP client like this: (source)

ddpclient.call('deletePosts',  ['foo', 'bar'], 
  function (err, result) {
    // assert on the result
  });

App-Level Client Integration

Test Mode #3

Dropping the SUT a level very slightly complicates things. Consider this code-snippet from a Mocha integration test that runs on the client: (source)

describe("Player Ordering", function(){
  it("should result in with the first player having more points than the second", function(){
    var players = Template.leaderboard.players().fetch();
    chai.assert(players[0].score >= players[1].score);
  });
});

The slight complication is in having to know how Template.leaderboard.players works and what fields exist within the documents. This is of course trivial if you're coding the app yourself and you've used Meteor for more than 3 days.

App-Level Server Integration

Test Mode #4

The server side test is exactly the same, and here's an example of some server-side test code for completeness: (source)

describe("Server initialization", function(){
  it("should insert players into the database after server start", function(){
    chai.assert(Players.find().count() > 0);
  });
}); 

App-Level Client Unit

Test Mode #5

Let's change the SUT focus to be the unit level. Now things get more complicated! Let's dive right into some code to see how and why: (source)

describe('rewardPlayer', function () {
  it('should add 5 points to the player score with the given id', function () {
    var playerId = 1;
    spyOn(Players, 'update');
    PlayersService.rewardPlayer(playerId);
    expect(Players.update.calls.argsFor(0)).toEqual([playerId, {$inc: {score: 5}}]);
  });
});

You'll notice there's a PlayersService that's not in the Meteor-provided leaderboard example. Jonas (the author of the Jasmine framework) knows that unit tests drive the design of your code. And if you want to be able to do unit-testing easily, you will need to structure your code that allows you to test it.

You can also see the use of a Spy in the code above. This is a test-double that allows the test code to control the dependencies of the SUT. Since the SUT we're interested here is at Unit level, it means all dependencies need to be controlled through the use of test-doubles.

App-Level Server Unit

Test Mode #6

Client unit tests have the same level of complexity as the server tests above, and here's an example again for completeness: (source)

describe('playersExist', function () {
  it('should return false when no players exist', function () {
    var cursor = {
      count: function () {
      return 0;
    }
  };
  spyOn(Players, 'find').and.returnValue(cursor);
    expect(PlayersService.playersExist()).toBe(false);
  });
});

Package-Level Testing

Packages provide snips of functionality and when you look at the way packages are structured, you can see they are like a Meteor application. The same 5 levels of testing that apply to app-level testing also apply to the package-level. The difference is in how you isolate the code and how you run the tests.

Package-Level End-to-End

It shouldn't come as a surprise that this is the easiest to do.

An option I came up with is to first create a simple application that configures and uses your package in way the package is intended to be used. For instance, consider this very blog you are reading. It's driven by the xolvio:md-blog package. To make the end-to-end blog package the SUT, you can create an app and add this package to it then write a test that pokes the app.

Test Mode #7

A package-level UI test would look something like this:

webdriverio
  .url('http://localhost:3000/blog')
  .click('#mdblog-new')
  .click('#mdblog-publish')
  .getTitle(function(err, title) {
    expect.title.toEqual('New Blog Post');
  })
  .end();

The cut-down webdriver code navigates to the blog page, clicks the new-blog button, then publishes it with the default content which should results in a blog post titled "New Blog Post".

Test Mode #8

If you wanted to write a DDP test for the package, you would use the DDP client just like #2 above.

Package-Level Client / Server Integration

Test Mode #9 and #10

At the time of writing this blog, the only way to test packages is using the MDG-provided tool TinyTest. What TinyTest does behind the scenes is copy the package code into the app-level and allow you to run tests against the package code. It effectively unwraps the package. The Velocity team are working to enable this unwrapping for other frameworks and a little birdie tells me respond.ly almost have a working solution.

Package-Level Client / Server Unit

Test Mode #11 and #12

TinyTest doesn't support these modes and the Velocity team is working on supporting these also. In the meantime, you can employ in-context unit testing techniques within the TinyTest tests that you can read about below.

In-Context Unit Testing

Test Mode #13, #14, #15, #16

These tests can be applied to both app-level and package-level and to both server and client unit testing. They are called in-context because they don't run in a truly isolated instance, they are actually integration tests that rely on test doubles (spies, mocks, fakes and stubs) to perform isolation.

Using test-doubles you can get an integration test to run like a unit test. When the Velocity team were experimenting with unit testing, we were able to temporarily remove dependencies from within a Meteor app, and then replace them when we finished testing. If you're very careful you can achieve unit-level testing in this way, but if you're not careful, one test can affect another and this is not good news since you don't want false negatives from test runs.

So Why 7? 

In-context testing, it's not really an option in my book (excuse the pun). They are just integration tests in disguise and I did mention that the pseudo testing was getting fancy! So lets remove these 4 modes and bring the total number of valid modes to 12.

As for package level testing, when seen from a different light, package-level testing is more of a build-step than it is a testing mode. Once the package is unwrapped, the same test modes that work in app-level testing work on the package code. So we can remove 6 modes from the list but let's add a new mode called package testing. Now we have a list that is very good for evaluating testing frameworks for Meteor!

Testing Mode Jasmine Mocha Web Cucumber Nightwatch CasperJS Robot
Server Unit - - - -
Client Unit - - - -
Server Integration - - - -
Client Integration - - -
End-to-end UI
End-to-end DDP - - -
Package Testing - - - -