End-to-End Testing for Web Apps (Meteor / Selenium / WebDriverJS Example)

If you like to just dive in, head over to our GitHub repo and download the boilerplate project, otherwise read on.

We're creating a continuous delivery pipeline for Meteor apps (although  this end-to-end testing approach can be used on any web-app). Our first challenge was doing isolated unit test with Meteor, you can read about that here. The challenge now is being able to do end-to-end testing. The success criteria is ensuring every interaction the user does can be done through tests without having to change the application code to suite the tests.

Our Approach

We like to think of these tests as virtual users, that run through your features to ensure the value has been delivered to the end-user. There is no one right answer when it comes to testing, but this is our right answer :)

As a rule of thumb, we only use two types of tests when doing TDD - Unit and end-to-end. We don't bother with functional/integration/inter-unit test. What this really means is we start with a feature definition, for example "Users can edit their profile", then we define the paths in that feature starting with the happy one(s). For example:

Feature: A user can modify their email address 

Happy Path: User [clicks the edit email icon] then [types their email address] then [clicks update] then [sees the updated email]. 

Alternate Path: User [clicks the edit email icon] then [types their email address] then [presses enter] then [sees the updated email]. 

Negative Path: User [clicks the edit email icon] then [types an invalid email address] then [presses enter] then [sees an error message].

Next we take those steps and code them in an end-to-end test:

    userClicksTheEditEmailIcon().
        then(typesTheirEmailAddress).
        then(clicksUpdate).
        then(seesTheUpdatedEmail);

    userClicksTheEditEmailIcon().
        then(typesTheirEmailAddress).
        then(pressesEnter).
        then(seesTheUpdatedEmail);

    userClicksTheEditEmailIcon().
        then(typesAnInvalidEmailAddress).
        then(clicksUpdate).
        then(seesAnErrorMessage);

This is real WebDriverJS code btw, and we'll get to the details shortly, but let's assume for now that the above code tests what it says. When we run it without any markup or code it'll fail obviously which is the TDD way. Now we can write the HTML code to satisfy the test and get as far as clickEditEmailIcon, but nothing would happen when it's clicked. Some people would argue that you need to write a unit test at this point, something like "clicking the edit email button changes the label into a field", but if you look at the end-to-end test you can see the DOM manipulations will be covered, so to avoid duplication and thus additional maintenance we would only write a unit test for the email address validation in the case above, where the permutations of emails would be tested. All the other interactions will otherwise be covered by the end-to-end test. Here's a sample unit test if anyone's curious what the difference is:

describe("Email validation helpers", function () {
    it("should only have 1 and only 1 @ sign", function() {
        expect(Helpers.validateEmail('@@tomexample.com').toEqual(false);
        expect(Helpers.validateEmail('tomexample.com@@')).toEqual(false);
        expect(Helpers.validateEmail('tom@@example.com')).toEqual(false);
        expect(Helpers.validateEmail('@tomexample.com@')).toEqual(false);
        expect(Helpers.validateEmail('tomexample.com')).toEqual(false);
        expect(Helpers.validateEmail('tom@example.com')).toEqual(true);
    });
    it("there should always be a dot after the @", function() {
        ...
    });
});

When thinking from a features point of view it's easy to run down the happy path and miss out the details, but here are a set of guidelines that have served us well in the past when it comes to thinking about good UI-driven test coverage. There are main four categories to think about: 

  • Rendering - Ensures all the UI elements have been rendered correctly. We don't always have to explicitly write tests for these as by using the elements through interactions and navigations we often get assertions of their existence for free, but not always.
  • Messaging - The end-to-end tests should exercise every message that can come out of the system once and only once. So if there is a message that says "you have entered an invalid email address" and another that says "your email address is already in use", we'd have UI tests that show both of those, and the unit tests would do all the permutations. Key here is thinking from the user's point of view and ensuring they can see the messages from the system bubble up to them.
  • Interaction - As we're moving towards single-page apps, more and more of these tests are required. An example is: "when the user keyboard focus is away from the email field, it will turn red", and more complex things like "When an icon is dragged and dropped over the trash, it will get deleted"
  • Navigation - Popular in apps with a site-map. Once we've taken care of all the above page-level tests, these tests ensure the site navigation is wired up correctly. For instance: "after the user registers, selects the package they want, then proceeds to the check out and pays, they receive a confirmation email"

A few tips and warnings 

  • Tests that rely on DOM structure can be brittle to markup changes, so it's usually better to ensure presence and not location of element within the DOM. I'm personally a fan of finding elements by what the user would call them by using the name field: "sign up button" for instance, because the test can say "when the user clicks the sign up button".
  • Like any code base, apply the DRY principle to the test codebase. The return on investment will be easier test writing and higher velocity as the interactions library gets bigger.
  • Make sure every feature test you write is independent and does not rely on the results of a previous run. This allows you to scale your testing further down the line as your test cases increase and so does your build time. The best way to do this is to think in these terms: Setup, Execute, Verify, where setup creates all the conditions your feature needs (see fixtures below), executing is doing the actions the user would take, and verify is as it says.
  • Avoid using the UI to setup the state needed for a test, instead use fixtures to speed up your testing. For instance, if you have a feature that expects the user to be authenticated and their profile to be filled out, create a fixture that allows you to setup that state quickly and then you just execute the action you're interested in testing for that feature, and verify. You can read more about fixtures below.

To make the case of avoiding duplication a little stronger, if you run a coverage tool like the excellent istanbul and combine the coverage report from both the unit test runs the end-to-end tests, you'll see it's possible to get 100% test coverage from this approach. This istanbul reports combining is something we're working on and is already in place for unit tests in the boilerplate project, but I digress!

Implementation Details

Let's dive into the some implementation goodness. If you're not familiar with WebDriverJS it's advisable to read the guide. You install it like this using NPM:

    npm install -g selenium-webdriver

You'll need to get the latest selenium-server-standalone-x.xx.x.jar. Now if you're brave you can use GhostDriver instead of selenium-server, however we had various intermittent issues so would recommend against it until it gets more stable. Now you can start selenium-server like this:

    java -jar ./selenium-server-standalone-x.xx.x.jar

The server is now waiting for WebDriver commands on port 4444, and we can leave it running in the background. You'll need to make sure this is running whenever you're running units tests or you'll get "connection error".

The example we're using here is the leaderboard application that comes with Meteor. If you look at the packages file and the template file you'll notice there's a login with the accounts-password added. This is to exemplify how to login using WebdriverJS, which is as follows:

var authenticate = function () {
var email = 'some@one.com';
var password = 'test1234';
var deferred = webdriver.promise.defer();
driver.findElement(webdriver.By.id('login-sign-in-link')).click();
driver.findElement(webdriver.By.id('login-email')).sendKeys(email);
driver.findElement(webdriver.By.id('login-password')).sendKeys(password);
driver.findElement(webdriver.By.id('signup-link')).click();
driver.findElement(webdriver.By.id('login-buttons-password')).click();
driver.findElement(webdriver.By.id('login-name-link')).getText()
.then(function (value) {
if (value.indexOf(email) !== 0) {
deferred.rejected(value + ' did not contain ' + email);
} else {
deferred.resolve();
}
});
return deferred.promise;
};

As you can see, WebDriverJS uses promises and deferred objects to chain commands (see the API guide). The idea is simple: Create a deferred object and return its promise, then the object will be resolved or rejected at a later time. This means when we want to use the code above, we can do this:

    authenticate().then(/* chain methods or more promises here */)

For most simple cases using this chaining is great, however sometimes more complex things are needed. For instance, in the leaderboard app, we wanted to find the player by name. This proved quite cumbersome due to asynchronous promises. Consider this code:

var flow = webdriver.promise.controlFlow();
var findPlayerByName = function (name) {
return function () {

var mainDefer = webdriver.promise.defer();

// (1) Find all the players
driver.findElements(webdriver.By.className('player')).then(function (players) {

players.map(function (player) {
// (2) Get each player's name...
player.findElement(webdriver.By.className('name')).then(function (playerName) {
// (3) asynchronously...
playerName.getText().then(function (value) {
// and when found, return it
if (value === name) {
mainDefer.resolve(player);
}
});
});
// (4) ERROR: This is not the last player so any pending in-flight
// asynchronous requests will return an error as they'll be out of context
});

});
return mainDefer.promise;
};
};

So here's how we made them a little more synchronous using flows:

// (1) Start by grabbing the controlFlow object
var flow = webdriver.promise.controlFlow();
var findPlayerByName = function (name) {
return function () { // NOTE 1 - See below

var mainDefer = webdriver.promise.defer();

driver.findElements(webdriver.By.className('player')).then(function (players) {

// (2) Create another deferred object for the match
var matchDefer = webdriver.promise.defer(),
matchPromise = matchDefer.promise,
resolved; // (3) We'll come back to this one below

players.map(function (player) {
player.findElement(webdriver.By.className('name')).then(function (playerName) {
// (4) Make sure every getText gets executed serially using the flow object
flow.execute(function () {
playerName.getText().then(function (value) {
if (value === name) {
// (5) We'll come back to this below, promise ;)
resolved = true;
matchDefer.resolve(player);
}
});
});
});
});

// (6) After getText() has been called for all the players
flow.execute(function () {
// (7) And _IF_ the match promise has been fulfilled
matchPromise.then(function (player) {
// (8) Then resolve the main deferred object
mainDefer.resolve(player);
});
// (8) Since there's no guarantee that a player is found, this line fails
// Jasmine if the player's not found
expect(resolved).toBe(true);
});

});
// return the main promise to allow chaining
return mainDefer.promise;
};
};

You'll notice this method returns a function as opposed to a promise. This is because findPlayerByName takes a parameter. If it didn't (like most other methods), you don't need to wrap this. I'd love for someone to show a better way, but this seems to do the trick.

We're also using fixtures in this app. The way we've done this in the past is using RESTFUL endpoints, and exposing calls like GET to /reset and /setupPlayers. So we did exactly that with Meteor, but since we don't want to pollute the main code with fixture code, we did this by having a separate fixture.js file living under the test codebase, and then we copy this file into a mirrored meteor app when running tests using grunt (on every save!). Here's the fixture code:

// (1) A method to allow to create routes in Meteor's middleware
var createRoute = function(route, handler) {
__meteor_bootstrap__.app.stack.splice(0, 0, {
route: '/' + route,
handle: function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
handler(req, res);
res.end(route + ' complete');
}.future()
});
};

// (2) provide the routes on startup
Meteor.startup(function () {

// (3) One for reseting
createRoute('reset', function() {
Meteor.users.remove({});
Players.remove({});
});

// (4) And another for setting up the players in a predictable order
// Since the example creates a difficult-to-test random set!
createRoute('setupPlayers', function() {
var names = ["Ada Lovelace",
"Grace Hopper",
"Marie Curie",
"Carl Friedrich Gauss",
"Nikola Tesla",
"Claude Shannon"];
for (var i = 0; i < names.length; i += 1) {
Players.insert({name: names[i], score: i * 10});
}
});

});

With the above pieces, we can now put it all together in a Jasmine test as follows:

describe("Leaderboard functionality", function () {

beforeEach(function () {
// (1) Create a new browser session for each aspect of this feature.
// This can probably be optimized to keep the same browser
driver = require('./drivers/selenium-server.js')(webdriver, {
// (2) Choose your browser. You can use safari/firefox/opera/IE/Android...
browserName: 'chrome'
});

// (3) Set webdriver defaults
driver.manage().timeouts().setScriptTimeout(5000);
driver.manage().timeouts().implicitlyWait(5000);

// (4) Before each test, kill the db, create expected player data and open the app
resetApp().
then(setupPlayers).
then(openApp);
});

// (5) Now test the feature
it("increases a players score by 5 when the increment button is clicked", function (done) {
// (6) Look at how sexy this English-like notation is!
authenticate().
then(findPlayerByName('Grace Hopper')).
then(verifyTheirScoreIs10).
then(selectPlayer).
then(giveThemFivePoints).
then(findPlayerByName('Grace Hopper')).
then(verifyTheirScoreIs15).
then(finish(done), error);
});

// (7) Write more tests to cover the paths through this feature
// ...
});

The full test class can be seen here.

And there you have it. We feel this is now a good enough starting point to embark on our new Meteor venture as we've got an approach to do TDD. As always, any feedback is welcome so let us know your thoughts.

:)