Nightwatch.js Gotchas

I’ve been using the Nightwatch testing framework for a few months now. I thought I’d write down some of the gotchas that I’ve run into while writing tests for it.

(At the time this post was written, Nightwatch was at v0.9.5.)

Gathering Files Is Cumbersome

Specifying your test files is strangely convoluted. You have to specify a folder or set of folders (src_folders), but you can specify a qualifying file pattern (filter) for inclusion, and/or an array of folders or file patterns to exclude (with exclude). Why 3 different properties for collecting the files? Why not a single glob?

If it’s easier, you can specify the test files on the command line, but note that your src_folders setting is then ignored.

The DSL Is Limited

Using its classic assertion library, the only way to locate elements on a page is through CSS selectors or XPath. The collection of assertions feels a bit thin, and creating assertions outside that set is cumbersome:

// Locating elements deep inside the DOM must follow the CSS/XPath syntax.
// There is no functionality for otherwise delving into an element's children.
browser.assert.elementPresent('#main article:nth-of-type(1) p.byline');

// Nothing like this exists.
browser.assert.elementCount('article', 4);

// Working with values that aren't part of the standard assertions,
//   or using checks other than "equals" requires extra code:
browser.getElementSize('#main', function(result) {
  this.assert.equal(result.value.width, 500);
  this.assert.notEqual(result.value.height, 200);
});

// Checking multiple elements or using different locator strategies
//   requires extra code:
browser.element('link text', 'Submit', function(result) {
  // do something with the resulting element
});
browser.elements('css selector', '#main a', function(results) {
  // do something with the resulting elements
});

The BDD-style library and page objectsmake this a little easier. It’s not horrible, but it would be nice to have the core framework make this easier.

(Also, the less said about the syntax for disabling tests, the better.)

afterEach Hook Has a Different Signature

Be very careful when defining an afterEach hook. Unlike with the other hooks (before, beforeEach, and after), the afterEach function must be defined with either no arguments, or two arguments. Trying to pass a single argument for the browser object will result in cryptic errors:

afterEach: function(browser) {
  // WILL NOT WORK. afterEach must be given zero or 2 arguments.
}

This is mentioned in the docs, but is still easy to forget. It also makes afterEach more annoying to work with if you aren’t performing any asynchronous operations, as you must still use and call a 2nd callback argument.

Assertions in Hooks Are Ignored

Don’t run assertions in before(Each) and after(Each) hooks. They will throw errors and exit the hook, but not be counted in the test results. This is by design. For an analog to before and after hooks, you can add a test at the beginning or end of your suite. If you want to assert something after every test, you have to call the same assertion in every single test.

Some Info Isn’t Available Until After Tests Are Run

…specifically, the Selenium session ID. The sessionId value doesn’t exist in before, beforeEach, or during a test — only in after(Each). It is also set to null when you cleanup your tests with end(). Though by design, this makes it hard if you need it, for example, to update a test running on a cloud service like Sauce Labs.

Async Operations Are Cumbersome

If you need to perform any asynchronous operations (that aren’t built-in), you’ll have to use perform(), which leads to the familiar callback hell. Here’s an example from Łukasz Czyżykowski:

'Test example': function(browser) {
  browser
    .url('https://www.example.com')
    .waitForElementVisible('body', 1000)
    .perform(function(client, done) {
      client.tableContent('table', function(rows) {
        client.assert.equal(rows[0], ['1', 'one']);
        done();
      });
    })
    .end();
}

Failing a Test in a Hook is Broken

The docs state:

Failing the test intentionally in a test hook is achievable by simply calling done with an Error argument

But as far as I can tell, it doesn’t work. Any argument passed to a hook callback is ignored:

afterEach: function(browser, done) {
  // Nightwatch still succeeds with no errors or messages.
  done(new Error('explicit failure'));
}

I have submitted a bug report, but for the time being this makes Nightwatch rather less flexible.

Don’t Use setUp & tearDown

I still see these mentioned (in older posts), but don’t use the setUp and tearDown hooks. They are not included in the docs. The setUp hook is ignored, and I believe tearDown only exists for backward compatibility.

Conclusion

I haven’t been using Nightwatch too long, but at the moment I still prefer Protractor, another end-to-end testing framework that I’ve worked with for the past 2 years.

Even though it was designed for testing Angular sites, Protractor has a better design to me. It’s got a more comprehensive API. Nightwatch is basically a thin layer over the Selenium API and Node’s assertion library. Nightwatch’s async operations rely on callback functions; Protractor leverages ControlFlow and promises to make browser commands and asynchronous operations pretty easy to work with. Some of Nightwatch’s design flaws come in the name of backward compatibility.

Nightwatch is certainly a competent framework for high-level testing. Check it out, but make sure you understand its limitations.