Skip to content

Writing integration tests

Jump to bottom
Jonathan Stray edited this page Apr 4, 2017 · 5 revisions

Yay, you're going to help us test our entire software stack to make sure all components work together? We want this to be fast, but when we have to choose between correctness and speed, we choose correctness.

Integration tests are run from an outsider's perspective. Outsiders see changes to the website -- nothing else. They cannot read any database, logfile, or other internal data. Integration tests can't, either.

Each test is one story, dealing with one feature. Each test is told from the user's perspective. It should be clear why the feature under test provides value to us -- money, fame, and so forth. If it's not clear how the feature gives us value, please comment the integration test to explain it.

Let's run them.

Running Integration tests

Set up the test suite:

  1. Install NodeJS. npm must be in your path.
  2. Run auto/setup-integration-tests.sh from your overview-server checkout. This will run an npm install command.

And now run the integration test suite:

./run # and keep it running in the background
auto/test-integration.sh

What a test looks like

Each test looks like this:

describe 'Login', ->
  ...
  it 'should log in', ->
    @userBrowser
      .tryLogIn(@userEmail, @userEmail)
      .waitForElementByCss('.logged-in')
      .shouldBeLoggedInAs(@userEmail)
      .logOut()

Follow along in test/integration/test/LoginSpec.coffee. A run-down explaining how stuff works:

  • The tests are written in CoffeeScript
  • The test framework is Mocha. In under 50 words: Mocha defines describe, before, after, beforeEach, afterEach, and it. They all share the same context: writing @variable in a before block will let you read it in an it block.
  • Matchers are Chai's BDD matchers. .should is automatically defined on all objects.
  • Tests are asynchronous.

Hold on. Asynchronous? What?

Asynchronicity

Mocha allows asynchronous tests. Here's how they normally work.

Synchronous:

  it 'should test the + operator synchronously', ->
    (1 + 1).should.equal(2)

Asynchronous:

  it 'should test the + operator asynchronously', (done) -> # "done" is a callback
    setTimeout(->
      (1 + 1).should.equal(2)
      done()
    , 10) # delay the function for 10ms

When a Mocha test takes a callback as a parameter, the test won't complete until you call the callback. (No other tests will run, either.) This is essential for testing code that uses the event loop.

But we go one further, for simplicity. We use mocha-as-promised to extend Mocha and chai-as-promised to extend Chai, to give ourselves nicer syntax. With mocha-as-promised, any test that returns a promise will be asynchronous.

Q is one library that returns promises. (Anything with a .then method can be a promise.) The Q folks have written a lovely guide on promises in JavaScript.

Less talk, more code? Okay. Here's some code using mocha-as-promised, chai-as-promised and Q:

  it 'should test + with a promise', -> # notice there's no "done" callback
    deferred = Q.deferred()
    promise = deferred.promise # this has the ".then" method

    setTimeout(->
      deferred.resolve(1 + 1)
    , 10) # delay 10ms

    promise.should.eventually.equal(2) # return another promise

Notice that:

  • The return value is a promise. So mocha-as-promised will ensure the test completes.
  • The returned promise includes a Chai assertion. eventually indicates that the assertion operates on a promise and returns a promise.

Promise chains

Promises do two things:

  1. They hold a value
  2. They implicitly encode a time

You can "chain" promises together to ensure things happen in a certain order. For instance:

  it 'should test + with a promise', -> # notice there's no "done" callback
    deferred = Q.deferred()

    setTimeout(->
      deferred.resolve(1)
    , 10) # delay 10ms

    deferred.promise              # first promise
      .then((value) -> value + 1) # second promise
      .then((value) -> value + 1) # third promise
      .should.eventually.equal(3) # final promise

Don't you like how CoffeeScript indentation helps out with the dots? It's exactly as if the whitespace weren't there.

WebDriver

Let's add a web browser. We use WD.js.

Here's a complete test file:

browser = require('../lib/browser')

describe 'google.com', ->
  before -> # runs once for the entire file
    @browser = browser.create() # a promise

  after -> # runs once for the entire file
    @browser.quit() # another promise

  beforeEach -> # runs before each test
    @browser # a promise!
      .get("https://google.com") # a promise -- resolved on page load

  it 'should have a search field', ->
    @browser # a promise!
      .elementByCss('[name=q]').should.eventually.exist # a promise

  it 'should search without needing to click a button', ->
    @browser # a promise!
      .elementByCss('[name=q]').type('integration testing')
      .waitForElementByXPath("//a[contains(em, 'Integration testing')]")
      .should.eventually.be.fulfilled

There are lots of subtleties:

  • before(), after() and beforeEach() all operate on promises.
  • There are lots of new methods on the @browser promise: elementByCss(), waitForElementByXPath(), get(), and so forth. Here's the wd.js API.
  • The it (test) code operates on the @browser promise. That should raise warning bells: don't we need the promise returned from .get() in beforeEach()? No. We don't need it, because the value of all the promises in the promise chain is identical: each value is a handle on the web browser. We only care about the timing; and we know that the promise in beforeEach() is resolved before the it code begins, so the timing is fine. We could have stored a separate @browser1 promise in beforeEach(), but that would be excessive, since we know that @browser and @browser1 would hold the exact same value when we reach the it code.
  • @browser.quit() is important: it lets future tests re-use the browser.
  • We use PhantomJS as our default browser.

Custom methods

Back to our original example:

    ...
    @userBrowser
      .tryLogIn(@userEmail, @userEmail)
      .waitForElementByCss('.logged-in')
      .shouldBeLoggedInAs(@userEmail)
    ...

Where did .tryLogIn() and .shouldBeLoggedInAs() come from?

We defined them higher up in the file:

testMethods = require('../support/testMethods')

describe 'Login', ->
  testMethods.usingPromiseChainMethods
    tryLogIn: (email, password) ->
      @waitForElementByCss('.session-form')
        .elementByCss('.session-form [name=email]').type(email)
        .elementByCss('.session-form [name=password]').type(password)
        .elementByCss('.session-form [type=submit]').click()

    shouldBeLoggedOut: () ->
      @elementByCss('.session-form')
        .should.eventually.exist

  ... # rest of the test file

testMethods.usingPromiseChainMethods adds some methods to the WD.js promise chain. They are added in a before call and removed in an after call, so you cannot use them in subsequent after blocks.

If your method is going to be useful in the entire unit test suite, consider adding it to lib/browser.coffee.

The test contract in Overview

Our tests must uphold these requirements. This is a reasonable balance between simplicity and speed.

  • Start browsers in before blocks and quit them in after blocks (that is, reuse the same browser for all tests in a file).
  • In general, create objects in before blocks and delete them in after blocks. For instance, if you're testing a DocumentSet, create it once and delete it once. If you're logging in as a user, create the user once and delete it once.
  • You can nest describe blocks. That works very well with before and after.
  • Each block must leave things as it found them. For instance, if you're testing login and each test is meant to start logged out, then make sure each test ends logged out.
  • All browsers must be cleaned, with .deleteAllCookies().quit(). (deleteAllCookies() requires a page to be loaded; by putting it at the end of all test suites instead of the beginning, we save a page load per suite.)

It would be cleaner to use beforeEach and afterEach exclusively. Unfortunately, page loads take hundreds of milliseconds (even seconds, in dev environments, as RequireJS fetches every JavaScript file individually). We should try and avoid unnecessary page requests, when that doesn't make the code too messy.

asUser

Most test pages will start with a browser and a logged-in user. Use support/asUser.coffee to reuse the pattern.

Tips for editing tests

  • If you haven't figured it out yet: tests are in test/integration. Run npm test from that directory to run them.
  • Want to run just one test? While you're editing a test, write describe.only instead of describe or it.only instead of it. That will tell Mocha to ignore all other tests or blocks. (Remember to remove the .only before committing!)
  • Better: run npm install -g mocha and then you can run tests with a plain old mocha. That's useful because you can add --grep: for instance, mocha --grep Login.
  • Your it code should be nothing but promise chains. Use helper methods for anything confusing.
  • To quickly test XPath strings, such as you'd pass to .elementByXPath(...), use Chrome's built-in $x(...) in the JavaScript Console, which works identically.
  • To quickly test CSS selectors, such as you'd pass to .elementByCss(...), use Chrome's built-in $$(...) in the JavaScript Console, which works identically.
  • Beware promises' .then method: it can be handy, but the promises it returns don't have any WebDriver methods.

For instance, this won't work:

  it 'should work', ->
    @browser
      .get("/some/url")
      .then(-> console.log("Got the url"))
      .elementByCss("[name=name]").should.eventually.exist # Promise doesn't have .elementByCss method

.then returns a promise (so if it's at the end of the chain, mocha-as-promised will run as expected). But that promise doesn't have any WebDriver methods: only WebDriver methods return promises that are enhanced with WebDriver methods.

But this will work:

  it 'should work', ->
    @browser1
      .get("/some/url")
      .then(=> @browser2.get("/some/other/url"))
      .elementByCss("[name=name]").should.eventually.exist

That's because the promise returned in the .then call is actually a WebDriver promise. It's a promise returned from @browser2, so .elementByCss will be called on @browser2.

And for debug logging in particular, consider .print(...) instead of .then(-> console.log(...)). .print(...) will output ... followed by the promise's result.

Clone this wiki locally