Jia Huang

Using Istanbul for code coverage

Published on
Last updated on

I’ve been using Istanbul, a JS code coverage library. Istanbul outputs a report that shows which lines are hit during unit tests:

istanbul report

Istanbul can be installed with:

1
npm install -g istanbul

How Istanbul works

Istanbul wraps all your code with functions that track how often each function, line, and branch in the code got hit while running unit tests. It stores this data in a coverage.json file which can then be used to generate html, lcov, or other kinds of reports.

The process of wrapping code with tracking functions is called “instrumentation”. Istanbul relies on the esprima package to parse JS source code. Then escodegen is used to add in the coverage code and to output a new instrumented source file.

Given a source file person.js that looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function Person(name, dob) {
  this.name = name;
  this.dob = dob;
}

Person.prototype.isMinor = function () {
  var currDate = new Date();
  if ((currDate.getTime() - this.dob.getTime())/(1000 * 60 * 60 * 24 * 365.25) < 18.0) {
    return true
  } else {
    return false
  }
}

module.exports = Person;

Istanbul can create the instrumented code via: istanbul instrument --no-compact -o tmp/instrument/person.js person.js. This creates a instrumented person.js file in tmp/instrument/person.js.

The tmp/instrument/person.js file looks like this. I added some comments to detail what is happening:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
// a temp __cov_X9N6pGXtRbutT8pzNmQHeA obj is created
var __cov_X9N6pGXtRbutT8pzNmQHeA = (Function('return this'))();

// there's a __coverage__ obj that is used to track coverage. The name of this object can change depending on settings, but __coverage is the default.
if (!__cov_X9N6pGXtRbutT8pzNmQHeA.__coverage__) { __cov_X9N6pGXtRbutT8pzNmQHeA.__coverage__ = {}; }
__cov_X9N6pGXtRbutT8pzNmQHeA = __cov_X9N6pGXtRbutT8pzNmQHeA.__coverage__;
if (!(__cov_X9N6pGXtRbutT8pzNmQHeA['/Users/jia/code/jiahuang.github.io/person.js'])) {

  // the initialized __coverage__ obj. This already has details for all of the main source code (such as line numbers, functions, and branches that have been tested)
   __cov_X9N6pGXtRbutT8pzNmQHeA['/Users/jia/code/jiahuang.github.io/person.js'] = {"path":"/Users/jia/code/jiahuang.github.io/person.js","s":{"1":1,"2":0,"3":0,"4":0,"5":0,"6":0,"7":0,"8":0},"b":{"1":[0,0]},"f":{"1":0,"2":0},"fnMap":{"1":{"name":"Person","line":1,"loc":{"start":{"line":1,"column":0},"end":{"line":1,"column":27}}},"2":{"name":"(anonymous_2)","line":6,"loc":{"start":{"line":6,"column":27},"end":{"line":6,"column":39}}}},"statementMap":{"1":{"start":{"line":1,"column":0},"end":{"line":4,"column":1}},"2":{"start":{"line":2,"column":2},"end":{"line":2,"column":19}},"3":{"start":{"line":3,"column":2},"end":{"line":3,"column":17}},"4":{"start":{"line":6,"column":0},"end":{"line":13,"column":1}},"5":{"start":{"line":7,"column":2},"end":{"line":7,"column":28}},"6":{"start":{"line":8,"column":2},"end":{"line":12,"column":3}},"7":{"start":{"line":9,"column":4},"end":{"line":9,"column":15}},"8":{"start":{"line":11,"column":4},"end":{"line":11,"column":16}}},"branchMap":{"1":{"line":8,"type":"if","locations":[{"start":{"line":8,"column":2},"end":{"line":8,"column":2}},{"start":{"line":8,"column":2},"end":{"line":8,"column":2}}]}}};
}

__cov_X9N6pGXtRbutT8pzNmQHeA = __cov_X9N6pGXtRbutT8pzNmQHeA['/Users/jia/code/jiahuang.github.io/person.js'];
function Person(name, dob) {
    // when this function is called, the number of times this function is called gets incremented
    __cov_X9N6pGXtRbutT8pzNmQHeA.f['1']++;
    // and the times this line is called also gets incremented
    __cov_X9N6pGXtRbutT8pzNmQHeA.s['2']++;
    this.name = name;
    __cov_X9N6pGXtRbutT8pzNmQHeA.s['3']++;
    this.dob = dob;
}
__cov_X9N6pGXtRbutT8pzNmQHeA.s['4']++;
Person.prototype.isMinor = function () {
    // similarly these also increment the number of times this function & lines are called
    __cov_X9N6pGXtRbutT8pzNmQHeA.f['2']++;
    __cov_X9N6pGXtRbutT8pzNmQHeA.s['5']++;
    var currDate = new Date();
    __cov_X9N6pGXtRbutT8pzNmQHeA.s['6']++;
    if ((currDate.getTime() - this.dob.getTime()) / (1000 * 60 * 60 * 24 * 365.25) < 18) {
        // because this is in an "if" statement, the number of branches called also gets incremented
        __cov_X9N6pGXtRbutT8pzNmQHeA.b['1'][0]++;
        __cov_X9N6pGXtRbutT8pzNmQHeA.s['7']++;
        return true;
    } else {
        __cov_X9N6pGXtRbutT8pzNmQHeA.b['1'][1]++;
        __cov_X9N6pGXtRbutT8pzNmQHeA.s['8']++;
        return false;
    }
};

So as the instrumented code is executed, a tmp object stores all the information about what lines have been called for a particular file. Note that the istanbul instrument command is only useful for browser based tests. I included the instrumented code here just to illustrate how Istanbul keeps track of coverage. For tests that can run on node, the regular istanbul cover command automatically instruments the code.

Istanbul for Node tests

If we had a jasmine test file person.spec.js like this:

1
2
3
4
5
6
7
8
var Person = require('./person');

describe("A person", function() {
  it("can be a minor", function() {
    var p = new Person("me", new Date("2000-03-06"));
    expect(p.isMinor()).toBe(true);
  });
});

This can be run with jasmine person.spec.js:

1
2
3
4
5
Started
.

1 spec, 0 failures
Finished in 0.005 seconds

Now to get the coverage of this file person.spec.js run istanbul cover jasmine person.spec.js to get the following output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Started
.

1 spec, 0 failures
Finished in 0.005 seconds

=============================================================================
Writing coverage object [/Users/jia/code/jiahuang.github.io/coverage/coverage.json]
Writing coverage reports at [/Users/jia/code/jiahuang.github.io/coverage]
=============================================================================

=============================== Coverage summary ===============================
Statements   : 92.86% ( 13/14 )
Branches     : 50% ( 1/2 )
Functions    : 100% ( 4/4 )
Lines        : 92.86% ( 13/14 )
================================================================================

The report is generated at coverage/lcov-report.index.html:

person report

Note that the statement % coverage is very high (88%). This is misleading because the tests only cover half of the branches (50%). Be sure to check coverage % across statements, branches, and functions.

Istanbul for browser tests

The previous example showed how to run Istanbul on node code. Running Istanbul on the browser requires a little more setup. For this example I’ll use the following:

There are a few steps to get Istanbul tests on the browser:

  1. Manually instrument the source code via istanbul instrument
  2. Make the specs use the instrumented source code instead of the actual code
  3. Get the coverage data from window.__coverage__ after all tests have been run
  4. Send the data from window.__coverage__ to a server so that the coverage data can be saved
  5. Run istanbul report to generate a report out of the coverage data

The full repo of a working demo is here: https://github.com/jiahuang/istanbul-phantom-jasmine

NYC

NYC is the newer Istanbul CLI that works with react & ES6. The problem with getting code coverage for ES6 is that only the babel-transformed version of the code actually runs. So a code coverage tool has to un-babelify the code that was run to get the original source code via a source map. Istanbul cannot currently do this, but NYC can.

One of the drawbacks of Istanbul is that it only checks files that have been required. There’s an open issue to fix this, but for now a workable solution seems to be just require all files. Being able to see code coverage makes it easier to discover fragile pieces of untested code, and write tests to cover edge cases.