Measure actual coverage, not "cell coverage"

[ceball]

Current cell coverage (num cells tested)/(num cells) is a proxy for actual coverage.

nbval includes a coverage measure - should investigate that.

Current coverage is relatively easy to fake, e.g.

if False:
    %cell

Something related to "what actually ran?" (e.g. detection that cell source was submitted to kernel) might be better than a static check.

[timkpaine]

@vidartf is this something you can help with? I see you've done it for nbval computationalmodelling/nbval#134

since we have the %cell injection, I would think all we need to do is maintain a map of nb line # to test line #, inject coverage.py into the test itself, then match back the covered line using the mapping.

[vidartf]

@timkpaine The nbval coverage is all about getting coverage data in python modules, from the execution of the notebook. It doesn't currently attempt to measure coverage in the notebook. The problem is that coverage will not be available in the same scope as the coverage gathering was started, i.e. the notebook global scope. There might be a clever work-around somehow, but I haven't dived deep enough into it to find anything.