Code coverage

If you have written automated tests for computer programs, you are very likely to be familiar with code coverage.  Almost everyone agrees that 100% code coverage from your tests doesn't mean that your code doesn't have any bugs.  I have been told this by experts, and I accepted it as such all this while.  Suddenly a few days back I could come up with an example where the test coverage is indeed 100%, and yet one bug goes unnoticed.

This example is an extremely simple case (you can even call it stupid) and the bug is on your face.  But in real systems bugs may not be so apparent.  Here is the code we are testing:

package in.manki.samples;

public class Coverage {
  public long factorial(int n) {
    long f = 0L;
    if (n > 3) {
      f = 1L;
    }
    for (int i = 1; i <= n; ++i) {
      f *= i;
    }
    return f;
  }
}

And here is the first version of the automated test:

package in.manki.samples;

import junit.framework.TestCase;

public class CoverageTest extends TestCase {
  public void testFactorial() {
    assertEquals(120L, new Coverage().factorial(5));
  }
}

The test passes and Emma says I have 100% code coverage:

[class, %] [method, %] [block, %] [line, %] [name]
100% (2/2) 36%  (12/33)! 17%  (91/549)! 23%  (34/150)! junit.runner
71%  (5/7)! 40%  (41/102)! 35%  (442/1272)! 38%  (123.8/324)! junit.framework
100% (2/2) 56%  (20/36)! 47%  (247/530)! 48%  (63.2/131)! junit.textui
100% (2/2) 100% (4/4) 100% (35/35) 100% (10/10) in.manki.samples

Adding the following test would expose the bug, but if we simply go by coverage numbers we would deceive ourselves that the code is thoroughly tested.

package in.manki.samples;

import junit.framework.TestCase;

public class CoverageTest extends TestCase {
  public void testFactorial() {
    assertEquals(120L, new Coverage().factorial(5));
  }
  public void test2Factorial() {
    assertEquals(2L, new Coverage().factorial(2));
  }
}

Google's Testing Blog has a very good post on code coverage; it's worth reading it if you haven't already.