If you try to fail, and succeed, which have you done?


From T.D.D. training two weeks ago I learned that failing is just as important as passing. More specifically certain tests are expected to fail and you need to verify that they do. It’s a little counter intuitive but think about it. The first kind of failing test that deserves our attention is the initial test. The first time you write a test it should fail and you should verify that it does. That ensures that you are actually adding the intended behavior when you finally do get a green bar.

Another type of important failure is the resilience test. These are test that check your code’s resilience to bad input. Lets just pretend that you have this:

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      hiro.telelportTo(new Planet(Planet.MARS));
   }
}

Here we are exercising Hiro’s ability to teleport to other planets which we would expect to fail. (It should fail because there’s no oxygen on Mars which would in effect kill our Hiro or Hero, if you will.) We want our Hiro to throw an Exception on inter-planetary travel… something like InterplanetaryException. This is a situation where we try to fail. If you run the test as is and it green-bars what have we done? Have we succeeded or have we failed? It’s a really mind-twisting bit of code the first time you write such a test because you have to think opposite then opposite of that opposite then rationalize the intent of double negatives and all.

So how do we assert an Exception in JUnit? (It turns out that the folks at CxxUnit have done a much better job at the task.) First you have to execute the code that you’d expect to throw the Exception as above then you have check that the exception is thrown. The easy way to do that is to put another line of code after the line you’re testing, an “if-we-get-here” line. Because if we get that far then the Exception hasn’t happened which means we’ve failed.

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      hiro.telelportTo(new Planet(Planet.MARS));
      fail("Hiro shouldn't teleport across planets!");
   }
}

We run the above and get our red bar which tells us to change/add the logic necessary to trap for bad input. Assuming the createTeleporter() method just returns “new HiroNakamura()” we’d do something like this.

class HiroNakamura {
   public void teleportTo(Object location) {
      if(location instanceOf Planet) 
      { throw new InterplanetaryException(); //extends RuntimeException }
      //...
   }
}

But wait! That causes our test to error because an exception is thrown. You have to catch the exception in your test so that the test framework doesn’t catch it and abend. We do something like this:

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      try {
         hiro.telelportTo(new Planet(Planet.MARS));
         fail("Hiro shouldn't teleport across planets!");
      } catch(Exception e) {
         //we expect an exception which means we're good
      }
   }
}

Now we’re in big trouble. It is so typical to make the above mistake then live with the consequences days or months later. (If you know where the problem is keep quiet so the people in the back can concentrate.) What happens if later on somebody removes the guard clause in HiroNakamura? Or what happens when we create a new Teleporter implementation (PeterPatrelli) and attempt to reuse the above test logic? Let’s consider the first case where somebody thinks that the Planet object would be necessary for qualifying a location. In other words we want to be able to teleport to “Planet earth = new Planet(Planet.EARTH)” after we’ve set a country on it like, “earth.setCountry(“Japan”);”. So we remove the check at the beginning of Hiro’s teleportTo method and the test passes! That’s no good because now we can also send Hiro to Pluto and freeze him to death. The problem comes from the fact that we set out to fail and succeeded in failing which gave us a false sense of success. (That makes sense, right?) Instead should we have set out to succeed in failing and properly asserted our failure for success? Let’s look at our test.

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      try {
         hiro.telelportTo(new Planet(Planet.MARS));
         fail("Hiro shouldn't teleport across planets!");
      } catch(Exception e) {
         //we expect an exception which means we're good
      }
   }
}

Why is it no longer failing when the Exception doesn’t happen? What’s really going on? Well we teleport and then we run the “we-shouldn’t-get-this-far” line and that should fail the test right? Well it would fail the test because it throws an AssertionFailedError which we blindly catch as Exception and silence with our naive comment that states stoopidly that we expect an Exception. (Never ever EVER catch or expect Exception!) Instead we should have caught the right type of exception which would not only self document the intent of the desired behavior but would have kept us safe from future modifications.

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      try {
         hiro.telelportTo(new Planet(Planet.MARS));
         fail("Hiro shouldn't teleport across planets!");
      } catch(InterplanetaryException e) {
         //we expect an InterplanetaryException which means we're good
      }
   }
}

So now I ask the above question. If you try to fail, and succeed, which have you done?

Ten T.D.D. Commandments


Update* A while ago I thought it was cool to use the file system during a test but a little bunny taught me otherwise. I’ve removed a redundant commandment as well.

Inspired by Ashcroft… not that Ashcroft but this one.

Ten Commandments of Unit Tests

1. I am the class being tested. Thou shalt not test any other class but me.
2. Thou shalt NOT write implementation without a unit test.
3. Thou shalt ensure the bar is red before any code is written.
4. Thou shall NOT write two tests which depend upon each other
5. Thou shall NOT test private methods.
6. Thou shall NOT access files during unit tests.
7. Thou shalt always refactor both test and implementation code after every green bar.
8. Thou shall NOT worship (or test) any false classes (mocks).
9. Thou shall NOT define more than one behavior in a unit test.
10. Thou shall NOT commit code without green bars on all tests.