If you try to fail, and succeed, which have you done?


From T.D.D. training two weeks ago I learned that failing is just as important as passing. More specifically certain tests are expected to fail and you need to verify that they do. It’s a little counter intuitive but think about it. The first kind of failing test that deserves our attention is the initial test. The first time you write a test it should fail and you should verify that it does. That ensures that you are actually adding the intended behavior when you finally do get a green bar.

Another type of important failure is the resilience test. These are test that check your code’s resilience to bad input. Lets just pretend that you have this:

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      hiro.telelportTo(new Planet(Planet.MARS));
   }
}

Here we are exercising Hiro’s ability to teleport to other planets which we would expect to fail. (It should fail because there’s no oxygen on Mars which would in effect kill our Hiro or Hero, if you will.) We want our Hiro to throw an Exception on inter-planetary travel… something like InterplanetaryException. This is a situation where we try to fail. If you run the test as is and it green-bars what have we done? Have we succeeded or have we failed? It’s a really mind-twisting bit of code the first time you write such a test because you have to think opposite then opposite of that opposite then rationalize the intent of double negatives and all.

So how do we assert an Exception in JUnit? (It turns out that the folks at CxxUnit have done a much better job at the task.) First you have to execute the code that you’d expect to throw the Exception as above then you have check that the exception is thrown. The easy way to do that is to put another line of code after the line you’re testing, an “if-we-get-here” line. Because if we get that far then the Exception hasn’t happened which means we’ve failed.

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      hiro.telelportTo(new Planet(Planet.MARS));
      fail("Hiro shouldn't teleport across planets!");
   }
}

We run the above and get our red bar which tells us to change/add the logic necessary to trap for bad input. Assuming the createTeleporter() method just returns “new HiroNakamura()” we’d do something like this.

class HiroNakamura {
   public void teleportTo(Object location) {
      if(location instanceOf Planet) 
      { throw new InterplanetaryException(); //extends RuntimeException }
      //...
   }
}

But wait! That causes our test to error because an exception is thrown. You have to catch the exception in your test so that the test framework doesn’t catch it and abend. We do something like this:

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      try {
         hiro.telelportTo(new Planet(Planet.MARS));
         fail("Hiro shouldn't teleport across planets!");
      } catch(Exception e) {
         //we expect an exception which means we're good
      }
   }
}

Now we’re in big trouble. It is so typical to make the above mistake then live with the consequences days or months later. (If you know where the problem is keep quiet so the people in the back can concentrate.) What happens if later on somebody removes the guard clause in HiroNakamura? Or what happens when we create a new Teleporter implementation (PeterPatrelli) and attempt to reuse the above test logic? Let’s consider the first case where somebody thinks that the Planet object would be necessary for qualifying a location. In other words we want to be able to teleport to “Planet earth = new Planet(Planet.EARTH)” after we’ve set a country on it like, “earth.setCountry(“Japan”);”. So we remove the check at the beginning of Hiro’s teleportTo method and the test passes! That’s no good because now we can also send Hiro to Pluto and freeze him to death. The problem comes from the fact that we set out to fail and succeeded in failing which gave us a false sense of success. (That makes sense, right?) Instead should we have set out to succeed in failing and properly asserted our failure for success? Let’s look at our test.

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      try {
         hiro.telelportTo(new Planet(Planet.MARS));
         fail("Hiro shouldn't teleport across planets!");
      } catch(Exception e) {
         //we expect an exception which means we're good
      }
   }
}

Why is it no longer failing when the Exception doesn’t happen? What’s really going on? Well we teleport and then we run the “we-shouldn’t-get-this-far” line and that should fail the test right? Well it would fail the test because it throws an AssertionFailedError which we blindly catch as Exception and silence with our naive comment that states stoopidly that we expect an Exception. (Never ever EVER catch or expect Exception!) Instead we should have caught the right type of exception which would not only self document the intent of the desired behavior but would have kept us safe from future modifications.

class TeleporterTest extends TestCase {
   public void testTeleport() throws Exception {
      Teleporter hiro = createTeleporter();
      try {
         hiro.telelportTo(new Planet(Planet.MARS));
         fail("Hiro shouldn't teleport across planets!");
      } catch(InterplanetaryException e) {
         //we expect an InterplanetaryException which means we're good
      }
   }
}

So now I ask the above question. If you try to fail, and succeed, which have you done?

4 thoughts on “If you try to fail, and succeed, which have you done?

  1. You probably know this but in junit 4 you could write your test like this, which would fail if the exception isn’t thrown. I believe there is a way in junit3, using a exception suite or something from the extensions package to do something similar.

    class TeleporterTest {

    @Test(expected=InterplanetaryException.class)
    public void testTeleport() throws Exception {
    Teleporter hiro = createTeleporter();
    hiro.telelportTo(new Planet(Planet.MARS));
    }

    Great blog btw.

  2. Yeah I knew that JUnit4 and even TestNG have a nicer way to handle the issue. Even CxxTest has exception assertions. The point of the post is to stress an important and often overlooked step in TDD… the red bar. I also wanted to stress the importance of the concepts of making sure the code you write both in the test and in the tested object is necessary and correct. Thanx for the compliments y’all.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s