Write your own JUnit runner

I've been looking at how to integrate the JUnit test running model with a custom test framework.
This is a handy thing to have, because it allows your custom test framework to be used in all existing JUnit runners - and gives you IDE integration for free.

The RunWith annotation allows you to specify a custom runner for your class.
Extend the JUnit Runner class and over-ride the functions neccessary.

@RunWith(MyTargetTestClass.TheRunner.class)
public class MyTargetTestClass {
 static int count = 0;
 public boolean doStuff() {
   try {
     Thread.sleep(1000);
   } catch (InterruptedException e) {}
  
   count ++;
  
   if(count % 2 == 0) { throw new RuntimeException("A Failure"); }
  
   return !(count % 3 == 0);
 }

 public static class TheRunner extends Runner {
   List descriptions = new ArrayList();
   private final Class<? extends MyTargetTestClass> testClass;
   private final MyTargetTestClass testContainingInstance;
   private Description testSuiteDescription;
  
   public TheRunner(Class<? extends MyTargetTestClass> testClass) { 
     this.testClass = testClass;
     testContainingInstance = reflectMeATestContainingInstance(testClass);
     testSuiteDescription = Description.createSuiteDescription("All my stuff is happening now dudes");
     testSuiteDescription.addChild(createTestDescription("first bit happening"));
     testSuiteDescription.addChild(createTestDescription("second bit happening"));
     testSuiteDescription.addChild(createTestDescription("third bit happening"));
   }


   @Override
   public Description getDescription() {
     return testSuiteDescription;
   }

   @Override
   public void run(RunNotifier notifier) {
     for(Description description : testSuiteDescription.getChildren()) {
       notifier.fireTestStarted(description);
       try {
         if(testContainingInstance.doStuff()) {
           notifier.fireTestFinished(description);
         }
         else {
           notifier.fireTestIgnored(description);
         }
       } catch (Exception e) {
         notifier.fireTestFailure(new Failure(description, e));
       }
     }
    
   }

   private MyTargetTestClass reflectMeATestContainingInstance(Class<? extends MyTargetTestClass> testClass) {
     try {
       return testClass.newInstance();
     } catch (Exception e) {
       throw new RuntimeException(e);
     }
   }

   private Description createTestDescription(String description) {
     return Description.createTestDescription(testClass, description);
   }
  
 }
}

It seems that the ability to write custom runners was added pretty late on, and as such the RunNotifier class has some methods that are indicated as being internal use only. Just make sure you don't call them and you'll be fine!

Using Junit 4 to run tests repeatedly

We had a problem with a test that would fail intermittently on continuous integration.
The best way to find the problem was by running the test hundreds of times to see the problem.
We used the code below to easily do this:

@RunWith(MyRunner.class)
public class UnusualAndRareProblemTest {
...
..
.
}

public static class MyRunner extends JUnit4ClassRunner {

  public MyRunner(Class klass) throws InitializationError {
    super(klass); 
  }

  @Override
  public void run(final RunNotifier notifier) {
    for(int i=0; i<1000; i++) {
      super.run(notifier);
    }
  }
}

It turns out the class under test was spawning threads and we were getting race/dead-locking conditions due to the way we were trying to retrieve the outcomes from different threads.
PS. We fixed this by re-writing the class so that instead of spawning threads directly it made calls to a 'action factory' that we could mock and avoid multi-threading in the test entirely.