Testing multithreaded code

Java threads are managed by the underlying operating system scheduler, which is always (afaik) non-deterministic. Therefore threads execution order and timing is impossible to predict and reproduce.  How do you test an program in these conditions ?

 

Example scenario

A service invokes a long-running calculator operation on a separate thread.

 

Testing option1: unit test with timeout

  1. start the service and invoke the calculator logic.
  2. wait for the calculator to execute a callback
  3. check the actual value returned by the callback.

The difficulty here is in step 2 : defining how long the test should wait for the callback to be invoked. If the wait time is too long then the test will take much longer than required to complete. If it’s too small then the test will fail. On top of that the ideal timeout value(supposing it can be found) may change as the application evolves and the calculator  speeds ups (or slows down).

  • pros: comprehensive test of the business logic and threading mechanism.
  • cons: slow and fragile, subject to randomly break the build. Tests which take too long or randomly fail usually ends up being ignored.

 

 

Testing option2 : the humble object

The idea is to separate the hard-to-test threading mechanism from the easy-to-test business logic, i.e acknowledge that a unit test is not the ideal place to test the threading model (usually best done  as an end-to-end or manual test instead), while the business logic is best tested in a single thread.

1. create a single-threaded test to check that the service actually invokes the  calculator

2. create another single-threaded test to check the calculator callbacks the service with the expected calculated value.

  • pros: single-threaded, fast, deterministic tests
  • cons: the threading model is left untested., although this can be covered by end to end tests and code reviews.

 

Which option is best  ? option 2 does not provide a 100% coverage of the code but crucially the tests are fast and robust. On the other hand testing threads and logic altogether is a recipe for slow and fragile tests which will end up being ignored very quickly, bringing the test coverage to zero.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s