Rapid test-driven development with continuous testing

TIPS
1

Executive summary

This is a quick “tip” post about the continuous testing feature of Bnd(tools), which can help in this area by significantly speeding up the test-develop cycle for integration tests in OSGi systems. For a while I have been meaning to do an in-depth blog post on this, but I haven’t found the time - but given that I recently discovered that even some seasoned Bndtools developers were unaware of this new feature, I thought it was time to at least put something out there to point people in the right direction. I may at some time do a more in-depth blog post.

Background

We all know that testing is important, and that we should do more of it. Ideally, we should use the Test-Driven Development principle, where we code in small increments, starting with the test first, checking that it fails (hence proving that the test itself works), and then incrementally write the code to make the test pass. Doing it incrementally means lest gaps in our test coverage.

In practice, we often don’t do this - especially when it comes to integration testing. One of the main reasons that we don’t, of course, is that it can be slow, which interrupts our ordinary development flow.
If your integration tests run in an environment that takes seconds (maybe even minutes) to start up, you waste a lot of time waiting for the environment to restart if you attempt to do TDD. As a result, we tend to increase the size of our increments and risk leaving gaps in our test coverage.

This is where the continuous testing feature of Bndtools can help.

How it works

For a long time, Bnd and Bndtools have supported “live coding”. This feature leverages the hot-code-swapping ability of OSGi to make it easy to develop and quickly test the changes.

Continuous testing takes this concept one step further. Instead of (or perhaps even as well as) hot-deploying incremental changes to your code to allow you to test them manually, continuous testing will hot-deploy incremental changes to your code and test them automatically by re-running the corresponding automated tests. This feature has been available for some time but Bndtools 5.0 introduced significant enhancements and improved IDE integration which improves its utility.

How to configure continuous testing

Admittedly, the UI is perhaps not immediately obvious on this point which might explain why the feature has gone unnoticed. We’re happy to take feedback on this point on ways to improve it. But once it’s been shown to you how to use it, it’s really quite easy.

  1. Start with a working integration test. I will assume that you know how to do this - if you don’t, it’s best you learn how to crawl first. :smile:
  2. You will ideally need to use the most recent tester implementation (biz.aQute.tester.junit-platform ). Available since Bndtools 5.0, its main purpose is to support JUnit 5 and its pluggable test engine architecture - but (as we’ll see) it also better integrates continuous testing with the IDE. I will not go into more detail about this step here, but if there is interest I may write another tip post on it. For now, the advanced user may find pointers in the osgi-test project.
  3. You need to manually edit the Eclipse “Run Configuration” for your .bnd/.bndrun test launch. You can either create one from scratch, or edit an existing one by going to Run|Run Configurations… and scrolling to the OSGi Framework JUnit Tests category to find the run configuration for your bnd(run) file. The following key configuration options should be checked: launch
    1. Update bundles during runtime. This tells Eclipse to listen for changes to your bundles and re-deploy the changed bundles into the running OSGi framework.
    2. Continue running framework after tests have completed. Without this, the tester will call System.exit() once it has finished running the first test run (this is equivalent to setting the tester property tester.continuous=true).
    3. Display JUnit results in IDE every time the tester runs tests. Without this, only the first test run’s results will be displayed in the IDE. The tests will still re-run but you won’t see the results appear in the JUnit view of the IDE. This setting is ignored by the older (default) biz.aQute.tester.

How to use continuous testing

Once the above is configured, launch your integration test. Assuming you’ve configured and resolved your integration test properly, it will start and then run all of the integration tests that it can find (that is, all the classes listed in a Test-Cases header of a bundle). The results of the tests will display in the JUnit View of Eclipse, with all the useful UI features that that gives you (click navigation to failed tests, etc). Now you can start your TDD.

  • Make a change to your test code to test for your new feature, and save it - the test bundle will recompile, re-bundle, redeploy, and finally re-run. The results of the test will again appear in the IDE (and hopefully your newly-created test fails, indicating that it is testing what it is supposed to be!)
  • Now change the code-under-test to implement the new feature that you just wrote the test for, and save it. Eclipse will recompile, re-bundle and redeploy the bundle-under-test. Usually, the test bundle has an explicit dependency on the bundle-under-test, so when Bndtools restarts the bundle-under-test, it will also restart the test bundle. When the test bundle restarts, the tester will notice and automatically re-run all of its tests.

Happy TDD!

4 Likes
2

If you want to have more control you can add this to your .bnd/.bndrun file.

-runproperties: \
	tester.continuous=true,\
	launch.services=true

tester.continuous Continue running framework after tests have completed - like in the launcher ui.
*(At the moment it seems that this does not work correctly all time) @juergen-albert and @stbischof *

launch.services addes this gogo commands:

  • tester:runTests (ARGS) - runs the specified tests
  • tester:getTesterNames (ARGS - retrieves the tester.names filter
  • tester:setTesterNames (ARGS - sets the tester.names filter
3 Likes
3

(At the moment it seems that this does not work correctly all time)

This is true. Unfortunately, for unknown reasons, it seems that Eclipse sometimes “loses” the running process - the process’ console output detaches from Eclipse, and so there’s no way to shut down the running application except using the OS’s “kill” command (or equivalent). It does still keep running though, and will respond to changes. We suspect that this is a bug in Eclipse rather than in Bndtools.

4

Looks like it might be an issue with Bndtools after all - [bndtools tester] MAX_TEST_RUNS limit causing tester console to close · Issue #4716 · bndtools/bnd · GitHub

Hopefully a fix to come soon.