Data driven testing: testing the application configuration

What started Testaco was a Spring Framework based application. When this application grew to a non-trivial size, it became apparent that the "wiring" done in xml in Spring is every much a part of the code as any java code, and should be tested as it was back in the EJB days - you need to test those JNDI lookups in order to have a good test coverage. To try to address this issue, the tests were moved from unit-style tests towards integration-style tests where several Spring beans would participate in the test. Unit tests were still used where integration tests could not exercise the code properly.

This worked well enough that an attempt was made at pushing the idea to the extreme, where only the behaviour provided by the container was taken out of the application (in this case, driving the servlets and providing a JNDI-based configuration system). This seemed to work well.

The reason this made sense were that the applications were data capture or data delivery style applications. Data would be transferred from A and placed in B. Either A or B would be one or more databases, and the tests would just ensure that what got sent in was what appeared in the database. Other application types do significant processing on the information, and do not lend themselves as easily to this style of testing.

It was also suddenly far easier to communicate with stakeholders. If some request broke some existing functionality, it was very easy to describe which system was affected, which use case was involved, and what would break - and a big red bar in an IDE stopped any discussions about whether something would actually break or not. The tests were related to functionality the stakeholders cared about, not with some tiny unit that only handled part of a use case.

There are drawbacks to this style of testing. The startup time for the first test will be high (10 seconds for a fairly complex database mapped with hibernate, approximately 500 spring singleton beans and 25k LOC on a fairly hefty workstation). This is right on the border line of being unusable due to the speed, but I hope that increases in CPU speed will render this a moot point during the next few years.

Tests will also seem to break much more often. This is because some use cases are "hot spots" - new functionality will tend to cluster around a very limited set of functionality, use cases and therefore around a couple of tests. This can be awkward if several teams work on related functionality, since it is very hard to merge these kinds of tests when repository conflicts occur. These issues will hopefully be handled in the future.

The tests themselves would look something like this:

public class SimpleTest (...) {
 @LoadDb(datafile = "tests-data/empty.xml")
 @CheckDb(datafile = "tests-data/singleentry.xml")
 public void testSimple() throws Exception {
   (...)
 }
}

The database manipulation has been put into annotations to enable easier tooling. Tooling for this test style turned out to be absolutely necessary, since maintaining the dbunit xml files turned out to be a major time drain - and a tedious one, at that. The tasks involved in maintenance of these files were not hard, and they lent themselves to automatization.

Evolving the data model through refactoring

This is where the library was in 2006. I had attended a talk by Scott Ambler by this time about database refactoring. Database refactoring is what we would all love to be able to do - we want to grow and shrink our data model as our problem domain evolves and we have to put new features into our applications.

When hearing about database refactoring from Mr. Ambler, a couple of things clicked into focus. 99% of the maintenance "overhead" that came with this kind of test style was adding and removing columns and tables from the different databases that were part of the system. This is what Mr. Ambler calls "database refactoring", and what I find very hard not to think of as "evolving your domain model". These are not maintenance tasks - they are directly related to adding new features.

Scott Ambler and Pramodkumar Sadalage have written a book called "Refactoring databases: Evolutionary database design". There is also a recorded version of Mr. Amblers talk on InfoQ. These are two references to more information about database refactoring. I will try not to reiterate this information here beyond what is absolutely necessary.

Refactoring is a development process which prepares the system for functional additions. In this contest, addition of a database column would be done by first ensuring that all tests ran, then adding the column without tying any behaviour to it, and then retesting to see that no existing functionality broke as part of the test. Then new behaviour could be added to the system. Testaco is a tool that tries to reduce the time it takes to do the steps that go before adding new behaviour,

It is possible to do these kinds of operation without having a test suite to help ensure that no bugs are introduced, but this will of course mean that you will handle this risk some other way.

As Mr. Ambler points out, this operation mostly concerns the "database architect" role in a development team. The software developers in an "enterprise development team" have been "refactoring" their code for a number of years already. Developers move methods, fields, classes, packages around like nothing.

The same thing now seems to be happening to the data model. The reason why it has taken so long is that it is much harder to do functional changes to a data model than it is to do localized, nonfunctional changes to your code, like renaming a method. The reason for this is that more people are involved.

In any reasonably successful enterprise, there are going to be many applications and people living off the data in your database. When the database changes, you have to handle all of these applications. What happens to your business reports, which are running as homemade cron jobs, written in some obscure script languag which nobody but your reporting expert knows, running on a box you don't know about, and which is going to break horribly now that you are going to fix the typo in a column name in one of your core tables?

It is very common to try to handle these kinds of changes by not doing them, or doing them as seldom as possible, just because something might break. There are many very successful database architects out there who have made this their core function in a company. The underlying idea seems to be that the harder it is to change your data model, the more people will think, and the more people who think about something, the more errors you will catch.

This is in direct opposition to what the business end of any enterprise wants. When a marketing or a product department sees a new emerging market or a changing business requirement, they need any neccessary changes to the business model and therefore the data model to happen fast - or at least faster than any of the competitors are able to do the change - so that their enterprise is the one which defines the speed of the market. No wonder there is a never-ending discussion between business-savvy people and developers - some of us are working towards totally different goals.

Testaco is an attempt to handle some of these issues, specifically the issues that are to do with development of software.