Integration vs Unit Tests

When people are talking about automated testing, unit tests get all the buzz. Alpha geeks do them all the time, Uncle Bob Martin blogs about them, Oprah invites them to her show, Mr. President... well, you got the idea. When somebody mentions integration tests, it's like, "you should write integration tests like a good boy/girl, or you'll be punished". Those of you who are just starting automated testing probably think, "ok these integration tests must be very difficult, l'm really scared". This post will ease your fears, I promise!

What's the Difference, Again?

When we look closer at these two types of tests, we discover that there's no clear distinction between them. People tend to think that unit tests are those that test only one class. But a "unit" can sometimes consist of several closely related classes serving the same purpose. Some folks even dare to say that, regardless of what you test, unit tests are those that run fast. Ayende, for example, openly supports including interaction with a database (via NHibernate) in your unit tests. A couple of years ago, you'd be banned from the community for such rebellious statements!

Integration tests are meant to check how good several units can work together as a team. (There are also "functional" and "acceptance" tests that do the same, and some folks even claim to understand the difference.) People are scared to write integration tests, because they think they'll have to test the system all the way from the input screen to the database, or another screen, or a call to an external service.

Rejoice, 'cuz Your Fear Will Liberate Ya!

If you don't feel like writing tests for your UI, pick a testing framework that makes it easy (like this one), or start your test just one level below the UI. While the first option will keep you confident that your UI is correctly integrated with the rest of the system, the other one will help you isolate the UI from the code, improving the system design. This is the general rule: moving towards integration testing gives you more confidence, whereas unit testing helps in improving your design.

As you see, you can include in your test any number of your classes, and sometimes even external services, and depending on the current trend it could be called whatever. Let's say some tests are more "unit-y" than the others, but there's no clear distinction between the two, just a continuous spectrum. For the purpose of this post, let's talk about "smaller" and "bigger" tests.

Unit Tests are Shiny

Some say that laziness drives all progress. When you write a unit test, you want to be really really lazy. You want to make a tremendous effort to write as little code as possible. And you should become hypersensitive to writing code that is not strictly necessary for testing this particular piece of functionality. You start asking yourself: do I really have to create all these objects and save them to the database in order to test a simple string property? Or can I make a shortcut? Do I have to write lots of code just to calculate the value I need to check? Maybe I just expose it via a property?

The moment you allow your laziness influence your code is huge. All the programmer Gods smile, all the Alt.Net geeks nod approvingly, and all the cheerleaders start to dance. In addition, your code becomes easier to use, to fix, to improve, and sometimes even to understand. Why this happens -- is a big mystery, and also a topic for a future blog post. But the basic idea is simple to understand, when you try to write a smaller test, you end with a smaller class under test.

An additional benefit of a unit test is when it breaks, you know exactly which part of the system doesn't work correctly, since the test covers only a small portion of functionality.

Can a small test document the system's functionality? Sometimes yes (provided you've given it a clear name), but only a small portion of it. Can a small test document the system's usage? Yes, but mostly in libraries. While end-to-end systems are typically used through the UI, libraries are used via code, and the unit tests provide sample usage for some exotic cases. Can unit tests provide a safety net against regression bugs? Only a part of it: with all units working as intended, you can reconnect them in a way that breaks the system, while keeping all your unit tests pass.

Integration Tests are Mighty

While all devs tend to prefer unit tests, newbies tend to write integration tests, without actually knowing it. Despite common belief, integration tests are sometimes easier to write. You just follow the user's story. For example, if you have the following requirement:

If a client has the "Preferred" status (i.e, she has made more than 3 orders last month for the total amount of 3500 Ottoman piastres), she should receive an email about every sale.

The test would be very straightforward: create a client, add some orders with a sufficient total, then start a sale, and verify that an email message is sent (the latter is not so simple, but we can use Typemock, or a special helper class in Ivonna, in order to check that). A new user would read a test and know exactly what happens here. If you bother to name the test class and its methods according to the tested features, she won't even have to look at the test code.

I'm Confused, Can't You Just Do a Straightforward Comparison?

Ok, if you want it..

Driving the Design

• Unit tests are best for it. They assist you in extracting individual responsibilities into separate classes that are focused on implementing a single thing, and are easy to replace.
• Integration tests don't drive your design at all.

Providing the Safety Net Against the Regression Bugs

• Unit tests guarantee (more or less) that individual units work as expected. They don't guarantee that the system works as expected as a whole. What's worse, unit tests often give you false positives: you can change the implementation of a particular feature, and some unit tests might fail, although the system actually works fine. This is because unit tests often test the implementation details rather than the behavior of the system.
• Integration tests confirm that the system works as a whole, independent of the implementation. They can, however, fail if they touch an external system which is unavailable for some reason.

Maintainability

• Unit tests are super-easy to maintain. Since they cover a small piece of functionality, when a test fails, it is clear which part of the system should be fixed.
• Integration tests are hard to maintain. When an integration test fails, you better have the functionality covered by unit tests, or you've got a debugging session ahead.

Documenting the System

• Unit tests can, with some effort, document small pieces of the application's workflow, stuff that most users are not interested in. This is useful when you want to implement and plug in a custom FactoryShmactory (a unit test would provide an example), or just learn the system's inner details.
• Integration tests document the main features of the system. They demonstrate the end-to-end user actions and their output.

This is Some Boring Theory, Gimme an Example Already!

Let's start from a user story:

I've got a database of kitties, and want to be able to search them by name.

The simplest way to test the search process, at least known for me, is to create two instances, one that satisfies the search condition, the other not, and apply the search to the collection of these two. If we get the "good", but not the "bad" one, we can assume that our search works correctly.

We'll start with an integration test, in which we test the whole scenario: save our two kitties to the database, and then perform a database search (don't worry, this post is SQL-free). Let's assume that we use NHibernate for database access, and StructureMap as our IoC container. Let's call SearchQuery, the class we're trying to test-drive. The idea is that we create an API that makes it easy to write a test. The easiest way would be to create a method called JustGimmeAllKittiesNamedOptimusPrimeNow, but it makes sense to introduce a search parameter. Our test might look like this:

[Test]
public void QueryByNameReturnsTheMatchingKittiesFromTheDb() {
	//Arrange
	var session = ObjectFactory.GetInstance<ISession>();
	var optimus = new Kitty {Name = "Optimus Prime"};
	session.Save(optimus);
	var tron = new Kitty {Name = "Megatron"};
	session.Save(tron);
 
	var query = ObjectFactory.GetInstance<SearchQuery>();
 
	//Act
	IQueryable<Kitty> result = query.FindByName("Optimus Prime");
 
	//Assert
	Assert.AreEqual(1, result.Count());
	Assert.AreEqual("Optimus Prime", result.First().Name);
}

Most sane developers will confirm that this is an integration test, since it involves a database. (Side note: as always with integration tests, you should be extra careful in order to leave the system in the same state as before the test. I'm usually achieving this by creating a fresh in-memory SQLite database instance, but that's out of scope of this article.) As a result, our class is coupled to NHibernate, which means that we cannot use it with NoSQL databases, or other data sources (chances are, we'll never need that).

On the positive side, our test verifies that the system works correctly as a whole.

Now let's try to move in the "unit" direction. Our TDD book says, thou shalt not touch thou database, Hands off! But how do we test the database search without touching the database itself? Easy, let's use the IQuerable interface. In our test, we'll "fake" the database by using IQuerable<Kitty> as the source of our query. In our production code, we'll substitute it with the "real" database data.

[Test]
public void SearchSpecReturnsTheMatchingKittiesFromTheSource() {
	//Arrange
	var optimus = new Kitty {Name = "Optimus Prime"};
	var tron = new Kitty {Name = "Megatron"};
	var source = new[] {optimus, tron}.AsQueryable();
 
	var spec = ObjectFactory.GetInstance<SearchByNameSpecification>();
 
	//Act
	IQueryable<Kitty> result = spec.Filter(source, "Optimus Prime");
 
	//Assert
	Assert.AreEqual(1, result.Count());
	Assert.AreEqual("Optimus Prime", result.First().Name);
}

The SearchByNameSpecification is what we test-drive here. This class is not coupled to the database, it can filter any IQuerable<Kitty> source. Since it has less responsibilities, it is easier to maintain. The test itself is easier to maintain as well, because we don't have to setup a database.

Concluding Remarks

This post has become too big already, so I thought I'd rather leave it without any conclusion. However, if you feel like you know something more about integration, or unit, tests, or you want to start a heated debate, you're welcome to contribute in the comments. And don't forget to check the other posts in the Tough Choices series!