Click here to Skip to main content
15,867,686 members
Articles / Programming Languages / MSIL

Getting Started with Automated White Box Testing (and Pex)

,
Rate me:
Please Sign up or sign in to vote.
5.00/5 (19 votes)
28 Jan 2009Ms-PL9 min read 125.9K   602   121   23
Pex is a new tool that helps in understanding the behavior of .NET code, debugging issues, and in creating a test suite that covers all corner cases -- fully automatically.
This article provides a step by step introduction to Pex in Visual Studio 2008 or 2010 CTP. You will learn how to analyze existing code with a few clicks in the code editor, how to create test cases that reproduce issues that Pex finds and debug such issues, how to let Pex generate and save an entire test suite, how to write parameterized unit tests and also why parameterized unit tests will change the way you write unit tests.

Update: See this article as a video on Channel9!

pex/newpexillustration.png

Introduction

Pex is a new tool that helps in understanding the behavior of .NET code, debugging issues, and in creating a test suite that covers many corner cases -- fully automatically. Internally, Pex operates very much like a professional white box tester: it runs the code, learns about its behavior by monitoring, and finally comes up with new inputs to cover previously uncovered branches by thinking hard about the program behavior.

It only takes a single click to launch Pex from the editor, are you ready to run it on your code?

This article provides a step by step introduction to Pex in Visual Studio 2008 or 2010 CTP. In particular,

  • how to analyze existing code with a few clicks in the code editor,
  • how to create test cases that reproduce issues that Pex finds,
  • how to debug such issues,
  • how to let Pex generate and save an entire test suite, which is small, yet often achieves high code coverage,
  • how to write parameterized unit tests,
  • why parameterized unit tests will change the way you write unit tests.

Running Pex for the First Time

If you want to follow this tutorial on your machine, we assume that you have installed Pex on your machine.

Let's start from scratch, and write some code. We will write a method that turns a string consisting of separate words into a capitalized identifier, where the leading characters of the original words have been turned into upper-case, and dots have been escaped to underscores.

We start by creating a new C# class library to hold the StringExtensions class, and give a first shot at the Capitalize implementation.

C#
public static class StringExtensions 
{
    // convert 'hello world' to 'HelloWorld'
    // punctuation is turned into '_', others are ignored.
    public static string Capitalize(string value) 
    {
        // WARNING: this sample is for demonstration only: it *contains* bugs.
        var sb = new StringBuilder();
        bool word = false;
        foreach (var c in value)
        {
            if (char.IsLetter(c))
            {
                if (word)
                    sb.Append(c);
                else
                {
                    sb.Append(char.ToUpper(c));
                    word = true;
                }
           }
           else
           {
                if (c == '!')
                    sb.Append('_');
                word = false;
            }
        }
 
        return sb.ToString();
    }
}

Right-click into the code of the method, and select Run Pex Explorations in the context menu.

pex/DiggerRunPexExplorations.png

What does it mean to Run Pex Explorations? Pex will run your code, possibly many times, with different inputs. Don't run Pex on code that could launch real rockets!

After a brief moment of deep thinking, Pex starts showing the results of its analysis as a table in a separate Pex Exploration Results window. Each row of the table represents an actual input/output behavior of Capitalize. The values of the input parameter value are shown in the second column, and the corresponding return values in the result column. If an exception is raised, it is displayed in the Summary/Exception column.

pex/DiggerTestTable.png

Why did Pex choose these inputs? Are they special, or simply random? At first sight, the inputs might appear to be just randomly chosen character sequences. But, take a closer look: The inputs contain only the characters ':', 'p', 'J', and '\0'. These characters are representative of punctuation, lower-case characters, upper-case characters, and everything else. Pex picked exactly one representative of these equivalence classes that the program distinguishes. You might see slightly different characters, depending on the mood of the constraint solver that Pex uses under the hood. Think about the following: What would be the probability of randomly chosen inputs to contain punctuation characters, when there are 2^16=65536 different characters?

When we click on a row, we see more details on the right. Here, we see a stack trace of the exception. We could click on the Details header to see details about the inputs that Pex chose for this row.

pex/DiggerTestDetails.png

The .NET guidelines say that a method should never throw a NullReferenceException. We could fix this issue ourselves, or we could simply drop someone an email or file a work item, to let someone else take care of the issue (assuming that we have someone else to take care of our issues). That's what the Send To button is for.

pex/DiggerSendTo.png

But, let's face reality, we have to fix our own code by ourselves. Actually, we don't have to -- for simple issues like this, Pex can do it for us. Click on Add Precondition..., and then on the Apply button.

pex/diggeraddprecondition.png

Pex adapts the beginning of our Capitalize method and inserts a check for a null input, similar to the following code snippet:

C#
public static string Capitalize(string value) {
    if (value == null)
        throw new ArgumentNullException("value");
    ...
}

When we run Pex again, instead of seeing a failing NullReferenceException, we now see the acceptable behavior of an ArgumentNullException. The red cross icon has turned into a friendly green checkmark (on top of which sits a little sun and disk icons, whose meaning we'll explain later).

pex/DiggerExpectedException.png

We have seen how Pex can create a table of interesting inputs and outputs, how to interpret the results, and how to fix an issue that Pex found. So far, all the results that Pex generated lived 'in memory', let's see how we can save them in the solution.

Saving a Test, and Debugging an Issue

Some of the inputs that Pex lists in the table look strange. It would be nice if we could debug the code for such an input and step through the code lines. Actually, that's easy with Pex! Select a row, and click on the Capitalize Save Test... button.

pex/DiggerSaveTest.png

Pex shows a dialog that illustrates a sequence of steps that Pex will take: Pex will create a new test project, then perform many more little steps that we'll ignore for the moment, and finally, Pex will create a Unit Test with the test inputs of the currently selected row.

pex/DiggerCodeUpdatePreview.png

What is a Unit Test? A Unit Test is a method that takes no parameters, similar to the Main method of an application. When a Unit Test raises an unexpected exception, it fails, otherwise it passes. There are a number of frameworks for .NET (NUnit, MbUnit, VSTest, ...) that automate the detection, execution, and reporting of such tests. In this tutorial, we use VSTest which comes with the Visual Studio Professional installation. Pex can also work with other test frameworks; check on CodePlex whether a custom extension for your favorite test framework has been written already.

The generated project contains several files. The C# source code of the generated saved test that corresponds to the current row in the table is stored in StringExtensionsTest.Capitalize.g.cs (similarly to designer files, this file is hidden under StringExtensionsTest.cs; unlike the designer files, you should really not edit the hidden file, as Pex may later on delete or regenerate its content).

pex/diggersolutionexplorer.png

We should now see the following C# code of the generated saved test case. The [TestMethod] attribute denotes that the method is a Unit Test, and [PexGeneratedBy(...)] attribute denotes that Pex generated the test.

C#
[TestMethod]
[PexGeneratedBy(typeof(StringExtensionsTest))]
public void Capitalize03()
{
    string s;
    s = this.Capitalize("\0");
    Assert.AreEqual<string>("", s);
}

Go back to the source code of Capitalize, and set a breakpoint on the first line.

pex/diggerbreakpoint.png

We can now press the Debug button in the Pex Exploration Results window under the details of the selected row.

pex/diggerdebug.png

We can now step through the code line by line.

Saving the Test Suite for VSTest

At this point, we can not only save individual tests, but we can save the entire table as a test suite. This test suite can serve as a regression test suite in the future, or as a fast-running build verification test (BVT) suite, a test suite that can be executed every time a code change is submitted to the source repository.

To save the entire table, select a row in the table, and press Ctrl-A. Then, click on the Save... button to the right of the selected rows.

pex/DiggerSaveAll.png

Again, Pex will show the dialog that details all the individual steps that Pex performs to save the tests as C# code. Press Apply.

We now have an entire test suite that we can execute without Pex. The generated tests are automatically recognized by VSTest, the Unit Test framework built into Visual Studio. Select Test->Windows->Test View.

pex/DiggerTestView.png

Select a test, and select all by pressing Ctrl-A, and press on the Run Selection button in the Test View window to run all tests:

pex/DiggerTestViewRunAll.png

Running the tests at this time will simply reproduce the same results that Pex reported earlier. However, running the tests in the future might discover breaking changes in the program behavior.

A Glimpse of Parameterized Unit Tests

We didn't point it out yet, but when we saved the tests, Pex not only created (traditional) Unit Tests that covered all the corner cases, but also created a Parameterized Unit Test (PUT) stub for the Capitalize method:

C#
[TestClass]
[PexClass(typeof(StringExtensions))]
[PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException))]
public partial class StringExtensionsTest
{
    [PexMethod]
    public string Capitalize(string value)
    {
        // TODO: add assertions to method StringExtensionsTest.Capitalize(String)
        string result = StringExtensions.Capitalize(value);
        return result;
    }
}

This method lives in the file StringExtensionsTest.cs, while the individual (traditional) Unit Tests were saved in StringExtensionsTest.Capitalize.g.cs. The ...g.cs file should never be modified by hand. If we want to customize the way Pex generates tests, or if we want to add checks that verify that the tested code works correctly, then we should always edit the PUT in the top-level file.

For example, here we could adapt the PUT to check that Capitalize does not change the number of letters:

C#
[PexMethod]
public void CapitalizeMaintainsLettersCount(string input)
{  
    string output = StringExtensions.Capitalize(input);
    Assert.AreEqual(
        LettersCount(input),
        LettersCount(output));
}
static int LettersCount(string s)
{
    return Enumerable.Count(s,
        c => char.IsLetter(c) || c == '_');
}

Another PUT could check that Capitalize is idempotent, i.e., if we capitalize an already capitalized string, then we get back the same string:

C#
[PexMethod]
public void CapitalizeIsIdempotent(string input)
{
    string capitalized = StringExtensions.Capitalize(input);
    string capitalizedTwice = StringExtensions.Capitalize(capitalized);
    Assert.AreEqual(capitalized, capitalizedTwice);
}

We can state that the result of Capitalize only contains letters and underscores:

C#
[PexMethod]
public void CapitalizeReturnsOnlyLettersAndUnderscores(string input)
{
    string output = StringExtensions.Capitalize(input);
    PexAssert.TrueForAll(output,
        c => char.IsLetter(c) || c == '_');
}

You get the idea.

Parameterized Unit Tests vs. Traditional Unit Tests

What's the big deal with Parameterized Unit Tests (PUTs)? Well, they allow you to separate two concerns that are often lumped together when writing Unit Tests:

  • the description of the intended program behavior,
  • the particular inputs that are needed to cover corner cases in the code.

While you have to write the PUTs, a tool like Pex can take care of generating the test inputs that cover the corner cases. Pex saves these inputs as traditional Unit Tests. As a result, you have to write and maintain less test code: When you change the corner cases in the implementation, you can simply re-generate the individual Unit Tests from the Parameterized Unit Tests with Pex, and you are back to high or full code coverage, or Pex might point out which corner cases cause failures.

You can (and should) start writing PUTs even before writing any code. And, you can mix PUTs with traditional Unit Tests, when it just doesn't make any sense to factor out values as parameters.

Further Reading

Further Watching

Further Questioning

History

  • 28th January, 2009: Added Channel9 link
  • 13th January, 2009: Updated links
  • 21st November, 2008: Initial version

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)


Written By
Engineer
United States United States
Jonathan de Halleux is Civil Engineer in Applied Mathematics. He finished his PhD in 2004 in the rainy country of Belgium. After 2 years in the Common Language Runtime (i.e. .net), he is now working at Microsoft Research on Pex (http://research.microsoft.com/pex).

Written By
Software Developer (Senior) Microsoft Research
United States United States
Nikolai Tillmann is a Principal Research Software Design Engineer at Microsoft Research.

His main area of research is program specification, analysis, testing, and verification. He is leading the Pex project (http://research.microsoft.com/Pex), a framework for runtime verification and automatic test case generation for .NET applications based on parameterized unit testing and dynamic symbolic execution.

Previous projects he worked on include AsmL, an executable modeling language, and the Spec Explorer 2004 model-based testing tool. He contributed to XRT, a concrete/symbolic state exploration engine and software model-checker for .NET code. Spec Explorer 2007 is based on this engine, which is now productized internally by the Protocol Engineering Team at Microsoft to facilitate quality assurance of protocol documentation.

He received his M.S. ("Diplom") in Computer Science from the Technical University of Berlin in 2000.

Comments and Discussions

 
QuestionExcellent Article Pin
Paul Conrad5-Sep-15 18:57
professionalPaul Conrad5-Sep-15 18:57 
GeneralMy vote of 5 Pin
Kanasz Robert28-Sep-12 5:48
professionalKanasz Robert28-Sep-12 5:48 
Generalproblem with NUnit frame work using pex Pin
ravi.vellanky1-Feb-10 23:56
ravi.vellanky1-Feb-10 23:56 
GeneralRe: problem with NUnit frame work using pex Pin
Jonathan de Halleux2-Feb-10 4:17
Jonathan de Halleux2-Feb-10 4:17 
QuestionPex is applicable for web services also? Pin
ravi.vellanky1-Feb-10 23:54
ravi.vellanky1-Feb-10 23:54 
GeneralBroken Link Pin
Fabio Franco13-Jan-09 7:19
professionalFabio Franco13-Jan-09 7:19 
GeneralRe: Broken Link Pin
Jonathan de Halleux13-Jan-09 10:32
Jonathan de Halleux13-Jan-09 10:32 
GeneralMissing something Pin
Fabio Franco12-Jan-09 8:49
professionalFabio Franco12-Jan-09 8:49 
GeneralRe: Missing something Pin
Jonathan de Halleux12-Jan-09 13:52
Jonathan de Halleux12-Jan-09 13:52 
GeneralRe: Missing something Pin
Fabio Franco12-Jan-09 23:56
professionalFabio Franco12-Jan-09 23:56 
GeneralLicense PinPopular
Thomas Weller25-Nov-08 3:35
Thomas Weller25-Nov-08 3:35 
GeneralRe: License Pin
Jonathan de Halleux25-Nov-08 3:57
Jonathan de Halleux25-Nov-08 3:57 
GeneralRe: License Pin
Dmitri Nеstеruk26-Nov-08 8:52
Dmitri Nеstеruk26-Nov-08 8:52 
GeneralRe: License Pin
Jonathan de Halleux26-Nov-08 10:18
Jonathan de Halleux26-Nov-08 10:18 
GeneralUmm Pin
Dmitri Nеstеruk25-Nov-08 2:23
Dmitri Nеstеruk25-Nov-08 2:23 
GeneralRe: Umm Pin
Jonathan de Halleux25-Nov-08 3:49
Jonathan de Halleux25-Nov-08 3:49 
GeneralRe: Umm Pin
Dmitri Nеstеruk26-Nov-08 2:13
Dmitri Nеstеruk26-Nov-08 2:13 
QuestionRe: Umm Pin
itisier26-Nov-08 2:34
itisier26-Nov-08 2:34 
AnswerRe: Umm Pin
Jonathan de Halleux26-Nov-08 3:57
Jonathan de Halleux26-Nov-08 3:57 
QuestionRe: Umm Pin
kamlendraS18-Feb-09 20:59
kamlendraS18-Feb-09 20:59 
GeneralRe: Umm Pin
Jonathan de Halleux26-Nov-08 3:51
Jonathan de Halleux26-Nov-08 3:51 
There is no database or random guessing. Pex does a pure whitebox analysis and, thus can generate inputs beyond simple datatypes (i.e. Pex also generate object graphs).

The Channel9 movie explains how it performs this but in a nutshell, Pex traces the code as it executes and build the path condition on each branching point (i.e. boolen formula in terms of the inputs). Then, iteratily, it picks a branch that has not been covered yet, flips the guarding condition, conjuncts it with the condition to get to that location and asks a constraint solver to find a solution. If the constraint solver finds a solution, Pex translates that solution back to code and executes it. So by construction, it hits a new branch in the program.

Jonathan de Halleux - My Blog

GeneralPeli's back! Pin
Member 63015024-Nov-08 22:57
Member 63015024-Nov-08 22:57 
GeneralRe: Peli's back! Pin
Jonathan de Halleux25-Nov-08 5:04
Jonathan de Halleux25-Nov-08 5:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.