Click here to Skip to main content
15,867,568 members
Articles / Web Development / XHTML

Test Explorer

Rate me:
Please Sign up or sign in to vote.
4.39/5 (9 votes)
14 Nov 2013CPOL6 min read 49.5K   939   18   16
Test automation tool to run Visual Studio test classes

Contents

Introduction

Test automation is an important part of the software development cycle. Being able to run these tests and to automate the jobs to run this test is also important. Thanks to Visual Studio, writing test automation on .NET Framework has been an easy task. But to run a set of test cases, first we have to load Visual Studio and then select and run the test cases. When we want to run a set of tests from different assemblies and different projects, this task becomes more complicated, needing to have several instances of Visual Studio open and running, one for each different test project.

As a result of this need, I came up with the idea of test explorer. Test explorer is a solution that will scan a directory looking for all the assemblies that contain test classes and test methods. It will then build a tree with all the assemblies found that have at least one test method, adding filter options like priority and owner. The user can select a set of tests from different assemblies/projects and schedule MSTest to run the tests. The results will be saved as a trx file on a result folder specified by the user, and optionally, these results can be sent by email as trx files or as HTML.

Image 1

Test Explorer

This application will scan a predefined folder and look for all the assemblies that contain test classes. On my team, we have defined that all our test projects (test assemblies) are named with the keyword 'testmodule' making the scanning process easier.

C#
string dropFolder = "drop\\Debug"; //Folder that contains the test assemblies 
string testFolder = "drop\\TestResults"; //Folder to place the test results

For each assembly found on the drop folder, let's scan it and retrieve all the test methods. The properties that are of our interest, for possible filtering options, are: “OwnerAttribute”, “PriorityAttribute”.

C#
public TreeNode GetTestMethods(string AssambleyPath)
{
    TreeNode _node = new TreeNode();
    _node.Text = Path.GetFileNameWithoutExtension(AssambleyPath);
    
    Assembly _assembly = Assembly.LoadFile(AssambleyPath);
    Type[] types = _assembly.GetExportedTypes();
    foreach (Type type in types)
    {
        Attribute _classAttribute  = type.GetCustomAttribute(typeof(TestClassAttribute));
        if(_classAttribute != null)
        {
            TreeNode _n = new TreeNode();
            _n.Text = type.Name;
            MemberInfo[] members = type.GetMembers(BindingFlags.Public| 
              BindingFlags.Instance| BindingFlags.InvokeMethod);
            foreach (MemberInfo member in members)
            {                        
                Attribute _att = member.GetCustomAttribute(typeof(TestMethodAttribute));
                if (_att != null)
                {
                    TestMethod _t = new TestMethod(member.Name);
                    foreach (var t in member.CustomAttributes)
                    {
                        if (t.AttributeType.Name == "OwnerAttribute")
                            {
                            _t.Owner = t.ConstructorArguments[0].Value.ToString();
                        }
                        else if (t.AttributeType.Name == "PriorityAttribute")
                        {
                            _t.Priority = 
                             Int32.Parse(t.ConstructorArguments[0].Value.ToString());
                        }
                    }
                    //Adding owner to the owner lists
                    if (!ownerCheckboxList.Items.Contains(_t.Owner))
                    {
                        ownerCheckboxList.Items.Add(_t.Owner);
                    }
                    //Adding priority to the priority lists
                    if (!priorityCheckBoxList.Items.Contains
                                  (string.Format("P{0}",_t.Priority)))
                    {
                        priorityCheckBoxList.Items.Add(string.Format("P{0}",_t.Priority));
                    }
 
                    string key = string.Format("K{0}", TotalCountOfTests++);
 
                    _TestMethodLists.Add(key, _t);
                    _n.Nodes.Add(key, member.Name);
                }                    
            }
            _node.Nodes.Add(_n);
        }
    }
    return _node;
}

Building the Command

Once all test methods are loaded and the proper filters (priority and owner) are displayed, the user can select as many tests from the different assemblies to run. If at least one test method on an assembly is selected, an MSTest command will be built. If you didn't know, MSTest.exe is a command-line tool that comes with Visual Studio used to run tests. You can find more information about the command line arguments to run MSTEST here.

The command used to run the test will have:

  • the assembly path, used on the option testcontainer
  • Results file path, used on the option resultsfile
  • a list of the selected test method names for that assembly, used on the option detail:testname

Result path:

Image 2

Setting options:

As an optional parameter, you can specify a testsetting file to run the test methods.

Image 3

As a sample, I'm providing a generic test settings file which specifies the timeout property for each test case (30000 milliseconds in the code shown below):

XML
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="TestSettings1" id="29046417-d7da-4b0a-a412-d4ede2bc9050" 
      xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Execution>
    <Timeouts testTimeout="300000" />    
    <AgentRule name="LocalMachineDefaultRole">
    </AgentRule>
  </Execution>
</TestSettings>

Process Wrapper

In order to execute the test methods, I´ll be calling MSTest. Visual Studio comes with a set of command line tools that are pretty handy when trying to automate processes.

The process Wrapper class is just a wrapper class that will call MSTest using the System.Diagnostics Process.

The first step is to get MSTest path. In my case, since I have installed Visual Studio 2012, this would be VS110COMNTOOLS.

C#
vsStudioPath = Environment.GetEnvironmentVariable
               ("VS110COMNTOOLS", EnvironmentVariableTarget.Machine);
if (vsStudioPath.EndsWith("\\"))
{
    vsStudioPath = vsStudioPath.Substring(0, vsStudioPath.LastIndexOf('\\'));
}
vsStudioPath = vsStudioPath.Substring(0, vsStudioPath.LastIndexOf('\\'));
_MsTestPath = vsStudioPath + "\\IDE\\MSTest.exe";

The next step is to retrieve the process and redirect the standard output and standard error to show it to the user as the log result of running the test.

C#
p = new Process();
p.StartInfo.UseShellExecute = false;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.FileName = _MsTestPath;
p.StartInfo.Arguments = Command;

And finally, once the process ends, get standard output:

C#
_StdOutput = p.StandardOutput.ReadToEnd();
_StdErr = p.StandardError.ReadToEnd();

Test Method Class

This class is designed to hold important data related to each test method we found on the assembly. In this quick demo, the only data that I added to the Test Method object is the owner, the name of the test method and the priority. I chose these fields based on the common queries/filters I use to schedule a test run. But feel free to add as many additional fields as you want.

C#
/// <summary>
/// Gets or Sets the Test Method Priority
/// </summary>
public int Priority
{
    get
    {
        return _Priority;
    }
    set
    {
        _Priority = value;
    }                
}
 
/// <summary>
/// Gets or Sets the Test Method Owner
/// </summary>
public string Owner
{
    get
    {
        return _Owner;
    }
    set
    {
        _Owner = value;
    }
}
 
/// <summary>
/// Gets the Test Method Name
/// </summary>
public string Name
{
    get
    {
        return _Name;
    }
}

Convert TRX File to HTML

Another cool feature is the ability to transform the Visual Studio Test Results file to HTML using an external library called trx2html (http://trx2html.codeplex.com/). Test results files can be converted to HTML with aggregate information, making the inspection of test results easier. With the results on HTML format, we can easily mail an automation report with the HTML body. As a second option, we can still email the results with the traditional trx files as an attachment.

C#
trx2html.ReportGenerator _Parser = new trx2html.ReportGenerator();
foreach (string _t in _ResultsFilePaths)
{
    _Parser.GenerateReport(_t);
    _progress.UpdateProgressBar();
    HtmlReportFilesPath.Add(_t + ".htm");
}

Email Reports

Sending an email with the test pass results:

This class allows you to automatically send the test pass results to a list of users (example: testers, team lead, etc.) so they can take action on failed test cases.

Using C# SmtpClient and MailMessage to send the email, the list of recipients will get the report of test results as file attachments (the TRX files generated by Visual Studio), or as HTML if the 'convert to HTML' option was selected before the test run.

It is important to set the client with the appropriate port and host of a valid SMPT email server and to provide valid network credentials.

C#
MailMessage msg = new MailMessage();
SmtpClient client = new SmtpClient();
client.Port = smtpPort;        
client.Host = smtpHost;
client.EnableSsl = smtpSsl;
client.Credentials = new NetworkCredential(userName,password);

Adding the sender details:

C#
msg.From = new MailAddress(_From, _displayName, System.Text.Encoding.UTF8);

Adding the recipients list:

C#
msg.To.Add(emailTo);

Sample method to attach a file:

C#
public void AddAttachment(string fileName)
{
     //Agrego el archivo que puse en la ruta anterior "PathFile", y su tipo.
    Attachment Data = new Attachment(fileName, MediaTypeNames.Application.Octet);
 
    //Obtengo las propiedades del archivo.
    ContentDisposition disposition = Data.ContentDisposition;
    disposition.CreationDate = System.IO.File.GetCreationTime(fileName);
    disposition.ModificationDate = System.IO.File.GetLastWriteTime(fileName);
    disposition.ReadDate = System.IO.File.GetLastAccessTime(fileName);
    //Agrego el archivo al mensaje
    msg.Attachments.Add(Data);
}

Enhancement:

As optional use, we can create a distribution list on a text file that will contain a list of valid email recipient addresses. In my sample, I created one called EmailDL.cfg. The program will load the recipients list and send the email to them. This way, it is easier to modify/add email addresses.

Cleanup Directories

As part of the test run, deployment folders are created by MSTest to hold the binaries and assemblies needed to run the test. As part of the cleanup process, I wanted to delete those folders. If you don’t wish to delete the deployment folders, you can simply avoid the call to the CleanUpDirectories().

C#
/// <summary>
/// Cleans up all folder directories in the TestResults folder
/// </summary>
private void CleanUpDirectories()
{            
    string[] filePaths = Directory.GetDirectories(_ResultsPath);
 
    foreach (string folder in filePaths)
    {
        CleanDirectory(new DirectoryInfo(folder));
        Directory.Delete(folder);
    }
}

/// <summary>
/// Cleans a single directory content
/// </summary>
/// <param name="directory"></param>
private void CleanDirectory(DirectoryInfo directory)
{
    foreach (FileInfo file in directory.GetFiles())
    {
        file.Delete();
    }
    foreach (DirectoryInfo subDirectory in directory.GetDirectories())
    {
        subDirectory.Delete(true);
    }
}

Conclusion

This project proved to be quite interesting and fun to develop. I think the app resulted from this project is really useful to run tests without the need to open Visual Studio. There are still several enhancements and to-dos that I would like to add in order to improve the utility of this app. Among the main features, I would like to add in future iterations of this project are:

  • Asynchronous/Threading support in order to show a progress bar with the overall status of the tests, the current running test, remaining tests, and cancel button to abort the current test run.
  • Task scheduling support: This would be a really nice feature to have. Using the windows task scheduler feature, we could schedule several test runs in advance, and recurrent test runs. In my team, we currently run our entire test suite once every day at midnight. Having the option to choose and schedule a test run with the selected test cases within the app would boost its utility.
  • Charts and summary report within the application. It's nice to have all the reports, TRX files and HTML files; but it would also be great to have a chart showing the success/failure rate and a brief summary showing the failed/passed test cases.

History

  • Revision 0 - New article
  • Revision 1 - Fixed typographical and grammatical errors

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
United States United States
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
QuestionCan this tool be used to test CodedUI Automated tests? Pin
Member 1362868421-Jan-18 3:28
Member 1362868421-Jan-18 3:28 
QuestionHow do I Fix the compiler Errors? Pin
Fl0wmastr24-Aug-17 11:05
Fl0wmastr24-Aug-17 11:05 
QuestionCan't fill up the tree view Pin
Member 1055885630-Jan-14 5:56
Member 1055885630-Jan-14 5:56 
AnswerRe: Can't fill up the tree view Pin
deoxys3-Feb-14 5:55
deoxys3-Feb-14 5:55 
GeneralRe: Can't fill up the tree view Pin
Member 105588563-Feb-14 10:31
Member 105588563-Feb-14 10:31 
GeneralRe: Can't fill up the tree view Pin
deoxys9-Feb-14 11:12
deoxys9-Feb-14 11:12 
GeneralRe: Can't fill up the tree view Pin
Member 1055885610-Feb-14 12:43
Member 1055885610-Feb-14 12:43 
GeneralRe: Can't fill up the tree view Pin
deoxys12-Feb-14 11:02
deoxys12-Feb-14 11:02 
QuestionTester looks good so far Pin
dparvin20-Jan-14 11:03
dparvin20-Jan-14 11:03 
AnswerRe: Tester looks good so far Pin
deoxys21-Jan-14 5:50
deoxys21-Jan-14 5:50 
GeneralRe: Tester looks good so far Pin
Member 109986607-Aug-14 10:40
Member 109986607-Aug-14 10:40 
GeneralRe: Tester looks good so far Pin
deoxys16-Sep-14 7:25
deoxys16-Sep-14 7:25 
Questionpretty good Pin
BillW3314-Nov-13 2:56
professionalBillW3314-Nov-13 2:56 
AnswerRe: pretty good Pin
deoxys14-Nov-13 4:34
deoxys14-Nov-13 4:34 
QuestionThanks! Pin
Oleksandr Kulchytskyi13-Nov-13 22:01
professionalOleksandr Kulchytskyi13-Nov-13 22:01 
AnswerRe: Thanks! Pin
deoxys14-Nov-13 4:33
deoxys14-Nov-13 4:33 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.