After more than three months it is time for a new article part of the most successful Automate The Planet's series- Design Patterns in Automated Testing. In the next few publications, I will try to solve a major problem in automated testing in my belief- the tests' failure analysis. Usually, the errors displayed when a test fails a far from the real reason why it fails. In this first part, I am going to explain to you my first approach to the problem using the Chain of Responsibility Design Pattern.
Abstract UML Class Diagram
Participants
The classes and objects participating in the original Chain of Responsibility Design Pattern are:
- Handler-defines an interface for handling requests. Contains the HandleRequest method.
- ConcreteHandler- Holds the actual logic for handling a request. It has access to its successor. If it cannot manage the request itself, passes the execution to its successor.
- Client- initialize the concrete handlers' chain and initiates the request.
What Are the Problems That We Try to Solve?
As I already mentioned at the beginning of the article, the most time-consuming activity in automated testing after the code writing is the tests' failure analysis. It is possible in the long run this to be even the most time consuming depending on the complexity of the tests' framework and the code quality. When the tests' failure is more accurate and understandable for people who didn't write the tests, I believe the required time for maintenance will be decreased.
Chain of Responsibility Design Pattern for Failed Tests Analysis
My main idea here is that almost in 99% of the cases the UI tests fail because some element is not found. So we can plug-in the analysis engine at this exact point when we catch these exceptions. I decided that the Chain of Responsibility Design Pattern is a good fit for the job because we need an algorithm similar to exception handling in C#. We start the analysis from the most common reason causing failures and if the check doesn't succeed we continue with more uncommon tests. Such check can be for example 'yellow exception screen', file not found, page not found and so on. These checks can be shared for multiple applications or can be strictly specific for your current case. I developed the idea even further with the introduction of test case specific handlers where you can plug-in unique handlers only for a particular action on the page. For example, if you click a button, and a validation message is displayed instead of redirecting to another page. Usually, the test will fail because some element on the second page cannot be found. Instead, you can add custom test case handler which can check for the validation message if the element location logic fails.
Handler
public abstract class Handler
{
protected abstract string DetailedIssueExplanation { get; }
public Handler Successor { get; private set; }
public void SetSuccessor(Handler successor)
{
this.Successor = successor;
}
public void ClearSuccessor()
{
this.Successor = null;
}
public void HandleRequest(Exception ex = null, params object[] context)
{
if (string.IsNullOrEmpty(this.DetailedIssueExplanation))
{
throw new ArgumentException(
"The detailed exception explanation should be specified.");
}
if (this.IsApplicable(context))
{
throw new AnalyzedTestException(this.DetailedIssueExplanation, ex);
}
else if (Successor != null)
{
Successor.HandleRequest(ex, context);
}
}
protected abstract bool IsApplicable(params object[] exceptionData);
}
The main class in this implementation is the abstract Handler. As explained in the abstract UML class diagram it defines an interface for the concrete handlers. Moreover, it contains the implementation of the HandleRequest method. It is modified to fit our needs for the Exception Analysis engine. It requires its successors to implement the IsApplicable method where the actual exception analysation logic is present. If the check is successful, a new AnalyzedTestException is thrown containing the DetailedIssueExplation. The last holds the exact reason why the test failed which will guide us if there was a bug. Through SetSuccessor method we chain the next handler if the initial check is unsuccessful, we call the successor if one is present.
AnalyzedTestException
public class AnalyzedTestException : Exception
{
public AnalyzedTestException()
{
}
public AnalyzedTestException(string message)
: base(FormatExceptionMessage(message))
{
}
public AnalyzedTestException(string message, Exception inner)
: base(FormatExceptionMessage(message), inner)
{
}
private static string FormatExceptionMessage(
string exceptionMessage)
{
return string.Format(
"\n\n{0}\n\n{1}\n\n{2}\n",
new string('#', 40),
exceptionMessage,
new string('#', 40));
}
}
This is the custom exception type that all handlers will throw if their validation is successful. Below you can find a sample analysed exception message.
Concrete UML Class Diagram
Participants
The classes and objects participating in the modified Chain of Responsibility Design Pattern are:
- Handler-defines an interface for handling requests. Contains the HandleRequest method. Through SetSuccessor and ClearSuccessor a new handler can be added or removed from the chain. The IsApplicable method should be implemented in the concrete handlers.
- HtmlSourceExceptionHandler- checks in the HTML source code of a page if specific text is present if so a beatified exception is thrown.
- IExceptionAnalyser- defines an interface for easier global work with handlers.
- ExceptionAnalyzer- implements the IExceptionAnalyzer interface. It holds the main handler for the application and the whole chain that will be executed in case of an exception.
- ElementFinderService- is the class responsible for localisation of elements on the page and it uses the ExceptionAnalyzer in case some element is not found.
HtmlSourceExceptionHandler
public abstract class HtmlSourceExceptionHandler : Handler
{
public HtmlSourceExceptionHandler()
{
}
protected abstract string TextToSearchInSource { get; }
protected override bool IsApplicable(params object[] context)
{
IBrowser browser = (IBrowser)context.FirstOrDefault();
if (browser == null)
{
throw new ArgumentNullException("The browser cannot be null!");
}
bool result = browser.SourceString.Contains(this.TextToSearchInSource);
return result;
}
}
Above you can find the code of the concrete handler for finding particular text in the HTML of a page. It uses the IBrowser interface part of the hybrid test automation framework. You can read more about it in my Design & Architecture Series. The another member of the class is the property TextToSearchInSource which holds the text that we will search for.
ServiceUnavailableExceptionHandler
public class ServiceUnavailableExceptionHandler : HtmlSourceExceptionHandler
{
protected override string DetailedIssueExplanation
{
get
{
return "IT IS NOT A TEST PROBLEM. THE SERVICE IS UNAVAILABLE.";
}
}
protected override string TextToSearchInSource
{
get
{
return "HTTP Error 503. The service is unavailable.";
}
}
}
The above code prints a beatified message if there are problems with the service and it is unavailable.
FileNotFoundExceptionHandler
public class FileNotFoundExceptionHandler : HtmlSourceExceptionHandler
{
protected override string DetailedIssueExplanation
{
get
{
return "IT IS NOT A TEST PROBLEM. THE PAGE DOES NOT EXIST.";
}
}
protected override string TextToSearchInSource
{
get
{
return "404 - File or directory not found.";
}
}
}
Similarly, the FileNotFoundExceptionHandler notify us if the file or directory we are trying to open is not found.
CustomHtmlExceptionHandler
public class CustomHtmlExceptionHandler : HtmlSourceExceptionHandler
{
private readonly string textToSearchInSource;
private readonly string detailedIssueExplanation;
public CustomHtmlExceptionHandler(
string textToSearchInSource,
string detailedIssueExplanation)
{
this.textToSearchInSource = textToSearchInSource;
this.detailedIssueExplanation = detailedIssueExplanation;
}
public CustomHtmlExceptionHandler()
{
}
protected override string TextToSearchInSource
{
get
{
return this.textToSearchInSource;
}
}
protected override string DetailedIssueExplanation
{
get
{
return this.detailedIssueExplanation;
}
}
}
As I mentioned earlier, the idea about the exception analysis can be developed even further. It can be applied just for specific test cases. For example, if you click a button and the page is not redirected to the next one because of a validation message. Through the above CustomHtmlExceptionHandler, we can create such use-only-once Handlers for particular tests.
IExceptionAnalyzer
public interface IExceptionAnalyzer
{
void AddNewHandler<TNewHandler>() where TNewHandler : Handler, new();
void RemoveLastHandler();
Handler MainApplicationHandler { get; set; }
}
The IExceptionAnalyzer interface defines methods for easier global work with Handlers. You can add and remove them and access the primary Handler of the app.
ExceptionAnalyzer
public class ExceptionAnalyzer : IExceptionAnalyzer
{
public ExceptionAnalyzer(Handler mainApplicationHandler)
{
this.MainApplicationHandler = mainApplicationHandler;
}
public ExceptionAnalyzer()
{
}
public Handler MainApplicationHandler { get; set; }
public void AddNewHandler<TNewHandler>()
where TNewHandler : Handler, new()
{
var newHandler = new TNewHandler();
newHandler.SetSuccessor(this.MainApplicationHandler);
this.MainApplicationHandler = newHandler;
}
public void RemoveLastHandler()
{
MainApplicationHandler = MainApplicationHandler.Successor;
MainApplicationHandler.ClearSuccessor();
}
}
There is nothing special about the actual implementation of the ExceptionAnalyzer. It uses the methods of the Handler class to setup the chain.
Chain of Responsibility Design Pattern in Tests
ExceptionAnalizedElementFinderService
public class ExceptionAnalizedElementFinderService
{
private readonly IUnityContainer container;
private readonly IExceptionAnalyzer excepionAnalyzer;
public ExceptionAnalizedElementFinderService(
IUnityContainer container,
IExceptionAnalyzer excepionAnalyzer)
{
this.container = container;
this.excepionAnalyzer = excepionAnalyzer;
}
public TElement Find<TElement>(IDriver driver, Find findContext, Core.By by)
where TElement : class, Core.Controls.IElement
{
TElement result = default(TElement);
try
{
string testingFrameworkExpression = by.ToTestingFrameworkExpression();
this.WaitForExists(driver, testingFrameworkExpression);
var element = findContext.ByExpression(by.ToTestingFrameworkExpression());
result = this.ResolveElement<TElement>(driver, element);
}
catch (Exception ex)
{
# region 9. Failed Tests Аnalysis- Chain of Responsibility Design Pattern
this.excepionAnalyzer.MainApplicationHandler.HandleRequest(ex, driver);
#endregion
throw;
}
return result;
}
public IEnumerable<TElement> FindAll<TElement>(IDriver driver, Find findContext, Core.By by)
where TElement : class, Core.Controls.IElement
{
List<TElement> resolvedElements = new List<TElement>();
try
{
string testingFrameworkExpression = by.ToTestingFrameworkExpression();
this.WaitForExists(driver, testingFrameworkExpression);
var elements = findContext.AllByExpression(testingFrameworkExpression);
foreach (var currentElement in elements)
{
TElement result = this.ResolveElement<TElement>(driver, currentElement);
resolvedElements.Add(result);
}
}
catch (Exception ex)
{
# region 9. Failed Tests Аnalysis- Chain of Responsibility Design Pattern
this.excepionAnalyzer.MainApplicationHandler.HandleRequest(ex, driver);
#endregion
throw;
}
return resolvedElements;
}
public bool IsElementPresent(Find findContext, Core.By by)
{
try
{
string controlFindExpression = by.ToTestingFrameworkExpression();
Manager.Current.ActiveBrowser.RefreshDomTree();
HtmlFindExpression hfe = new HtmlFindExpression(controlFindExpression);
Manager.Current.ActiveBrowser.WaitForElement(hfe, 5000, false);
}
catch (TimeoutException)
{
return false;
}
return true;
}
private void WaitForExists(IDriver driver, string findExpression)
{
try
{
driver.WaitUntilReady();
HtmlFindExpression hfe = new HtmlFindExpression(findExpression);
Manager.Current.ActiveBrowser.WaitForElement(hfe, 5000, false);
}
catch (Exception)
{
this.ThrowTimeoutExceptionIfElementIsNull(driver, findExpression);
}
}
private TElement ResolveElement<TElement>(
IDriver driver,
ArtOfTest.WebAii.ObjectModel.Element element)
where TElement : class, Core.Controls.IElement
{
TElement result = this.container.Resolve<TElement>(
new ResolverOverride[]
{
new ParameterOverride("driver", driver),
new ParameterOverride("element", element),
new ParameterOverride("container", this.container)
});
return result;
}
private void ThrowTimeoutExceptionIfElementIsNull(IDriver driver, params string[] customExpression)
{
StackTrace stackTrace = new StackTrace();
StackFrame[] stackFrames = stackTrace.GetFrames();
StackFrame callingFrame = stackFrames[3];
MethodBase method = callingFrame.GetMethod();
string currentUrl = driver.Url;
throw new ElementTimeoutException(
string.Format(
"TIMED OUT- for element with Find Expression:\n {0}\n Element Name: {1}.{2}\n URL: {3}\nElement Timeout: {4}",
string.Join(",", customExpression.Select(p => p.ToString()).ToArray()),
method.ReflectedType.FullName, method.Name, currentUrl, Manager.Current.Settings.ElementWaitTimeout));
}
}
The ExceptionAnalyzedElementFinder is almost identical to the original one discussed in my Design & Architecture Series. The only difference is that if an element is not found, the IExceptionAnalyzer is called to analyse the page. The primary Handler's HandlerRequest method is used.
Extend ExecutionEngineBehaviorObserver
ServiceUnavailableExceptionHandler serviceUnavailableExceptionHandler =
new ServiceUnavailableExceptionHandler();
var exceptionAnalyzer = new ExceptionAnalyzer(serviceUnavailableExceptionHandler);
this.unityContainer.RegisterInstance<IExceptionAnalyzer>(exceptionAnalyzer);
this.unityContainer.RegisterType<IDriver, TestingFrameworkDriver>(
new InjectionFactory(x => new TestingFrameworkDriver(
this.unityContainer,
browserSettings,
exceptionAnalyzer)));
A small addition to the ExecutionEngineBehaviorObserver should be added to register the instance of the actual implementation of the ExceptionAnalyzer.
EmptyEmailValidationExceptionHandler
public class EmptyEmailValidationExceptionHandler : HtmlSourceExceptionHandler
{
protected override string DetailedIssueExplanation
{
get
{
return "The client was not successfully loged in instead an Empty Email validation was displayed!";
}
}
protected override string TextToSearchInSource
{
get
{
return "Enter your email";
}
}
}
This is a case-specific custom exception Handler which checks whether a particular validation message is displayed on a page or not.
LoginTelerikTests
[TestClass,
ExecutionEngineAttribute(ExecutionEngineType.TestStudio, Browsers.Firefox),
VideoRecordingAttribute(VideoRecordingMode.DoNotRecord)]
public class LoginTelerikTests : BaseTest
{
private IExceptionAnalyzer exceptionAnalyzer;
public override void TestInit()
{
base.TestInit();
this.exceptionAnalyzer = this.Container.Resolve<IExceptionAnalyzer>();
}
[TestMethod]
public void TryToLoginTelerik()
{
this.Driver.NavigateByAbsoluteUrl("https://www.telerik.com/login/");
this.exceptionAnalyzer.AddNewHandler<EmptyEmailValidationExceptionHandler>();
var loginButton = this.Driver.FindByIdEndingWith<IButton>("LoginButton");
loginButton.Click();
var logoutButton = this.Driver.FindByIdEndingWith<IButton>("LogoutButton");
logoutButton.Click();
this.exceptionAnalyzer.RemoveLastHandler();
}
}
To use the new custom-created EmptyEmailValidationExceptionHandler, we add it on top of the chain through the AddNewHandler method. After it is no more required, we remove it. The handler will be called if the login or logout buttons are not found, this is its life scope.
Reference:
Design Patterns in Automated Testing
The post Failed Tests Аnalysis- Chain of Responsibility Design Pattern appeared first on Automate The Planet.
All images are purchased from DepositPhotos.com and cannot be downloaded and used for free.
License Agreement
CTO and Co-founder of Automate The Planet Ltd, inventor of BELLATRIX Test Automation Framework, author of "Design Patterns for High-Quality Automated Tests: High-Quality Test Attributes and Best Practices" in C# and Java. Nowadays, he leads a team of passionate engineers helping companies succeed with their test automation. Additionally, he consults companies and leads automated testing trainings, writes books, and gives conference talks. You can find him on LinkedIn every day.
Submit an article or tip
