Mistakes – Are Your Automated Tests Really Testing?

Introduction

There’s a great deal of satisfaction to be had from seeing a consistently green dashboard for your automated testing, the business thinks the testers are writing good, reliable tests and their software is free of issues. The trouble is, it’s not that difficult to get a test that passes all the time if you’re not actually validating anything of note. Obviously, the validation you need to be performing in your tests depends entirely on your requirements and acceptance criteria, so there is no definitive list of what you must and musn’t check. To stick with what we are familiar with in previous articles, let’s use the term “assert”.

While there is no right or wrong assert, there is most definitely a good and bad assert.

Looking at a specific example, let’s take an assert of an expected string after performing an action on a page, in this example, a successful order being placed. Our acceptance criteria states that a successful order being placed should show a string saying:

"Thank you for your order! You will receive confirmation by email shortly!"

We could do an assert at the end of our test that says:

Assert.IsTrue(orderConfirmationPage.ConfirmationText.Contains
("Thank you for your order! You will receive confirmation by email shortly!");

Now the above will work, and will pass as long as our confirmation page contains that string. But that’s the trouble, if it CONTAINS the string. If our string came back as:

"Yo shopper. Just thought we’d say "Thank you for your order! 
You will receive confirmation by email shortly!" 
But in the mean time, you could have gone elsewhere and got it a lot cheaper, we’re overpriced!"

But because we’re only checking our string contains the message, it will still pass. It doesn’t care what else is before or after it, just as long as at some point, that string is there.

Which means basically, you aren’t really testing much at all. A better assert would be:

Assert.IsTrue(orderConfirmationPage.ConfirmationText.Equals
("Thank you for your order! You will receive confirmation by email shortly!");

Now, the only way our assert can pass is if our string is exactly what we are passing in. No extras, no typos, nothing. So straight away, our assert is a much better test of our order placement page.

But it’s important to point out that even in this most simple of examples, our assert for our test can be even more thorough to ensure even better test coverage. Let’s say our order confirmation page is on its own URL, and as well as that, it also has a ‘Continue Shopping’ button that will only appear when on that page. We can improve the validation of our test even more by adding the following:

Assert.IsTrue(orderConfirmationPage.ConfirmationText.Contains
("Thank you for your order! You will receive confirmation by email shortly!");
Assert.AreEqual($"{homePageUrl}/orderconfirmation", orderConfirmationPage.Url)
Assert.IsTrue(orderConfirmationPage.ContinueShoppingButton.Exists()

Now we are ensuring that not only is our text correct, but that we’ve also correctly navigated to the correct page and the expected button exists on the page. So we can be completely confident that if our test passes, we are testing everything, and now if we have a failure, something must be wrong.

It does not just asserts that can prove to be an issue with poor test validation. The test cases themselves can be the problem. If a test case is particularly troublesome to automate, then it can be tempting to adapt it in a way that you can at least automate part of it or a modified version of it. The trouble with this is that by moving away from the original test case, not only are you moving away from what needs to be tested, but now you potentially have a test case that no-one ever really asked for nor wanted.

These are also the types of tests that are easy to get to pass, because they do so little, or test nothing of note within the application/system they are written for. This is why it’s important at the start of a project, or even at the start of each sprint or iteration, to determine exactly what your test strategy will be so you can identify these areas unsuitable for automation early on.

The main focus of this article was to emphasise how important it is when writing automated tests, to not only have validation, but to have the correct validation. Never write tests with the intention of them always passing, write them with the test case and acceptance criteria in mind. This way, you rely on the development on being correct and the quality being high for your test to pass, rather than a weak test.

The post Mistakes – Are Your Automated Tests Really Testing? appeared first on Learn Automation.