See how users leverage Virtuoso's self-healing test to obliterate their maintenance overhead.
Bold claims are abundant with companies behind test automation tools, but this is not always reflected in the users’ experience. With this in mind, we are going to look at one of Virtuoso’s claims, that we drastically reduce test maintenance with self-healing tests, and see if it stands up to the scrutiny of data on how our users actually interact with Virtuoso. Let’s find out if Virtuoso does do exactly what it says on the tin when it comes to self-healing tests.
When automating an interaction on a web application, say to click on a button, you need to first locate the target element. In test automation, this is commonly done using XPath and CSS selectors, which respectively look something like:
/html/body/div[4]/div[2]/main/div[5]/a[4] and .header > div.block .flex-item .flex-wrap a.
Because these selectors rely on the current specific structure of the page, they are brittle with respect to changes -- that is, either tests start targeting the wrong elements, or fail to identify the same element again. This creates a significant maintenance burden, and simply adds to the overhead of maintaining tests.
Some solutions try to reduce the effort by using multiple CSS/Xpath selectors, but those still remain brittle to many changes and page restructurings.
Virtuoso takes a different approach and uses intelligent element identification along with the advances in AI/machine learning, to make tests think like humans do. Given a hint, Virtuoso bots intelligently identify page elements, and over time re-identify the elements, even when the page or the element structure changes (i.e., automatically heals the tests).
Or at least that’s the theory… but how does this claim play out in practice?
Over the past few months, those clever people in our development team have been reviewing test execution data to see if their assumptions on healing were right and make improvements. The results are interesting enough to warrant a blog post.
First of all 40% of all tests needed healing. - This shows that there is no need for messing around trying to find XPaths or selectors when authoring tests, simply type in a hint and Virtuoso finds the element. More importantly, if that element changes Virtuoso will select the element it thinks it should be. Virtuoso can detect changes and make suggestions for elements, which is all well and good, but do our users agree with the elements it selects?
A whopping 95% of Virtuoso’s decision on healing was accepted by our users, way to go Virtuoso. It is with this figure that we can be confident that Virtuoso does do exactly what it says on the tin. Virtuoso is able to select elements, detect and heal planned UI changes, and therefore act like a human, well at least 95% human. But what does this mean for organisations who use Virtuoso?
On average four steps are repaired for every test healed. With many other forms of test automation, these steps would have needed to be updated manually, which doesn’t sound too much like automation. In fact, four cycles of revisions would have been needed to fix this test, and if each execution takes a few minutes, with the manual steps involved in between, 30 minutes to an hour of time could have been spent having to maintain this single journey! . Even with just one test Virtuoso cuts out a lot of work, but imagine how Virtuoso impacts a large regression suite of thousands of test cases, each needing to be manually updated with every minor UI change.
Virtuoso’s bold claims of being able to obliterate your test maintenance overhead with self-healing tests are confirmed by the data on how our users leverage the function. Test maintenance is a real problem as 40% of tests needed to be healed. Virtuoso solves this problem as 95% of the approval decisions users made confirmed the healings were correct. Testers using Virtuoso save time, effort and ultimately money by removing manual test maintenance from their automation strategy.
