When we compare the different approaches to test creation, can we really say that one approach is definitely better than the rest?
In this post we provide a quick overview of three very different approaches to automated testing strategies; Record and Replay, Scripting, and Natural Language Programming (NLP), their pros and cons and ask ourselves “Is there a better way?”
Let's start with Record and Replay. This approach allows the user to easily follow a natural journey through the application without ever having to think that they are actually "testing" it. The user doesn't need to know what commands to use, and it requires little or no training to start with automation. While recording, the user's full attention is focused on achieving the journey rather than automated testing needs.
Record and Replay is more precise with element selection. It can automatically capture a wide spectrum of deterministic selectors that will exactly match the element the user interacts with. But, it's also a more “organic” approach to test authoring because it's much closer to how a genuine user would likely behave. With Record and Replay, testers can simply interact with the application as if they were a real customer.
In testing, there will always be scenarios that are so exceptional and specific that codeless interfaces will not be able to address them. In these cases, you would be able to rely on the power of programming languages within an automation framework.
Being able to script means you are able to cover every single angle and detail inside your application scope as well as interact with external systems. Whether it's preparing the test environment conditions or validating that a third party received and processed an event triggered by the application, the number of practical applications for scripting is near incalculable.
The way scripting can be designed as a library for reusability and its content easily edited makes it even more flexible and versatile for your needs. The control and accuracy that a user has over interactions with the application, environment, and external systems is probably the strongest advantage of scripting.
However, the downside of scripting is simply the cost. Development cost is significant because the user needs to know how to code. He/she also needs to understand the basic principles of software engineering to be able to address different systems and environments.
The maintenance cost is also high because when the application changes, the scripts are often no longer valid, fail to execute, and require fixing. Because of these disadvantages, the use of scripting should be a last resort when everything else fails.
NLP testing is also known as "codeless test automation," but it's a phrase that many testers shy away from since codeless automation has been notoriously unreliable in the past. But hear us out before you pass judgment; codeless automation has come far in recent years, and now you can have the usability and accessibility of codeless along with all of the power of scripted automation.
Since NLP tests are deliberately written or converted into plain English, they are more likely to be readable. Also, since users have already seen the tests once, they'll recognize them more easily and require less time to inspect them in the future. Studies in software testing support this (see a paper published by our own Head of Engineering, Sina Shamshiri) Using NLP also gives users the ability to write automated tests regardless of whether the current application version has been deployed or not. This gives users the power to effectively shift left and start building test automation earlier in the development phase because the tests do not depend on an exact version of the application.
Similar to scripting tools, NLP has increased reusability, control, and accuracy over the automatic recorders. These often have to guess user interactions and suffer when recording complex interactions due to corresponding technical challenges. With NLP, the user has full control and can describe exact user interactions.
Maintenance and augmentation of tests created using Record and Replay can be cumbersome, often requiring a replay of the journey. Meanwhile, in most cases with NLP, users can make the changes directly to their tests.
Let's not forget about portability, too. You can use a wide spectrum of technologies to write tests, from your smartphone or tablet to a simple piece of paper. For privacy-conscious users, with NLP there is no risk involved with installing an extension that can record everything they do and write.
Moreover, there is no dependency on a software application needing to be installed on your machine or browser for you to be able to write automated tests in plain English.
At Virtuoso, we ultimately believe the closer we can bring the authoring tools together, the better we can bring teams together and provide a richer user experience. Therefore, we support and offer all three approaches, seamlessly integrated with one another.
Here are some examples of how a hybrid approach could feature as part of a test automation strategy.
In short, there is no one-size-fits-all approach to the testing process. The varied demands placed upon QA by dynamic applications and Continuous Integrations pipelines call for a mix of techniques, plus a testing tool that provides Continuous Testing. The reality is that some applications are more complex to test, as they integrate with more systems as part of complex business processes. Scripting will be required in part. But for the lion's share of testing, why script and maintain code when you can take advantage of speedier, more accessible ways of testing? It all comes down to need and preference.
What sets the codeless test automation tool that is Virtuoso apart from competitors is the fact that everything is combined into one platform. Whether you record it, script it, describe it in plain English, or let bots generate the tests for you in a sort of automated exploratory testing, Virtuoso takes your testing further with lightning-fast execution, maintenance, and self-healing capabilities.
