Functional testing and visual regression testing are two distinct types of testing. Sure, they both look at the UI, but they have different approaches, different objectives, and different toolsets. But are there advantages to using the same toolchain for both while respecting the fact that they require completely different approaches? Well, that is what this blog post is going to try to find out. Let's start by defining exactly what we mean by functional and visual regression testing.

• Functional testing validates that the application under test meets functional requirements/specifications. To put it simply, does the application work as it should? 
• Visual regression testing validates the look and feel of the user interface against the requirements/specifications.

From these definitions, you can see why they both require a completely different approach. But both do answer the same fundamental questions: is my application working as expected, and is it ready for release?

Although these two types of testing have been opened up to test automation, they have traditionally been done by manual testers checking that both the functionality and look and feel of an application are as expected. When we view these processes from the prism of manual testing, the two become less mutually exclusive and we can see an overlap. A manual tester wouldn't ignore a layout issue whilst testing for functionality or vice versa. But the disadvantages of manual testing are well known:

• Test creation and execution are slow
• Test coverage is reduced, as it is not scalable.
• Is prone to human errors, especially in the UI layer
• Repetitive unimaginative tasks.
• Defects are found later in the SDLC phases at a higher cost
• Slower release cycles
• Higher risk of defects not being found until the software is in production

To address those issues, teams automate tests at all levels, including functional and visual regression using scripts. At the UI level, the irony is that only functionality is tested by the automation tool in most cases. This is not real visual regression testing, which introduces its own problems:

• Still takes a long time to create and maintain the UI scripts
• Try to simulate UI validation, but no real visual regression tests are done at all
• A working application is needed to be able to automate against, which tends to mean you manually test first and then automate when it is working – a very slow approach to testing.

When automating testing in this way, you merely simulate visual regression testing as scripting rules for visual regression testing can only validate applications at the DOM level. You cannot guarantee that the UI meets the visual specifications. Validating an element's position is not the same as ensuring that the element is shown according to the specifications. Validating that an image has the correct given URL and is loading successfully does not mean that the image is the right one, as the designer may have saved the wrong image, etc.

Snapshot testing

The snapshot approach to automation is very different from the one used for functional testing. Is it then wise to use the same tools and processes for both? As we have seen, manual testers cover functional and visual regression tests page by page and state by state. A path is followed through the application which ends at a certain page or state of the application with baseline images. Functional tests typically follow these same journeys. Therefore, to combine functional testing with visual regression testing, we just need integrations with tools that offer support or the right tool that supports both functional testing and layout testing. Automated testing-wise, the effort for the visual tests comes almost for free, as we are executing snapshot testing based upon journeys created by functional testing. By using the same tool for functional and visual regression testing, you not only drastically reduce the time it takes to create and maintain, but you also half the costs of executing both processes separately.

Several tools have attempted to offer a solution, which was a great step forward. These tools use Selenium or other similar frameworks to drive the functionality as normal but append into the asserts a UI check. This removes the need to have two sets of automation tests, but still leaves you with:

• Tests that are slow to create and maintain (although faster and better than a pure Selenium approach to do simulated visual regression testing)
• Needing a working application to be able to automate against, which tends to mean a manual test first needs to be available and then automated when it is working – a very slow approach to testing

Virtuoso picks up the baton from here and lets us leap over these remaining two hurdles. And when you consider the speed at which DevOps is expected to deliver and therefore test, overcoming these hurdles is critical. The speed at which tests can be authored using Virtuoso is comparable with authoring a manual test script. The script can also be created much earlier - before the application is developed. Put simply, if you can write a manual test script, you can now write an automated script that covers both functional and visual regression testing processes. How this fits into your CI/CD pipeline is explained in this shift-left blog post.

With Virtuoso, the maintenance is also reduced, as only one script is needed. But Virtuoso goes one step further to reduce maintenance: Virtuoso self-healing technology reduces the maintenance to as close to zero as you can get.

What is the advantage of separating functional and visual regression into different tools and processes?

By having the same process, it doesn't mean there can't be two different reports coming out of a single execution. It is, in fact, important that these two reports are distinct. You want a report for functional testing and a report for visual regression testing, as you don't want to fail every test because there is a minor UI issue. So having two reports in the same process, not only saves time, effort, and money, it also allows you to:

• Have different teams to validate results
• Clear reasons for failures
• Keep functional and UI test fails separate
• Focus on what has changed, and prioritize testing

The disadvantages of not combining the two are increases in:

• The amount of work
• The testing time
• The cost to execute

Conclusion

There are clear reasons why functional testing and visual regression testing require a different approach. However, this does not mean that the test automation tools and processes used should not be combined. Reusability of automation scripts for functional and layout testing is key to speeding up the testing process and reducing test maintenance and execution costs. While having functional and visual regression separated at the reporting level, you can easily distribute work across a team, clearly identify reasons for failure, and keep functional and visual regression test fails separate. In doing so, we can create the right conditions to have the answers that give us the confidence to release, release often, and release faster.