Case Study
Bridging the gap between SDETs and QAs
Virtuoso QA
Published on
See Virtuoso in action
Book a walkthrough on your applications and your workflows. Bring a requirement, a user journey, or a brittle Selenium script, and watch the loop run on something you recognise.
Our Story
The more we have moved to the cloud, web, and mobile interfaces, the harder this has become for teams. Tests and test automation have become a burden to create, maintain, and scale.
We have reimagined how software is tested by developing a game-changing platform that is already being used by the biggest names in software. We passionately believe that anyone should be able to create and maintain tests regardless of their coding ability, and that quality is a key driver for change and growth. The latest advances in AI and Machine Learning have been leveraged to produce test automation software that thinks like a human, empowers everyone to test, and for the first time, delivers on the promise of codeless test automation.
About the Client
The Nutanix Cloud Platform comprises hybrid cloud infrastructure and multi-cloud management along with unified storage, database services, and desktop services to support multiple applications and workload across locations.
Key Requirement: Web Application Testing
Types of Testing: Functional testing, Automation testing
Total 2 teams: R&D Calm, PIT
Duration: Ongoing
Key challenges faced by Nutanix before Virtuoso
1. Calmtest (in-house automation suite) limitations:
High skill requirement in coding (both design and implementation)
Difficult to keep up pace with UI feature development, specifically when UI/UX change rate (common problem) is high
This led to longer ship times or incomplete testing
2. A gap between an SDET and QA:
SDET with high coding skills and poor QA mindset is equally inefficient and ineffective to a seasoned QA with high QA mindset and poor coding skills. We needed to bridge this gap.
3. Less application coverage
This was a result of Calmtest limitations.
4. Test maintenance issues
Calmtest - based on popular open source framework Pytest - needed a lot of refactoring of scripts after any changes in the application.
Challenges resolved using Virtuoso
1. Ease of learning and setup:
Virtuoso brings true codeless automation to the table
Skill requirement is minimum with manual testers creating scripts using simple English-language statements
Users transformed into Virtuoso automation champions within a week
2. Self-healing and quick test maintenance enables QA to keep pace with rapid changes to the UI
Because it has self-healing properties, there is almost zero breakage when locators for UI element changes
Pre-commit tests have been running smoothly without any breakage
Re-usability: changes in one checkpoint reflects throughout
3. Shorter ship times compared to Calmtest
Test Automation development now roughly 10X faster
Direct saving of 15-20 mins on every code commit after using Virtuoso. That’s a direct impact on productivity
4. Bridges gap between SDET and QA:
True shift-left. Even Developers can fix automation bugs when they break it
5. Greater application coverage:
Approx. 300 complex test scenarios were automated using Virtuoso. On an average, three journeys were created every day. A total of two webapps and two browser-OS configurations were automated
6. Better scaling:
Cross-browser, cross-device, and execution planner scales up execution rapidly
7. Lesser cost:
Less infra costs as platform is purely cloud based
Cost savings through improved efficiency and engineer productivity
Hiring time significantly reduced
Some more Virtuoso benefits:
Covered testing of applications that are not hosted publicly through Virtuoso bridge
Covered testing of complex, end-to-end flows handling multiple iFrames and page load issues effectively through best practices like adaptive waits
Live Authoring to the rescue: increase in accuracy through Live Authoring, ensuring tests don’t fail on reruns
Dedicated support from Virtuoso team for help with test building and advanced support for API tests, etc. whenever required. Dedicated slack channel for quick query resolution and super fast turnarounds.
R&D Calm team leading the way with highest automation numbers
Users become automation testers in record time.
A user new to Virtuoso became a pro in just one week of training and practice
Authoring speed is fantastic at an average of 20 journeys created per day during peak authoring times
The current level of automation coverage was achieved with just ten regular Virtuoso users - big savings on resource costs
Currently, Virtuoso is being used by two teams across the company with the R&D Calm team emerging as the champion in terms of achieved automation coverage
Significant increase in automation coverage
Overview
A high volume of manual tests (approx. 300 complex test scenarios) were automated using Virtuoso. On an average, three journeys were created every day. A total of two webapps and two browser-OS configurations were automated
Virtuoso Stats
31 Projects
82 Goals
628 Active journeys (including drafts) as of June 2022
Each journey testing almost 150-200 steps on an average
