The client with whom we engaged in this project belongs to a group of global retail and commercial banks with over 145 million customers and 14,000 branches – more than any other bank in the world. Since its founding in Spain over 150 years ago, the banking group has become an industry leader in the United Kingdom, continental Europe and Latin America.
The Uruguayan division of the banking group was undergoing a restructuring of the design of its retail banking software application while migrating to a new version.
The bank needed to carry out both web and mobile testing to ensure that everything that existed on the business level in the previous version would continue to work in the new version.
A business-critical system, the application handles aspects related to client accounts, cards and money transfers. Therefore, it was also imperative to test the system’s performance in terms of response times, resource usage, and maximum use capacity in order to achieve system reliability and stability under the expected concurrent load of users in production.
Ultimately, the client sought to feel confident that the new version would offer a frictionless user journey with an improved, easy-to-use interface.
Having formed a strong relationship with Abstracta over the years from previous engagements, the banking division chose to partner with our team yet again for its banking application testing needs.
For best results, we assembled a team composed of one Quality Engineering Lead, one senior tester, and four junior testers. This Abstracta team worked in close communication with the bank’s IT Manager and Digital Channel & ATM Manager, their corresponding teams, and the bank’s external software solution provider.
As part of our onboarding process, the Abstracta team hosted meetings with the bank's electronic channels team, its technology team and the software provider in order to gain an understanding of the functionalities within the scope of testing.
Then, the Abstracta testers created test cases in Excel and used Microsoft Team Foundation Server (TFS), to report incidents. We carried out functional and usability testing directly on multiple browsers, operating systems, and mobile devices, focusing on those that had been identified as the most important to the users.
Our testers performed the testing activities in an independent test environment, yet they had access to the production services and were able to configure the users and their profiles through the tool, BackOffice, and other satellite applications with which the banking system interacts.
After executing all the test cases and once they had passed, prior to release, the Abstracta team performed several smoke tests in an exploratory testing fashion. These tests helped to increase test coverage outside of the defined test cases, focusing especially on critical areas that, if they were to fail, would generate a negative impact on the business.
Abstracta’s performance engineers carried out several performance tests, simulating a concurrence of users with real data to minimize risks before releasing to production. After the release, we executed more tests, but this time in production, changing the scenario to minimize the risks of running in this environment.
Each performance test simulated a different load over a period of one hour each. When we executed the scenario that was designed to simulate the highest expected load in production for the most-used transactions by the banking clients, we uncovered key insights that would help the development team to improve the performance of the system.
Main findings from the load tests:
- While testing in the production environment, we found that the system managed to process up to 75% of the highest expected user load making transactions in 1 hour with an error rate of less than 14%
- We detected response times that were higher than desired in some of the most critical functionalities, including the login step
- In general, we found an increase in response times with the increase in user concurrency
- We detected deadlocks in the database that were corrected during the tests
Over approximately three months of functional test execution, the Abstracta team found 250 incidents (18% of which were critical and 25% of high severity) before launching the new banking software to production, decreasing risk in the most critical areas of the business and ensuring a seamless migration and a frictionless end user experience.
On the load testing side, Abstracta helped the client to find and remedy performance related issues in its retail banking system that would have been nearly impossible to find before going live without having run these tests. This allowed the client team to reduce the risk of system failure during periods of high user activity and to identify the main performance bottlenecks.