Testing is an essential part of any software deployment project, whether a new eCommerce gateway or migrating from SAP ECC to S/4HANA.
But ERP testing SAP and other packaged applications differs from standard web or mobile application testing. Instead of answering the question, “does my application work?” ERP testing addresses the question, “can I conduct business?” And unlike standard web and mobile testing, ERP testing is:
- Multi-variant,
- Cross-technology,
- Cross-actor, and
- Requires institutional knowledge.
Yet, despite the differences found in ERP testing, the technology requirements for testing packaged applications fall neatly into three categories:
- Discover, or understand and document what to test and how to test it.
- Test, or perform the tests required to ensure correct functionality.
- Protect, or mitigate risk by creating or masking test data while ensuring compliance and coherent delivery.
This blog takes a closer look at these three categories and shows how this unique approach to testing packaged applications enables successful ERP testing.
ERP Testing Step #1: Discover
When testing ERP systems, put the daily practice of the business first. Understand specifically how the organization uses the system to achieve business outcomes. By discovering business processes across actors – most people in large enterprises will not be familiar with every aspect of the business – teams can then test from the perspective of solving the larger business problem of ensuring operational continuity after any change.
Yet, discovery can be an expensive, challenging, and even disruptive process.
In this first step to successful ERP testing, it is essential to capture how business processes are executed in an application in fine detail and at scale and in aggregate across the enterprise. This approach allows numerous actors and regular process participants to easily contribute knowledge to the repository of business process testing.
Plus, conducting discovery in this way creates a robust set of testing requirements from actual practice to ensure business continuity after change.
Some enterprises will require additional diligence for testing in the form of audit and traceability of functional specifications to production. Many industries including pharmaceuticals, medical devices, ISO 9001 manufacturing, and others must prove for each change tests ensuring critical functionality are executed and successful, with documentation.
Unregulated industries often have a practical requirement for test management, reporting, and traceability. Consider an SAP S/4HANA migration where custom Z-code is modified in an existing ECC system and replicated during a brownfield migration to a project landscape: Traceability ensures regression is not introduced accidentally at go-live for the new S/4HANA environment.
ERP Testing Step #2: Test
Testing business applications presents a host of challenges that do not exist in standard manual testing or test automation. Often, meeting the requirements of ERP testing necessary to ensure continued business support after making changes is more complex than traditional functional application testing.
Today’s enterprise environments are large, complex and typically mix on-premises and cloud-based solutions, increasing the range of support and functionality required for a testing solution.
A comprehensive solution for packaged application and ERP testing should include the following types of testing at a minimum:
Functional Testing
Functional business process testing leverages the automated business process discovery performed during the discover phase.
Load and Performance Testing
Performance testing properly evaluates realistic transaction volumes and distributions of workload. Ideally, this solution leverages elastic cloud capacity, which is important if the ERP solution under test includes integrations with customer- or field-facing user experiences that typically generate high volumes of activity.
Coverage Analysis
Test coverage analysis identifies potential gaps in testing plans, as well as business availability exposures from incomplete testing. When evaluating testing coverage, the scope should include known business process practice, testing library analysis, and test execution reporting to reveal user activities without associated test cases.
Mobile Testing
While the use cases for testing ordinary mobile apps are well understood, in enterprise use cases additional requirements such as two-factor authentication, biometric information, and cellular network or VPN testing may be functional requirements. Additionally, sufficient mobile testing requires support for virtual and physical device automation.
API Testing
API testing is important as well, especially for use cases involving integration and data exchange.
Service Virtualization
Service Virtualization enables “shift-left” testing, and for ERP testing virtualizes dependencies from the application chain. These virtual services reduce the total assembly required in a test lab to begin functional testing, as well as save infrastructure costs and provisioning requirements.
ERP Testing Step #3: Protect
In the world of application testing, there is a popular quote: “Half of testing is test data.” A version of this quote has been the life of many testing teams, and data remains a persistent challenge for ERP testing. Common data challenges in packaged application testing include:
- Exhausting existing test data in systems suitable for test conditions and scenarios.
- Creating or harvesting realistic test data for test conditions and scenarios.
- Identifying suitable coherent scenarios across end-to-end processes, especially ones touching dozens of actors, systems, and transactions.
- Protecting sensitive production data for regulatory compliance or trade-secret reasons.
Solving for the comprehensive range of test data challenges often requires a combination of approach and technology. Three common approaches and technologies work well in tandem, and should be considered a minimum functionality set for teams evaluating robust ERP testing solutions:
End-to-End Execution
End-to-end test design has significant business value because they are inherently real-world examples of performing complex business processes and touch many aspects of regular business practice, identifying failure along the way. End-to-end tests tend to be quite durable and resilient to change, since they create their own scenario data for consumption later in the test.
The one drawback of this type of test is the longer run-time and, due to complexity, the odds activities later in the business process test case are less tested by failures earlier in the business process.
Synthetic Test Data Generation
In load and performance testing for customer-facing interfaces, synthetic test data generation is almost always a necessity to simulate significant volumes and not get false readings of performance due to system or data cache and related technical reasons. Synthetic data may also play a valuable role when using Service Virtualization instead of real-world integrations for creating plausible responses from decoupled third-party integration dependencies.
Data Masking
The use of production data often includes masking to maintain compliance with data governance regulations or simply to protect sensitive company information from unnecessary non-production exposure.
For many ERP testing requirements, this is the only effective choice as the complexity of end-to-end business process scenarios across applications or data processing performed in applications makes coherent synthetic data impractical in practice. The testing solution uses “real” scenarios and data from production, but after preprocessing with a masking solution to obfuscate the actual data and create the same exact scenarios and coherence across applications with masked information instead.
Advanced solutions may support the concept of Continuous Data Delivery, though this feature may be less effective in practice. Data exhaustion can still occur triggering a system refresh (or snapshot reset) and may prove problematic when configuration transport differences between production and QA environments complicate continuous replication.
Having all three approaches available makes the availability of test data practical, regardless of the testing requirements and infrastructure situation.
Bottom Line
This blog explored a unique approach to successful ERP testing by implementing the Discover-Test-Protect method. But many enterprises struggle to implement this comprehensive method effectively, due to the sheer number of vendors needed to handle each aspect of the packaged application testing process.
With Perforce, enterprises can rely on a comprehensive ERP testing solution for enterprise continuous testing capabilities, while also reducing their vendors they are working with by 65%.
Experience Perforce in action. Get your custom demo of our platform today.