Blog
June 22, 2022
The financial services and insurance (FSI) sector has seen a slew of changes in recent years.
Changing customer expectations are being met by new FinTech market entrants creating fierce competition. Meanwhile, traditional and heritage firms are hamstrung by complicated legacy systems — and data that’s absolutely mission critical — racking up ever-greater maintenance costs. Compliance risks, data challenges and speed-to-market constraints are hindering their move to becoming truly digital-first businesses.
A key bottleneck to innovation for everyone, however, is getting test data to development teams when they need it. As the pace of software development increases, and as automation becomes widespread in DevOps, a comprehensive DevOps test data management (TDM) strategy has become critical.
The five challenges of test data management
Test data management isn’t new. Large organizations have already centralized their test data functions using some form of internal process. The efficiencies achieved, however, have given way to a whole new set of challenges created by ever-faster, iterative release cycles.
Firstly, setting up data-ready testing environments using legacy approaches is a slow, manual and high-touch process. It can take days or weeks to provision test data, partly because most organisations operate queue-based request systems.
In legacy setups, software development teams also often don’t have access to good quality test data anyway. All too often, they have to work with stale copies of production data due to the complexity of setting up a new test bed. The result is lost productivity due to time spent resolving data-related issues such as accuracy, compliance and lack of visibility.
Moreover, to enable continuous testing cycles, companies need to distribute the data to their teams effectively and efficiently. So there needs to be a way of continually provisioning that test data effectively, such as via API.
Also, processes for securing test data and enabling compliance traditionally add major operational overheads. This is essential if your data contains financial or credit card information. Yet, an end-to-end process to secure data using homegrown scripts or legacy data masking tools masking process may take an entire week, which can prolong test cycles and delay time to market.
Finally, test data storage and cloud service provider costs are rising continually. IT organisations almost always create multiple, redundant copies of test data, resulting in poor use of infrastructure. The very act of standing up dedicated test systems that will not be highly utilised is costly and inefficient. Operations teams often have to prioritise who gets access to what and when, resulting in further delays to critical projects.
Strategies for effective test data management
IT organisations therefore need to adopt a DevOps TDM approach to make the right test data available to their project teams. That’s the only way of ensuring they will be able to test at anything like the speed of development. A comprehensive approach should seek to improve test data automation in each of the following areas:
- Test data speed: Reducing the time to provision test data.
- Data quality: Ensuring developers get high-fidelity test data.
- Security and compliance: Maximizing security without compromising agility.
- Sustainable business value: Reducing the costs of storing and archiving test data.
Test data speed
FSI companies need to streamline the process of making a copy of production data available to downstream testing environments. They need to create a path towards fast, repeatable data delivery and optimal digital experiences, featuring:
- Automation — Making it effortless to deliver copies of test data and providing a low-touch approach to standing up new test data environments.
- Toolset integration — To improve automation and reduce handoffs between teams with key DevOps tools such as masking, subsetting, synthetic data creation and more.
- API-first approaches — Giving developers interfaces that are purpose-built for their needs, enabling efficient delivery, effective control, and no need for queueing.
- Ease of use — From a well-orchestrated approach, eliminating wait times, enabling earlier testing in the software development lifecycle (SDLC), and so reducing costs.
Data quality
Development teams need access to high-quality data that meets their needs across three dimensions:
- Data age — So that data doesn’t become stale and cause costly errors, the latest production data needs to be readily available in minutes.
- Data accuracy — To enable multiple datasets to be provisioned to the same point in time and simultaneously reset to quickly validate complicated testing scenarios.
- Data completeness — Cutting operational costs by eliminating the practice of data subsetting by provisioning fully complete test data sets that take up a fraction of the space.
Security and compliance
Masking tools irreversibly replace sensitive data with fictitious yet realistic values. Data masking can therefore completely eliminate the risk of data breach in test environments, ensuring regulatory compliance. This is now the de facto standard for protecting test data, enabling developers to focus on accelerating production, and should offer:
- End-to-end repeatability — To automate and speed up the masking process – identifying sensitive data, applying masking to that data, and then auditing the resulting test dataset.
- No need for development expertise — With modern, lightweight masking tools that can be set up using a low/no-code approach without scripting or specialized development expertise.
- Integrated masking and distribution — Where the process is tightly coupled with a data delivery mechanism to help ensure the masked data can be delivered wherever it’s needed.
Sustainable business value
A DevOps TDM platform can enable organizations to make the most efficient use of infrastructure resources, meeting the following criteria:
- Data consolidation — To reduce redundant data and curb storage costs associated with legacy approaches.
- Data archiving — To optimize storage use, enable fast retrieval and so enable libraries of test data to be made easily available.
- Environment utilization – To maximize returns on investment on IT through improved timesharing and the ability to leverage ephemeral test data with fast spin up and spin down.
Transform Your Software Delivery with Perforce Delphix
Your development teams have automated building, testing, and deployment. But manual test data provisioning creates bottlenecks that slow down entire release cycles. Perforce Delphix eliminates this constraint with enterprise-grade data automation built for modern DevOps.
5-Minute Demo: How Delphix Eliminates Bottlenecks
In this quick demo, my colleague Ajay Thotangare shows you how to eliminate development bottlenecks by using Delphis for fast data delivery:
Related blog >> What is Delphix?
Accelerate Data Delivery for DevOps Teams
Delphix leverages data virtualization to automatically deliver complete, virtual data copies into test environments. Break free from slow, queue-based provisioning systems with a platform designed for continuous delivery. Advanced data operations like versioning, branching, and instant refresh become seamless parts of your development workflow.
Built-in Security for DevOps Test Data Management
Protect sensitive data without sacrificing development speed. The Delphix DevOps Data Platform automatically applies consistent masking policies across all environments. Rest assured that your test datasets meet GDPR, HIPAA, and PCI compliance standards. Your developers get realistic, masked data that maintains data relationships and referential integrity.
Accelerate Innovation with Delphix
Leading FSI organizations are already transforming their data operations with Delphix. See how companies like Fannie Mae accelerate innovation while maintaining the highest security standards. Try Delphix and see how it automates the delivery of high-quality, compliant test data. Request a no-pressure demo today of the next generation of test data management solutions.