Imported from previous forum
Jacob, thank you for raising this subject, I think you make some very good and interesting points in relation to the need for more testing, the challenges faced within the automated test environment and the opportunities presented by continuous testing. I just wanted to add to this discussion, because one of the issues we come across as a business is that although a growing number of firms are now conducting automated functional testing, automated coverage is often sparse when it comes to performance testing.
Trading system development teams are responsible for pushing out new applications, or updates to applications, and they need to understand the performance impact of the changes they have made, but often when it comes to performance and stability there is an element of “fingers crossed”. System implementers should measure performance against the metrics that matter to their business and then test how their software works in a stressed, but realistic environment. However, we find they frequently face two key challenges in doing so:
- Cost constraints, which limit the degree to which peak volumes can be simulated within the test environment
- Access to all of the testing information from one release to the next. The ad-hoc nature of much performance testing means that comparing results with historical norms for new roll-outs is often difficult, especially when testing over extended periods of time
These challenges can be overcome by ensuring that, as far as possible the volume spikes that a volatile market can present are effectively represented within the test phase. Additionally, by providing automated comparisons with detailed historical records, engineers can gain the insight needed to effectively examine performance relative to previous releases under the same test conditions. This is effective even where test environment resources are more limited. By implementing automated testing practices that include representative volume levels and comparing these automatically with previous test results, firms can take significant steps forward in achieving these goals. If performance testing could become more automated, firms would then benefit from reduced operational risk and increased efficiencies.
If you roll the clock back 10-15 years any form of automated, repeatable testing was limited, however whilst we have seen much improvement in functional testing through automated test suites, we possibly still have quite a way to go before automated, orderly and repeatable testing becomes the norm when examining performance and the stability of systems in real-world market conditions.
Read the post on the Lasalletech blog.
Trading System Failures Cannot Be Our Norm
Testing. Everybody responsible for trading systems knows they need to do more of it. Few have the time to do it right. Fewer have the resources to commit to it. The electronic trading industry has suffered a rash of system failures over the past few years that have each pointed to the need for more testing. Time and again it has been shown that the risk of not investing in testing is too large to ignore, yet this process still goes overlooked and has gone largely unchanged for close to 20 years.
Industry experts agree that automating the testing process is the most cost-effective approach, although it usually requires a large upfront investment. Few organizations have the internal resources to implement it properly.
The issues with automated testing
Those who have built substantial automated test environments know that automated testing is not always a silver bullet. The hardest part is finding a team with the expertise to develop a suite of automated tests. These individuals must have great knowledge of the domain, a grasp of programming concepts, and experience with scripting. Testers with a combination of these skills do not stay in testing roles for long. Outsourcing or bringing in consultants could be a good option, but the specialized domain knowledge and familiarity with your environment walks out the door as soon as the project is over.
An often overlooked area of automated testing is the cost associated with maintenance. The maintenance costs of hosting and developing an automated test environment can creep in from many places. Trading systems are always changing, driven by new regulations, functionality, or customer requirements. Unless a good process is in place to identify new use cases that should have automated tests, the test suite may become stale quickly. Test data is another source of maintenance costs. Maintaining static data like instruments and accounts is straightforward for equities. For those working in futures, options, or fixed income you have to deal with expiring contracts and other date-related events.
For firms routing orders, market data is another point of contention. There are cases where prices are driven by live market prices, or data from external test environments. In the best case this market data can be controlled within your own test environment. More often pricing data is subject to counterparty systems such as exchange test environments.
At the end of the day, a simple suite of functional tests is great for catching simple bugs in calculations and field mapping. The defects that are difficult to reproduce, such as race conditions and concurrency issues, need a completely different approach to automated testing. “Flash crashes do not result from glitches, evil spirits, or sunspots.” says Bob Binder, author of Testing Object-Oriented Systems. “They happen when an unusual pattern of trading volume and message flow provokes a latent behavior that provokes even worse latent behavior. Traditional testing approaches, even stress testing, cannot trigger this kind of failure. But, high-volume and high-variation test suites, running in a realistic test environment, can.”
The bright side
When it comes to automated testing in the electronic trading industry, we are quite fortunate. The business cases are well-defined. There are standard APIs like FIX that are easy to drive test cases through. The question you have to ask yourself is why is automated testing not easier than it currently is?
The answer is that it not only could be easier - it should be easier. There are many areas where efficiencies can be gained. Most test automation projects in our industry entail rewriting the same tests over and over again. The differences between similar organizations cover less than 20% of the tests that are written. Should you really pay a consulting firm to rewrite the same test they have been writing at different organizations for the past decade?
Continuous Testing
Electronic trading firms have achieved productivity gains by employing continuous improvement processes like Agile and Continuous Delivery. In a similar vein, continuous testing is needed to ensure the quality of our trading systems.
A continuous testing strategy for electronic trading takes into account changes in trading system behavior, test data, and counterparty behavior. In this way the tests are always up to date and comprehensive. Operators constantly receive feedback as to the quality of their trading system.
Continuous testing in the electronic trading industry is achievable through the combination of well-defined business cases and easy to test interfaces. Having well-defined business cases provides the ability to automate the full testing lifecycle. Identifying system behavior, writing tests and managing test data can all be automated. Having standard messaging APIs like FIX makes simulating counterparty behavior and virtualizing services possible.
Those responsible for reliable operation of trading systems should expect more value from their testing services and use higher levels of automation to improve system reliability and integrity and reduce operational risks. The highest value, lowest risk and most sustainable approach is continuous testing. Continuous testing automatically takes into account changes in trading system environments over time and provides a greater breadth and depth of testing than any other approach to testing.