Imported from previous forum
[ original email was from Sam Johnson - sam.johnson@transacttools.net ]
Has FPL ever considered facilitating some objective performance benchmarking of commercial FIX engines? Maybe even as a special supplement to FIX Global? There are an awful lot of vendors claiming to be the fastest, and yet precious little data supporting those claims. They can’t all be the fastest, can they. TransactTools would certainly accept an invitation to participate in something like this. To make it meaningful and helpful to the community, the gamut of benchmarking tests would need to cover a range of scenarios for varying numbers of concurrent connections across a server and on various hardware and operating system platforms. Benchmark tests should reflect real-world scenarios and include the following (throughput AND latency, both):
- message received on the wire, validated, persisted
- message created, populated, sent out over the wire
- message routed without translation: received, validated, persisted, sent
- message routed with translation: received, validated, persisted, translated, validated, persisted, sent
It seems to me that one really valuable service an independent organization like FPL could provide to the community would be to help filter through some of the conflicting (and occcasionally unsubstantiated) claims made by vendors.
Has FPL ever considered facilitating some objective performance
benchmarking of commercial FIX engines? Maybe even as a special >supplement to FIX Global? There are an awful lot of vendors claiming >to be the fastest, and yet precious little data supporting those >claims. They can’t all be the fastest, can they. TransactTools would >certainly accept an invitation to participate in something like this.
Onix Solutions agree completely.
Without detailed description of the hardware test bench as well as benchmarking tests any performance numbers are not too meaningful.
To make it meaningful and helpful to the community, the gamut of
benchmarking tests would need to cover a range of scenarios for
varying numbers of concurrent connections across a server and on
various hardware and operating system platforms. Benchmark tests
should reflect real-world scenarios and include the following
(throughput AND latency, both):
- message received on the wire, validated, persisted
Different persistent storages could be used, e.g. flat files, DBMS, etc. So maybe it could be useful to specify them.
- message created, populated, sent out over the wire
- message routed without translation: received, validated,
persisted, sent
Some engines can validate required fields only, others validates conditionally required fields too. Some engines validate field values, others – don’t. So it could be several tests for different validation modes.
- message routed with translation: received, validated, persisted,
translated, validated, persisted, sent
What do you mean under “translation”? Something like generating Execution Report on each incoming New Order or something else?
Message type and size could significantly influence the benchmark numbers, so it would be interested to run tests with different messages (e.g. no repeating groups, 1-level repeating groups, embedded repeating groups).
Number of messages in the test would play a significant role too, because small numbers of messages could just enjoy CPU and HDD caches.
It could be interesting also to test how the benchmark numbers depend on the number of simultaneous FIX sessions, because some FIX Engines are optimized for multi-session mode, others work better in single-session mode.
It seems to me that one really valuable service an independent
organization like FPL could provide to the community would be to help
filter through some of the conflicting (and occcasionally
unsubstantiated) claims made by vendors.
Onix Solutions would be very glad to accept an invitation in something like this too.
FIXForge .NET FIX Engine Developer Edition set that is available at
www.onixs.biz/downloads.php
already includes open-source benchmarking tests, so maybe they could be useful as a first proof-of-concept prototype.
Kind Regards,
Onix Solutions / www.onixs.biz
[ original email was from Alexander Mologoko - avm@btobits.com ]
We at B2BITS traditionally have used third-party benchmarking through leading hardware vendors in the financial services industry (such as Sun Microsystems).
We now believe that it makes sense to develop benchmarks under FPL governance and engage an independent organization such as hardware
vendor(s) for carrying this test over.
Anyone interested in studying the method and the results of such independent benchmarks of B2BITS FIXAntenna V12 Engine can access a public report at http://b2bits.com/docs/b2bits_independent_performance_tests.pdf
Several thoughts on extending real-world scenarios:
- tests with a mix of data (order-execution-allocation chains for
example) - size of messages is an extremely important factor in FIX benchmarking, therefore the program must maintain a set of reference messages of different type and size (NewOrder-Single, ExecutionReport, basket of 1000 orders, basket of 10000 orders etc), which must be used by all vendors in their benchmarks
- it would also be fair to separately publish benchmark results for engines based on different technologies (i.e. C++ engines, .Net engines, Java engines, exotic platforms)