Imported from previous forum
[ original email was from Charles Easton - upchuk@optonline.net ]
Does anyone know of documented test conditions for benchmarking performance? Many products say they can do 10,000 m/s, but I am wondering what this means.
Do they only tracking sending messages?
Do they track how long it takes the remote side to receive the messages?
Do they track how long a round-trip of a message takes?
Do the have a minimum size for a message?
Thanks…
Does anyone know of documented test conditions for benchmarking
performance? Many products say they can do 10,000 m/s, but I am
wondering what this means.Do they only tracking sending messages? Do they track how long it takes
the remote side to receive the messages? Do they track how long a round-
trip of a message takes? Do the have a minimum size for a message?Thanks…
Mr. Easton:
The black art of benchmarking FIX has a long history and to answer your wonderment … everyone should question what the numbers mean. Of course, the reputable FIX vendors all have benchmarks and they will tell you their best benchmarks under the best conditions. I suspect everyone includes admin messages in the total.
FIX engines operate within an infrastructure increasingly distributed. Buy side and sell sides will have different performance loads. So to answer your first and second questions buy sides are easier to simulate than sell sides. Daily trading is usually bursty (a Poisson process) while benchmarking may be done under a sustained transmission. The messages may be committed and replicated before being sent. Infrastructure middleware, in my experience, affects throughput as well. Databases, OS’es and hardware configurations may cause latency issues. The more reliability like message parsing or commits and replication, the lower your throughput will be. Usually IOI and Order to Execution reports are simulated. I leave out network and load balancing.
We got a product I was managing a few years back to execute 3000 messages per second using flat files for store with parsing on and no commits round trip(order <> execution report). Given Moore’s, Law I suspect that 10,000 messages / second now is not out of the question but you should ask how they got those numbers.
Please refer to Ryan Pierce’s insightful messages of 2001.
Best Regards
I have seen vendors tout high performance but this is of no relevance if the systems on each end cannot handle it or persist it and deal with it at the same speed…
Does anyone know of documented test conditions for benchmarking
performance? Many products say they can do 10,000 m/s, but I am
wondering what this means.Do they only tracking sending messages? Do they track how long it
takes the remote side to receive the messages? Do they track how long
a round- trip of a message takes? Do the have a minimum size for a
message?Thanks…
Mr. Easton: The black art of benchmarking FIX has a long history and to
answer your wonderment … everyone should question what the numbers
mean. Of course, the reputable FIX vendors all have benchmarks and
they will tell you their best benchmarks under the best conditions.
I suspect everyone includes admin messages in the total. FIX engines
operate within an infrastructure increasingly distributed. Buy side
and sell sides will have different performance loads. So to answer
your first and second questions buy sides are easier to simulate
than sell sides. Daily trading is usually bursty (a Poisson process)
while benchmarking may be done under a sustained transmission. The
messages may be committed and replicated before being sent.
Infrastructure middleware, in my experience, affects throughput as
well. Databases, OS’es and hardware configurations may cause latency
issues. The more reliability like message parsing or commits and
replication, the lower your throughput will be. Usually IOI and
Order to Execution reports are simulated. I leave out network and
load balancing. We got a product I was managing a few years back to
execute 3000 messages per second using flat files for store with
parsing on and no commits round trip(order <> execution report).
Given Moore’s, Law I suspect that 10,000 messages / second now is
not out of the question but you should ask how they got those
numbers. Please refer to Ryan Pierce’s insightful messages of 2001.Best Regards
[ original email was from Charles Easton - upchuk@optonline.net ]
I started a FIX library projet at SourceForge and wanted to follow some standard approach for benchmarking performance. So I am not benchmarking another product to verify it can actually sustain the rate they claim. This is why I am interested in how to benchmark a FIX engine.
I would be great if everyone followed some standard approach for bechmarking and ofcoarse include the hardware used. Then I could see how good or bad my library works.
PS: If you’re interested in seeing the product just visit the product discussion form for a link. I am always intersted in comments/suggestions.
I have seen vendors tout high performance but this is of no relevance if
the systems on each end cannot handle it or persist it and deal with it
at the same speed…Does anyone know of documented test conditions for benchmarking
performance? Many products say they can do 10,000 m/s, but I am
wondering what this means.Do they only tracking sending messages? Do they track how long it
takes the remote side to receive the messages? Do they track how
long a round- trip of a message takes? Do the have a minimum size
for a message?Thanks…
Mr. Easton: The black art of benchmarking FIX has a long history and
to answer your wonderment … everyone should question what the
numbers mean. Of course, the reputable FIX vendors all have
benchmarks and they will tell you their best benchmarks under the
best conditions. I suspect everyone includes admin messages in the
total. FIX engines operate within an infrastructure increasingly
distributed. Buy side and sell sides will have different
performance loads. So to answer your first and second questions
buy sides are easier to simulate than sell sides. Daily trading is
usually bursty (a Poisson process) while benchmarking may be done
under a sustained transmission. The messages may be committed and
replicated before being sent. Infrastructure middleware, in my
experience, affects throughput as well. Databases, OS’es and
hardware configurations may cause latency issues. The more
reliability like message parsing or commits and replication, the
lower your throughput will be. Usually IOI and Order to Execution
reports are simulated. I leave out network and load balancing. We
got a product I was managing a few years back to execute 3000
messages per second using flat files for store with parsing on and
no commits round trip(order <> execution report). Given Moore’s,
Law I suspect that 10,000 messages / second now is not out of the
question but you should ask how they got those numbers. Please
refer to Ryan Pierce’s insightful messages of 2001.Best Regards
I started a FIX library projet at SourceForge
Charles, are you still active with this project to discuss it some?
Mike
urbansound[at]sbcglobal.net