Exchange Test platforms

, ,
1

Imported from previous forum

2

[ original email was from Sreedhar Tulluri - stulluri@paxclearing.com ]
Having access to a reliable exchange test platforms for all trading activities, is a real challenge.
We are at the mercy of the exchange to provide a reliable,fully automated test servers.

Client’s system that works on exchange test platform may not work in their production environment because
the rules/validations and the version of software running on test are often different from production.
Also, the test servers are not completely automated. It requires manual intervention from tech support
to test certain test cases.(simple cases like getting partial fills , UR OUTs etc. - not kidding)

Here are the typical problems with Exchange test environments:
- Lack of proper test beds for clients - Often find ourselves in dev servers
- Test servers are not close to production environment in most cases
- Minimal support for test servers
- Lack of test scripts to test different scenarios of order execution
- Availability of test environment - Never available and requires advance notice
to get access to test platforms - Can not test emergency fixes
- Requires manual intervention on Exchange end to replicate certain scenarios
- Test servers are not automatic at all
- Do not allow regression testing

What does it take for an exchange to provide a real test environment, which is
completely automatic, always available and reliable that mimics their
production platform ? Is it competetion or technology ?

I think its competition !

Exchanges are too lazy to enhance their systems and their customers are not going
anywhere. Their help desk is a joke - it took an exchange help desk about 10min to track down an order in
production(I don’t want to name the exchange here) - just tells you what kind of systems they have in place.

There are few vendors out there supporting testing platforms for FIX/CMS protocols but not Exchange
specific protocol extensions. Also, they don’t support market making,trade match/allocations etc.

What we should have is product(an open source is nice) which provides a simulation
environment for all ECN’s/Exchanges supporting all exchange interfaces(FIX or CMS or native API)
for order flow/market making/trade match/price feed etc - A one stop shop for testing all trading related
activities on major exchanges.

The following features would be fine:
- Support for all exchange APIs - FIX/CMS/native/Socket etc.
- Access to test platforms of all major exchanges for the following purposes:
Equity/Derivative Order Entry (FIX/CMS etc.),
Remote Market Making,Autoexecs, Trade executions etc.
- Tests beds that mimic production environment
- Allow regression testing
- 24X7 availability
- Technical support
- Support to at least 2 production versions

Any ideas of such product?

3

Having access to a reliable exchange test platforms for all trading
activities, is a real challenge. We are at the mercy of the exchange to
provide a reliable,fully automated test servers.

Client’s system that works on exchange test platform may not work in
their production environment because the rules/validations and the
version of software running on test are often different from production.
Also, the test servers are not completely automated. It requires manual
intervention from tech support to test certain test cases.(simple cases
like getting partial fills , UR OUTs etc. - not kidding)

Here are the typical problems with Exchange test environments:
- Lack of proper test beds for clients - Often find
ourselves in dev servers
- Test servers are not close to production environment
in most cases
- Minimal support for test servers
- Lack of test scripts to test different scenarios of
order execution
- Availability of test environment - Never available and
requires advance notice to get access to test
platforms - Can not test emergency fixes
- Requires manual intervention on Exchange end to
replicate certain scenarios
- Test servers are not automatic at all
- Do not allow regression testing

What does it take for an exchange to provide a real test environment,
which is completely automatic, always available and reliable that mimics
their production platform ? Is it competetion or technology ?

I think its competition !

Exchanges are too lazy to enhance their systems and their customers are
not going anywhere. Their help desk is a joke - it took an exchange
help desk about 10min to track down an order in production(I don’t want
to name the exchange here) - just tells you what kind of systems they
have in place.

There are few vendors out there supporting testing platforms for FIX/CMS
protocols but not Exchange specific protocol extensions. Also, they
don’t support market making,trade match/allocations etc.

What we should have is product(an open source is nice) which provides a
simulation environment for all ECN’s/Exchanges supporting all exchange
interfaces(FIX or CMS or native API) for order flow/market making/trade
match/price feed etc - A one stop shop for testing all trading related
activities on major exchanges.

The following features would be fine:
- Support for all exchange APIs - FIX/CMS/native/Socket etc.
- Access to test platforms of all major exchanges for the
following purposes: Equity/Derivative Order Entry (FIX/CMS
etc.), Remote Market Making,Autoexecs, Trade executions etc.
- Tests beds that mimic production environment
- Allow regression testing
- 24X7 availability
- Technical support
- Support to at least 2 production versions

Any ideas of such product?
Yes, there are several companies that have these types of tools in various forms. But, we are evaluating a tool that pretty much does everything you would want from an exchange simulator (ECN, SUMO, NASD/SIAC, MKT DATA, etc.) and a fix client simulator. It’s a product called TradeTest and several financial firms have been using it including ARCA and Sunguard. The company that makes it is call Objective Concepts and here’s contact info and website:
Objective Concepts, L.L.C.

Tel: 908.206-9517 Cell: 917.855-2740 Fax: 908.354-5514
One Gateway Center, Suite 2600, Newark, N.J. 07102
website: http://www.objective-concepts.com/

The tool delivers what it claims to, the only complaint I have is that the user interface is a little clunky but they’re planning on improving that at some point. Otherwise the guts of the system is robust. They also provide very good support and are quick to turn around any changes that may be needed. If you have an IT department that is serious about testing and willing to pay for it (it’s not cheap), then I think this is the tool that you’re looking for. I could go into a laundry list of reasons, but I’ll let you read the material and make your own decision.

4

Having access to a reliable exchange test platforms for all trading
activities, is a real challenge. We are at the mercy of the exchange to
provide a reliable,fully automated test servers.

Client’s system that works on exchange test platform may not work in
their production environment because the rules/validations and the
version of software running on test are often different from production.
Also, the test servers are not completely automated. It requires manual
intervention from tech support to test certain test cases.(simple cases
like getting partial fills , UR OUTs etc. - not kidding)

Here are the typical problems with Exchange test environments:
- Lack of proper test beds for clients - Often find
ourselves in dev servers
- Test servers are not close to production environment
in most cases
- Minimal support for test servers
- Lack of test scripts to test different scenarios of
order execution
- Availability of test environment - Never available and
requires advance notice to get access to test
platforms - Can not test emergency fixes
- Requires manual intervention on Exchange end to
replicate certain scenarios
- Test servers are not automatic at all
- Do not allow regression testing

What does it take for an exchange to provide a real test environment,
which is completely automatic, always available and reliable that mimics
their production platform ? Is it competetion or technology ?

I think its competition !

Exchanges are too lazy to enhance their systems and their customers are
not going anywhere. Their help desk is a joke - it took an exchange
help desk about 10min to track down an order in production(I don’t want
to name the exchange here) - just tells you what kind of systems they
have in place.

There are few vendors out there supporting testing platforms for FIX/CMS
protocols but not Exchange specific protocol extensions. Also, they
don’t support market making,trade match/allocations etc.

What we should have is product(an open source is nice) which provides a
simulation environment for all ECN’s/Exchanges supporting all exchange
interfaces(FIX or CMS or native API) for order flow/market making/trade
match/price feed etc - A one stop shop for testing all trading related
activities on major exchanges.

The following features would be fine:
- Support for all exchange APIs - FIX/CMS/native/Socket etc.
- Access to test platforms of all major exchanges for the
following purposes: Equity/Derivative Order Entry (FIX/CMS
etc.), Remote Market Making,Autoexecs, Trade executions etc.
- Tests beds that mimic production environment
- Allow regression testing
- 24X7 availability
- Technical support
- Support to at least 2 production versions

Any ideas of such product?

Sreedhar,

Aegis Software’s Exchange Simulator supports the specific functionality you are looking for and is currently running in about four dozen of the biggest buy and sell-side firms in the industry. For additional information, feel free to visit our website at www.aegisoft.com and contact me offline to discuss.

Regards,
Steve

5

[ original email was from Sam Johnson - sam.johnson@transacttools.net ]
> Having access to a reliable exchange test platforms for all trading

activities, is a real challenge. We are at the mercy of the exchange to
provide a reliable,fully automated test servers.

Client’s system that works on exchange test platform may not work in
their production environment because the rules/validations and the
version of software running on test are often different from production.
Also, the test servers are not completely automated. It requires manual
intervention from tech support to test certain test cases.(simple cases
like getting partial fills , UR OUTs etc. - not kidding)

Here are the typical problems with Exchange test environments:
- Lack of proper test beds for clients - Often find
ourselves in dev servers
- Test servers are not close to production environment
in most cases
- Minimal support for test servers
- Lack of test scripts to test different scenarios of
order execution
- Availability of test environment - Never available and
requires advance notice to get access to test
platforms - Can not test emergency fixes
- Requires manual intervention on Exchange end to
replicate certain scenarios
- Test servers are not automatic at all
- Do not allow regression testing

What does it take for an exchange to provide a real test environment,
which is completely automatic, always available and reliable that mimics
their production platform ? Is it competetion or technology ?

I think its competition !

Exchanges are too lazy to enhance their systems and their customers are
not going anywhere. Their help desk is a joke - it took an exchange
help desk about 10min to track down an order in production(I don’t want
to name the exchange here) - just tells you what kind of systems they
have in place.

There are few vendors out there supporting testing platforms for FIX/CMS
protocols but not Exchange specific protocol extensions. Also, they
don’t support market making,trade match/allocations etc.

What we should have is product(an open source is nice) which provides a
simulation environment for all ECN’s/Exchanges supporting all exchange
interfaces(FIX or CMS or native API) for order flow/market making/trade
match/price feed etc - A one stop shop for testing all trading related
activities on major exchanges.

The following features would be fine:
- Support for all exchange APIs - FIX/CMS/native/Socket etc.
- Access to test platforms of all major exchanges for the
following purposes: Equity/Derivative Order Entry (FIX/CMS
etc.), Remote Market Making,Autoexecs, Trade executions etc.
- Tests beds that mimic production environment
- Allow regression testing
- 24X7 availability
- Technical support
- Support to at least 2 production versions

Any ideas of such product?

I’m not sure I agree with the “exchanges are lazy and their customers are not going anywhere” comments. In fact, a handful of exchanges (e.g. Arca, CME, Instinet, and others) do provide thorough, automated certification services to their customers. Based on their success, as well as ever-increasing competition in the marketplace, I suspect that more will follow suit. The marketplace is changing, and exchanges are competing for business.

For an automated testing/certification platform to work, it has to be kept absolutely in sync with the production system. This is increasingly important as exchanges and execution venues are inventing new order types and new order handling capabilities at an increasing rate. For a vendor to provide “all the specific functionality you’re looking for” would require that he have a very close relationship with the exchange to ensure that his platform is constantly in sync – or you run into the exact scenario you outlined above.

We have been in the business of providing thorough, automated certification solutions to exchanges for over four years, so we have first-hand experience. Many firms use our solutions in-house to simulate exchange and vendor interfaces – specifically because we have first-hand experience with these exchanges – but although this provides a lot of value, it is not the same thing as certification and it doesn’t address all of the difficult issues you outline above.

6

[ original email was from Sreedhar Tulluri - stulluri@paxclearing.com ]
I was referring to “Exchanges” not ECNs in my comments.

You are absoultely true. ECNs are far ahead of all exchanges and they have better testing and certification environments - above all they are
more stable and perform well. Hopefully exchanges will learn few tricks
from these ECNs and get better at some point in future.

Yes, it is difficult for a vendor to provide testing tools for each and every exchange specific protocol.
But I am sure they are several vendors and individuals who built tools
of some nature to address atleast some of the exchange protocols.
Collectively, they address most of the exchanges.

The idea is to create an open source environment where vendors and individuals contribute their testing tools/findings etc. , which could
potentially address the issue at a larger scale.

If one can get a FIX server for free(Thanks to QuickFix!) , I am sure its possible to get testing tools and possibly more in open source
arena.

Having access to a reliable exchange test platforms for all trading
activities, is a real challenge. We are at the mercy of the exchange
to provide a reliable,fully automated test servers.

Client’s system that works on exchange test platform may not work in
their production environment because the rules/validations and the
version of software running on test are often different from
production. Also, the test servers are not completely automated. It
requires manual intervention from tech support to test certain test
cases.(simple cases like getting partial fills , UR OUTs etc. - not
kidding)

Here are the typical problems with Exchange test environments:
- Lack of proper test beds for clients - Often find
ourselves in dev servers
- Test servers are not close to production environment
in most cases
- Minimal support for test servers
- Lack of test scripts to test different scenarios of
order execution
- Availability of test environment - Never available
and requires advance notice to get access to test
platforms - Can not test emergency fixes
- Requires manual intervention on Exchange end to
replicate certain scenarios
- Test servers are not automatic at all
- Do not allow regression testing

What does it take for an exchange to provide a real test environment,
which is completely automatic, always available and reliable that
mimics their production platform ? Is it competetion or technology ?

I think its competition !

Exchanges are too lazy to enhance their systems and their customers
are not going anywhere. Their help desk is a joke - it took an
exchange help desk about 10min to track down an order in production(I
don’t want to name the exchange here) - just tells you what kind of
systems they have in place.

There are few vendors out there supporting testing platforms for
FIX/CMS protocols but not Exchange specific protocol extensions. Also,
they don’t support market making,trade match/allocations etc.

What we should have is product(an open source is nice) which provides
a simulation environment for all ECN’s/Exchanges supporting all
exchange interfaces(FIX or CMS or native API) for order flow/market
making/trade match/price feed etc - A one stop shop for testing all
trading related activities on major exchanges.

The following features would be fine:
- Support for all exchange APIs - FIX/CMS/native/Socket etc.
- Access to test platforms of all major exchanges for the
following purposes: Equity/Derivative Order Entry (FIX/CMS
etc.), Remote Market Making,Autoexecs, Trade executions etc.
- Tests beds that mimic production environment
- Allow regression testing
- 24X7 availability
- Technical support
- Support to at least 2 production versions

Any ideas of such product?

I’m not sure I agree with the “exchanges are lazy and their customers
are not going anywhere” comments. In fact, a handful of exchanges (e.g.
Arca, CME, Instinet, and others) do provide thorough, automated
certification services to their customers. Based on their success, as
well as ever-increasing competition in the marketplace, I suspect that
more will follow suit. The marketplace is changing, and exchanges are
competing for business.

For an automated testing/certification platform to work, it has to be
kept absolutely in sync with the production system. This is increasingly
important as exchanges and execution venues are inventing new order
types and new order handling capabilities at an increasing rate. For a
vendor to provide “all the specific functionality you’re looking for”
would require that he have a very close relationship with the exchange
to ensure that his platform is constantly in sync – or you run into the
exact scenario you outlined above.

We have been in the business of providing thorough, automated
certification solutions to exchanges for over four years, so we have first-
hand experience. Many firms use our solutions in-house to simulate
exchange and vendor interfaces – specifically because we have first-
hand experience with these exchanges – but although this provides a lot
of value, it is not the same thing as certification and it doesn’t
address all of the difficult issues you outline above.

7

Yes, it is difficult for a vendor to provide testing tools for each and
every exchange specific protocol. But I am sure they are several vendors
and individuals who built tools of some nature to address atleast some
of the exchange protocols. Collectively, they address most of the
exchanges.

The idea is to create an open source environment where vendors and
individuals contribute their testing tools/findings etc. , which could
potentially address the issue at a larger scale.

If one can get a FIX server for free(Thanks to QuickFix!) , I am
sure its possible to get testing tools and possibly more in open
source arena.

Sreedhar, sounds like a very good idea to me. Just have a look at the WikiFIX on http://www.quickfixengine.org to collect some more ideas, exchanges and test scenario.

I am quite sure you could convince Oren to setup such a testing environment if enough material is available. QuickFIX may be well suited to provide a free, open source testing service.

Cheers, Jörg