Imported from previous forum
[ original email was from n d - ndalal@tradeblade.com ]
Hi:
We’re exploring ways to automate our function testing in FIX. So I thought I could borrow from this forum’s collective intelligence:
“How do you do your functional testing? Off-the-shelf tools? Custom built tools? Simpel scripts?”
Thanks for your time!
Being in on the “ground floor” of QuickFIX we naturally leveraged and enhanced the QuickFIX acceptance framework. Now after several years and many, many projects have the ability to automate repeatable functional testing of entire trading platforms including market-data driven applications. It is basically, FIX in expected FIX out (the i an e). We have also extended the frame work to non-fix protocols. So our experience is that the QuickFIX acceptance test runner is a good starting point for creating a full functional testing framework for FIX enabled applications.
However as you mentioned that task of creating the tests as they are is laborious and error prone (believe me I’ve written so many of these tests I know). We have made a few attempts at creating a higher level language to create the tests but thus far those attempts have fallen short. We currently have an internal project to use the QuickFIX data dictionary to drive a UI. So we too are interested in the experience of others.
Jim
Hi:
We’re exploring ways to automate our function testing in FIX. So I
thought I could borrow from this forum’s collective intelligence:“How do you do your functional testing? Off-the-shelf tools? Custom
built tools? Simpel scripts?”Thanks for your time!
There are several vendors that provided automated FIX testing tools. It is not appropriate in this forum to list specific tools - but you can check the vendors page or post a question on specific tools to the products forum. Tools range from simple spreadsheet to FIX integration products all the way up to very high end tools with rich GUI applications and languages for fully automating testing scenarios.
Being in on the “ground floor” of QuickFIX we naturally leveraged and
enhanced the QuickFIX acceptance framework. Now after several years and
many, many projects have the ability to automate repeatable functional
testing of entire trading platforms including market-data driven
applications. It is basically, FIX in expected FIX out (the i an e). We
have also extended the frame work to non-fix protocols. So our
experience is that the QuickFIX acceptance test runner is a good
starting point for creating a full functional testing framework for FIX
enabled applications.However as you mentioned that task of creating the tests as they are
is laborious and error prone (believe me I’ve written so many of these
tests I know). We have made a few attempts at creating a higher level
language to create the tests but thus far those attempts have fallen
short. We currently have an internal project to use the QuickFIX data
dictionary to drive a UI. So we too are interested in the experience
of others. JimHi:
We’re exploring ways to automate our function testing in FIX. So I
thought I could borrow from this forum’s collective intelligence:“How do you do your functional testing? Off-the-shelf tools? Custom
built tools? Simpel scripts?”Thanks for your time!
[ original email was from John Greenan - john.greenan@alignment-systems.com ]
When you say functional - what do you actually mean? And from which side - buy-side or sell side.
Typically a buy-side faces more issues around OMS functionality rather than FIX specifics. A sell side will want to ensure that there are no regressions between software upgrades so that buy-sides do not have to recertify.
There are a number of firms that offer tools - some with colourful names - some of which are very good. As Jim rightly states, this is not the forum for praise or naming and shaming.
In some cases I have seen firms build up massive libraries of Winrunner scripts to automate testing. Then the OMS vendor changes a GUI widget and it all breaks. Then you have two problems to solve and not one. I have also seen consulting firms that build a specific tool for a client and then try and sell that tool as a generic solution.
This is a very big subject - narrow the scope of your question and explain your problem and I am sure that many folks will be able to assist or offer experience.
John
Hi:
We’re exploring ways to automate our function testing in FIX. So I
thought I could borrow from this forum’s collective intelligence:“How do you do your functional testing? Off-the-shelf tools? Custom
built tools? Simpel scripts?”Thanks for your time!
[ original email was from n d - ndalal@tradeblade.com ]
Hi - Thanks for your reply. I was referring mainly to testing sell-side applications of FIX. By functional testing I meant testing of specific business logic in application-level tags as opposed to ‘acceptance’ or FIX-compliance testing.
A very simple example would be: After partially executing 25m out of the total 100m available, does the Execution Report (35=8) have the correct ‘size left’? Did the ‘size left’ adhere to the trade stipulation I had specified?
I understand the restrictions of this forum and for the same reason I was most interested in knowing how sell-side implementations have managed their testing efforts. Some personal experiences / testing-practices recommendations / lessons-learnt / FIX-testing-case-studies would be great.
Thanks to all those who took time to reply so far.
When you say functional - what do you actually mean? And from which side
- buy-side or sell side.
Typically a buy-side faces more issues around OMS functionality rather
than FIX specifics. A sell side will want to ensure that there are no
regressions between software upgrades so that buy-sides do not have to
recertify.There are a number of firms that offer tools - some with colourful names
- some of which are very good. As Jim rightly states, this is not the
forum for praise or naming and shaming.In some cases I have seen firms build up massive libraries of Winrunner
scripts to automate testing. Then the OMS vendor changes a GUI widget
and it all breaks. Then you have two problems to solve and not one. I
have also seen consulting firms that build a specific tool for a client
and then try and sell that tool as a generic solution.This is a very big subject - narrow the scope of your question and
explain your problem and I am sure that many folks will be able to
assist or offer experience.John
Hi:
We’re exploring ways to automate our function testing in FIX. So I
thought I could borrow from this forum’s collective intelligence:“How do you do your functional testing? Off-the-shelf tools? Custom
built tools? Simpel scripts?”Thanks for your time!
.
The Sell Side behavior will be somewhat specific to what you are connecting to. In your example, the outcome will vary between venues, and sometimes between vendors connecting to a same venue.
There are multiple ways of performing the validation.
Here are 2 I saw in the field:
- Create a core scenario which holds the business logic (send message, receive execution report, …) and apply different sets of message validation based on which Sell Side you are connecting to
- Generate your scenario off production log files and replay the logic at will.
Hi - Thanks for your reply. I was referring mainly to testing sell-side
applications of FIX. By functional testing I meant testing of specific
business logic in application-level tags as opposed to ‘acceptance’ or
FIX-compliance testing.A very simple example would be: After partially executing 25m out of the
total 100m available, does the Execution Report (35=8) have the correct
‘size left’? Did the ‘size left’ adhere to the trade stipulation I had
specified?I understand the restrictions of this forum and for the same reason I
was most interested in knowing how sell-side implementations have
managed their testing efforts. Some personal experiences / testing-
practices recommendations / lessons-learnt / FIX-testing-case-studies
would be great.Thanks to all those who took time to reply so far.
. The Sell Side behavior will be somewhat specific to what you are
connecting to. In your example, the outcome will vary between venues,
and sometimes between vendors connecting to a same venue.There are multiple ways of performing the validation. Here are 2 I saw
in the field: 1) Create a core scenario which holds the business logic
(send message, receive execution report, …) and apply different sets
of message validation based on which Sell Side you are connecting to
2) Generate your scenario off production log files and replay the
logic at will.Hi - Thanks for your reply. I was referring mainly to testing sell-
side applications of FIX. By functional testing I meant testing of
specific business logic in application-level tags as opposed to
‘acceptance’ or FIX-compliance testing.A very simple example would be: After partially executing 25m out of
the total 100m available, does the Execution Report (35=8) have the
correct ‘size left’? Did the ‘size left’ adhere to the trade
stipulation I had specified?I understand the restrictions of this forum and for the same reason I
was most interested in knowing how sell-side implementations have
managed their testing efforts. Some personal experiences / testing-
practices recommendations / lessons-learnt / FIX-testing-case-studies
would be great.Thanks to all those who took time to reply so far.
Simple scripts is what I have seen working effectively most often…
Herez my two cents on this though…
Creating “smart test cases” is far more important. I have seen instances wherein enormous effort in testing was done, it gave false sense of comfort and what had to break eventually did!
The developer/BA/tester can try and create smart cases - and then “less is the new more” in terms of testing efforts…