Imported from previous forum
Greetings FIX community,
We’re interested in FIX version conversion functionality- primarily though not limited to-- converting 4.0-clients’ orders to 4.2 (internal order destinations’ interfaces are 4.2), and vice-versa; converting 4.2 execution reports back to 4.0.
It would seem this is a common requirement, although (so far) I’ve found no native support in 3rd-party products or in open-source solutions.
Obviously, many software packages support custom implementations where any proprietary message transformation logic can be custom coded. But converting from standard, published protocol versions should be as easy as:
FIXMessage execMsg40 = convertExecReport(execMsg42, “42”, “40”);
like:
public FIXMessage convertExecReport(FIXMessage execMsg, String fromVersion, String toVersion) {
// Leverage existing validation modules to do conversion:
// Logic to derive data for any additional required fields (generally 4.0->4.2)
// Remove invalid fields (generally 4.2->4.0)
// Logic to convert datatypes (e.g., ExecID: String->Int)
// Return converted message
}
Or better yet, conversion would be performed implicitly based on the source and destination FIX versions.
Any relevant info, ideas, and experiences are kindly appreciated.
Hi Paul,
Aegis’ Athena Gateway Server supports this exact functionality. Feel free to shoot me an email or give me a call offline to discuss. You can also access additional information about our product offerings at www.aegisoft.com.
Thanks,
Steve
Greetings FIX community, We’re interested in FIX version conversion functionality-
primarily though not limited to-- converting 4.0-clients’ orders to 4.2
(internal order destinations’ interfaces are 4.2), and vice-versa;
converting 4.2 execution reports back to 4.0.It would seem this is a common requirement, although (so far) I’ve found
no native support in 3rd-party products or in open-source solutions.Obviously, many software packages support custom implementations
where any proprietary message transformation logic can be custom
coded. But converting from standard, published protocol versions
should be as easy as:FIXMessage execMsg40 = convertExecReport(execMsg42, “42”, “40”);
like: public FIXMessage convertExecReport(FIXMessage execMsg, String
fromVersion, String toVersion) { // Leverage existing validation modules
to do conversion: // Logic to derive data for any additional required
fields (generally 4.0->4.2) // Remove invalid fields (generally 4.2-4.0) // Logic to convert datatypes (e.g., ExecID: String->Int) //
Return converted message }Or better yet, conversion would be performed implicitly based on the
source and destination FIX versions.Any relevant info, ideas, and experiences are kindly appreciated.
Paul,
You are correct in that converting between different versions is a common requirement, and the ability to translate messages between different versions is a fairly standard feature for FIX engines.
In the Transacttools TCM engine, you can configure translations to occur as several different points during message processing, and your translations can be either configuration driven via XML, or code driven with Java or Javascript. The translations can occur conditionally based on the source and target versions of FIX.
For translating messages from 4.2 to 4.0, the most difficult part is mapping the ID values from Strings to integer values. Because these IDs are used by counter parties to match and reference previous messages, your implementation has to ensure that mapping is consistent and unique.
One simple way to do this is to keep a hash map of values, mapping a String ID to an integer ID during the translation, and assigning new integer IDs in numeric order as the messages are processed. Then if a future message references a previously seen ID, the assigned numeric value can be looked up on the map during the translation.
In addition, you need to make sure that your implementation can recover properly from an unexpected failure, doesn’t severely impact your performance, and can reverse this mapping for inbound messages as needed. Another consideration is how long you have to keep your mappings, as GTC and GTD orders may potentially stay active for many months, or even years. Add in other message types with ID values, such as IOIs, ATs, or Quotes, and you can see that the problem is not such a simple matter of replacing a string with a number.
In the case of TCM, we have a fairly extensive API built into the messaging engine that offers developers quite a bit of flexibility to implement such translations once you understand your particular requirements. We also have a professional services group that can provide advice and assistance as needed, for everything from understanding requirements to actual implementations, deployment, and product support.
Feel free to drop me an email if you have any further questions.
Andrew
Greetings FIX community, We’re interested in FIX version conversion functionality-
primarily though not limited to-- converting 4.0-clients’ orders to 4.2
(internal order destinations’ interfaces are 4.2), and vice-versa;
converting 4.2 execution reports back to 4.0.It would seem this is a common requirement, although (so far) I’ve found
no native support in 3rd-party products or in open-source solutions.Obviously, many software packages support custom implementations
where any proprietary message transformation logic can be custom
coded. But converting from standard, published protocol versions
should be as easy as:FIXMessage execMsg40 = convertExecReport(execMsg42, “42”, “40”);
like: public FIXMessage convertExecReport(FIXMessage execMsg, String
fromVersion, String toVersion) { // Leverage existing validation modules
to do conversion: // Logic to derive data for any additional required
fields (generally 4.0->4.2) // Remove invalid fields (generally 4.2-4.0) // Logic to convert datatypes (e.g., ExecID: String->Int) //
Return converted message }Or better yet, conversion would be performed implicitly based on the
source and destination FIX versions.Any relevant info, ideas, and experiences are kindly appreciated.
[ original email was from Toby Corballis - toby.corballis@rapidaddition.co.uk ]
At Rapid Addition we’ve already built translation mappings between the different 4.x and 5.0 versions of FIX (4.0 to 4.2, 4.4 to 4.2, etc) using Microsoft’s BizTalk tool. BizTalk itself doesn’t provide this functionality off-the-shelf but there’s nothing to stop you using that - or another - tool to do it yourself. Alternatively you could use a tool that already has it. I’m sure there are other vendors out there who’ve done this too.
Feel free to get in touch if I can help further.
Toby
Greetings FIX community, We’re interested in FIX version conversion functionality-
primarily though not limited to-- converting 4.0-clients’ orders to 4.2
(internal order destinations’ interfaces are 4.2), and vice-versa;
converting 4.2 execution reports back to 4.0.It would seem this is a common requirement, although (so far) I’ve found
no native support in 3rd-party products or in open-source solutions.Obviously, many software packages support custom implementations
where any proprietary message transformation logic can be custom
coded. But converting from standard, published protocol versions
should be as easy as:FIXMessage execMsg40 = convertExecReport(execMsg42, “42”, “40”);
like: public FIXMessage convertExecReport(FIXMessage execMsg, String
fromVersion, String toVersion) { // Leverage existing validation modules
to do conversion: // Logic to derive data for any additional required
fields (generally 4.0->4.2) // Remove invalid fields (generally 4.2-4.0) // Logic to convert datatypes (e.g., ExecID: String->Int) //
Return converted message }Or better yet, conversion would be performed implicitly based on the
source and destination FIX versions.Any relevant info, ideas, and experiences are kindly appreciated.
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> Any relevant info, ideas, and experiences are kindly appreciated.
I think I’ll break the mold here and offer some advice without also mentioning a product and/or vendor. 
What you’re asking about is a rather difficult and complex issue. While I agree with Andrew that ExecID mapping is a complexity, it’s only the tip of the iceberg. I see four classes of issues:
#1. Simple conversions. Moving from individual fields to the Parties block might appear somewhat complex on the surface, but it still can be done as a stateless operation. Some conversions require static lookup tables, like Reuters codes to MICs. But still, one could write a function that does the conversion, and it would require no dynamic state persistence.
#2. As Andrew suggested, mapping string ExecIDs to integers is more complex, as it requires persistent state storage. However, the model here is still simple; a message can still be translated, independently of all the other messages, so long as the persistent ExecID mapping is maintained.
The function model of #1 will not suffice. Persistent state storage is needed. A simple database-like object is needed to provide persistence. As has been said, it must be reliable to survive failures, and must keep stuff around for the lifetime of the order, particularly in the case of GTCs.
#3. Differences regarding state changes between FIX versions. This is the most difficult of the issues, in my opinion. As FIX has evolved, it has become more specific, and as such it requires more state and more precise state. FIX 4.0 was simple, although highly ambiguous. FIX 4.1 split out OrdStatus into two fields: OrdStatus and ExecType. Also in 4.1, the concept of ClOrdID chaining in Execution Reports (via ClOrdID and OrigClOrdID) was introduced. And later, one state of Pending Cxl/Replace diverged into distinct states of Pending Cancel and Pending Replace.
In this case, translating from FIX message to FIX message via the #2 method above won’t work. A simple message in -> message out approach, even with a simple database to map IDs, can’t do it. Example: An Execution Report in FIX 4.1 is received with OrdStatus = Pending Cxl/Replace. The conversion function has no way of knowing whether a change, a cancel, or both are pending! Even a database with simple ID field mapping doesn’t tell us what we need to know.
In this case, what is needed is some concept of context, i.e. a state machine. If a record is kept of what cancels and/or changes are pending on an order, then it can be deduced whether Pending Cancel or Pending Replace should be sent.
Of course, this state machine must process FIX messages in chronological order. If a change was submitted and accepted, but the state machine doesn’t know about the accept, it would be tempted to think that the order is still in Pending Replace state.
In short, one often can’t convert one FIX message to another in a vacuum; in these cases one must know some details on state.
#4. Impossibilities. For example, ExecType = Restated was introduced in FIX 4.2. If a FIX 4.2 sell-side sends ExecType = Restated to report a change in a GTC order due to a corporate action, that simply cannot be converted to, say, a FIX 4.1 buy-side, at least not while following the FIX standard. (Of course, these aren’t really impossible; non-standard solutions can always be devised. There is a time-honored tradition of pulling features from newer versions of the protocol into older versions.)
My opinions:
Stateless FIX version translation, especially with older spec versions, is largely a myth. Something has to keep state somewhere.
Converting down to a lower version of FIX is usually easier than converting up to a higher version. One can always drop state; one can’t create state out of thin air. Unfortunately, connecting two systems with different versions requires both a downward and an upward conversion.
In general, I find it easier to handle conversion if the buy-side is the lower version and the sell-side the higher. Most of the additional required state was added to the Execution Report, not the New Order - Single. Now if the buy-side is a higher version than the sell-side, that might be painful, depending on the versions. Trying to turn a FIX 4.0 Execution Report into a FIX 4.4 Execution Report, while possible, will require a lot of state, and is a more difficult task.
Personally, I find it easier to design an OMS or EMS to be capable of forming and receiving FIX messages in any version, than having the OMS/EMS speak one FIX version and letting something external, like the FIX engine, do conversion. The OMS/EMS has to keep business-level state about the orders anyway, so this reduces duplication of state. The down-side is that it means work must be done to the OMS/EMS to upgrade FIX versions.
Also, I find it helpful to define an enumeration of FIX version numbers. Treat it as an integer. Then code to support multiple versions becomes, in my opinion, easier to write and maintain, i.e.
if ( FIXVersion >= FIX_VERSION_4_4 ) {
// Add a field introduced in 4.4
}
if ( FIXVersion >= FIX_VERSION_4_2 ) {
// Add a field introduced in 4.2
}
if ( FIXVersion >= FIX_VERSION_4_1 ) {
// Add both ExecID and OrdStatus
} else {
// Add just OrdStatus
}
I wanted to thank Ryan for this excellent analysis of the issues involved with the conversion between messages. I concur with Ryan that stateless conversion is not possible even for minimal usage of FIX. This typically translates to complexity, time and effort.
I would like to add that one should carefully analyze the effort needed for the conversion and compare that with the effort it would take to move to FIX 5.0. One of the crucial features of this latest version of FIX is its ability to handle different versions of FIX within a single framework. It allows to use different versions with different counterparties and does not force a conversion for all but those that happen to have the same version.
On top of the one time effort to get a stateful conversion machine to work properly you need to consider the ongoing maintenance effort that is bound to be significantly higher if you try to convert between versions with a single counterparty. It will save you a lot of complexity in your FIX engine if you have multiple versions in parallel, assuming you cannot avoid them in the first place. Ideally, everybody should be moving to FIX 5.0 (and FIX 5.0 messages) as soon as possible but we are all aware of the fact that the lack of a business case usually prevents that.
The last point I would like to make is that complex conversions typically come at the cost of additional latency. Nowadays, fewer people are willing to pay this price and are looking for the fastest access path. My guess would be that this will always be the one without conversion between different FIX versions. With FIX 5.0 you can offer such a “fast” path to all your counterparties. Obviously, native FIX 5.0 is the fastest 
Regards,
Hanno.
Thanks Ryan. Your post provided valuable insights, as usual.
Any relevant info, ideas, and experiences are kindly appreciated.
I think I’ll break the mold here and offer some advice without also
mentioning a product and/or vendor.What you’re asking about is a rather difficult and complex issue. While
I agree with Andrew that ExecID mapping is a complexity, it’s only the
tip of the iceberg. I see four classes of issues:#1. Simple conversions. Moving from individual fields to the Parties
block might appear somewhat complex on the surface, but it still can
be done as a stateless operation. Some conversions require static
lookup tables, like Reuters codes to MICs. But still, one could
write a function that does the conversion, and it would require no
dynamic state persistence.
#2. As Andrew suggested, mapping string ExecIDs to integers is more
complex, as it requires persistent state storage. However, the model
here is still simple; a message can still be translated,
independently of all the other messages, so long as the persistent
ExecID mapping is maintained.
The function model of #1 will not suffice. Persistent state storage is
needed. A simple database-like object is needed to provide persistence.
As has been said, it must be reliable to survive failures, and must
keep stuff around for the lifetime of the order, particularly in the
case of GTCs.#3. Differences regarding state changes between FIX versions. This is
#the most difficult of the issues, in my opinion. As FIX has evolved,
#it has become more specific, and as such it requires more state and
#more precise state. FIX 4.0 was simple, although highly ambiguous. FIX
#4.1 split out OrdStatus into two fields: OrdStatus and ExecType. Also
#in 4.1, the concept of ClOrdID chaining in Execution Reports (via
#ClOrdID and OrigClOrdID) was introduced. And later, one state of
#Pending Cxl/Replace diverged into distinct states of Pending Cancel
#and Pending Replace.In this case, translating from FIX message to FIX message via the #2
method above won’t work. A simple message in → message out approach,
even with a simple database to map IDs, can’t do it. Example: An
Execution Report in FIX 4.1 is received with OrdStatus = Pending
Cxl/Replace. The conversion function has no way of knowing whether a
change, a cancel, or both are pending! Even a database with simple ID
field mapping doesn’t tell us what we need to know.In this case, what is needed is some concept of context, i.e. a state
machine. If a record is kept of what cancels and/or changes are pending
on an order, then it can be deduced whether Pending Cancel or Pending
Replace should be sent.Of course, this state machine must process FIX messages in chronological
order. If a change was submitted and accepted, but the state machine
doesn’t know about the accept, it would be tempted to think that the
order is still in Pending Replace state.In short, one often can’t convert one FIX message to another in a
vacuum; in these cases one must know some details on state.#4. Impossibilities. For example, ExecType = Restated was introduced in
#FIX 4.2. If a FIX 4.2 sell-side sends ExecType = Restated to report a
#change in a GTC order due to a corporate action, that simply cannot be
#converted to, say, a FIX 4.1 buy-side, at least not while following the
#FIX standard. (Of course, these aren’t really impossible; non-standard
#solutions can always be devised. There is a time-honored tradition of
#pulling features from newer versions of the protocol into older
#versions.)My opinions:
Stateless FIX version translation, especially with older spec versions,
is largely a myth. Something has to keep state somewhere.Converting down to a lower version of FIX is usually easier than
converting up to a higher version. One can always drop state; one can’t
create state out of thin air. Unfortunately, connecting two systems with
different versions requires both a downward and an upward conversion.In general, I find it easier to handle conversion if the buy-side is the
lower version and the sell-side the higher. Most of the additional
required state was added to the Execution Report, not the New Order -
Single. Now if the buy-side is a higher version than the sell-side, that
might be painful, depending on the versions. Trying to turn a FIX 4.0
Execution Report into a FIX 4.4 Execution Report, while possible, will
require a lot of state, and is a more difficult task.Personally, I find it easier to design an OMS or EMS to be capable of
forming and receiving FIX messages in any version, than having the
OMS/EMS speak one FIX version and letting something external, like the
FIX engine, do conversion. The OMS/EMS has to keep business-level
state about the orders anyway, so this reduces duplication of state.
The down-side is that it means work must be done to the OMS/EMS to
upgrade FIX versions.Also, I find it helpful to define an enumeration of FIX version numbers.
Treat it as an integer. Then code to support multiple versions becomes,
in my opinion, easier to write and maintain, i.e.if ( FIXVersion >= FIX_VERSION_4_4 ) { // Add a field introduced in 4.4
} if ( FIXVersion >= FIX_VERSION_4_2 ) { // Add a field introduced in
4.2 } if ( FIXVersion >= FIX_VERSION_4_1 ) { // Add both ExecID and
OrdStatus } else { // Add just OrdStatus }
