Imported from previous forum
Hi,
I’d like to know if it’s FIX compliant to implement application sequencing in Order Routing systems?
For example we have a trading backend which has several partitions, and we need to send order into all the partitions. We are planning to initiate one FIX session for sending orders into all the backend partitions, while for each partition, we maintain a application sequence number.
I’m just wondeing if this design is FIX compliant?
Cheers,
LiuAn
I assume that you mean that your trading backend issues partition specific application sequence numbers. Your emphasis on Order Routing systems could also have meant that you are asking whether the order submitter can use partition specific (application) sequence numbers.
A backend with several partitions is quite common and FIX application sequencing is not efficient if you only have a single sequence. Application level recovery needs to be on a partition level. For this reason, we have introduced the UDF (user-defined field) 5948 PartitionID which you can add to ApplID (1180) and ApplSeqNum (1181) to make the sequence number unique. ApplID could be just “O” for orders, for example.
You then also need to add the partition to the ApplicationMessageRequest as part of ApplIDRequestGrp for partition specific recovery.
Note that PartitionID (5948) is not a standard tag but, in general, the usage of UDFs is FIX compliant. Exceptions are when you use UDFs to circumvent FIX concepts, for example “MyExecType” instead of using ExecType (150) to define your own enum values and associated workflows.
Regards,
Hanno.
Hi,
I’d like to know if it’s FIX compliant to implement application sequencing in Order Routing systems?
For example we have a trading backend which has several partitions, and we need to send order into all the partitions. We are planning to initiate one FIX session for sending orders into all the backend partitions, while for each partition, we maintain a application sequence number.
I’m just wondeing if this design is FIX compliant?
Cheers,
LiuAn
Couldn’t the ApplID (1180) be used to include the identifier of the partition? Instead of having “O” as the ApplID for all partitions, one could have ApplIDs of “O1”, “O2”, “O3”, etc. That would eliminate the need for the custom field PartitionID (5948).
I assume that you mean that your trading backend issues partition specific application sequence numbers. Your emphasis on Order Routing systems could also have meant that you are asking whether the order submitter can use partition specific (application) sequence numbers.
A backend with several partitions is quite common and FIX application sequencing is not efficient if you only have a single sequence. Application level recovery needs to be on a partition level. For this reason, we have introduced the UDF (user-defined field) 5948 PartitionID which you can add to ApplID (1180) and ApplSeqNum (1181) to make the sequence number unique. ApplID could be just “O” for orders, for example.
You then also need to add the partition to the ApplicationMessageRequest as part of ApplIDRequestGrp for partition specific recovery.
Note that PartitionID (5948) is not a standard tag but, in general, the usage of UDFs is FIX compliant. Exceptions are when you use UDFs to circumvent FIX concepts, for example “MyExecType” instead of using ExecType (150) to define your own enum values and associated workflows.
Regards,
Hanno.Hi,
I’d like to know if it’s FIX compliant to implement application sequencing in Order Routing systems?
For example we have a trading backend which has several partitions, and we need to send order into all the partitions. We are planning to initiate one FIX session for sending orders into all the backend partitions, while for each partition, we maintain a application sequence number.
I’m just wondeing if this design is FIX compliant?
Cheers,
LiuAn
Yes and no. It is semantically not the best approach to mix functional elements (orders) with technical elements (partitions). The valid values of ApplID might also change frequently as partitions are adjusted to adapt to changing volumes of instruments being allocated to certain partitions. Maybe a generic ApplSubID field makes sense that could then also be used for something technical like a partition.
Bundling two elements into the same field also requires the gateway to parse the string field to extract the numerical value of the partition identifier before being able to forward the request. An explicit field is always superior in terms of performance.
Couldn’t the ApplID (1180) be used to include the identifier of the partition? Instead of having “O” as the ApplID for all partitions, one could have ApplIDs of “O1”, “O2”, “O3”, etc. That would eliminate the need for the custom field PartitionID (5948).
I’d like to know if it’s FIX compliant to implement application sequencing in Order Routing systems?
For example we have a trading backend which has several partitions, and we need to send order into all the partitions. We are planning to initiate one FIX session for sending orders into all the backend partitions, while for each partition, we maintain a application sequence number.
I’m just wondeing if this design is FIX compliant?
This is quite an interesting topic, with a lot of history.
Technically, the language of the spec forbids use of application sequencing for transactional messaging, like executions in response to orders. Its intended use is for pre-trade market data, post-trade reports, drop copies, etc.
However, there has been an increasing need raised by a number of exchanges for the ability to use application sequencing for messages from exchanges to consumers.
The issue is that a lot of exchange systems not originally built around FIX embed a sort of recovery mechanism and sequencing in their core matching engines, which may or may not be partitioned, and they run this in parallel with their native API’s. And FIX has its own sequence numbers, which increment in each session. FIX recovery is based off of its own sequence numbers. So this means the exchange must expend the latency to commit FIX transactions to stable storage twice: once in the matching engine, and once at the FIX engine.
This has two negative consequences:
-
The FIX connection will probably be slower than the native protocol. Customers using the native protocol can use the matching engine’s sequence numbers for recovery. FIX messages must be committed to stable storage again, as FIX users will use FIX session recovery, not the native exchange sequence numbers and recovery protocol.
-
The FIX engine must be hardened against failure. If the FIX engine crashes, it must be able to recover its state, or it will result in an inability for the customer to recover state using in-band means.
I recall Kevin Houstoun had proposed the concept of “Stateless FIX” that would make use of application sequencing for recovery. My understanding is that in cases where “Stateless FIX” would be employed, FIX engines would no longer need to store message state to stable storage. If a FIX engine crashes, the message store is lost, but this is not in any way a problem. The customer will use the FIX application sequencing mechanism to ask for exchange messages for each partition. This gets translated into the matching engine’s native recovery protocol.
I don’t know if anything has happened regarding this. From what I understand, doing this would require:
-
FPL deciding to remove the language prohibiting application sequencing for transactional messages like order flow from exchange to client. Adding notes to the spec of how to use this in an exchage environment would be useful.
-
Adding the application sequencing fields on more messages, e.g. it is on Execution Report today (for drop copy purposes) but, I don’t think, on Order Cancel Reject. There is work underway to create the concept of a Business Application Header, which could be added to all business messages and would contain certain fields; the application sequencing fields would be likely candidates for this.
We are considering offering a stateless FIX connection to our matching engine. We already offer a regular FIX connection and a native connection. We could close the latency gap between the FIX and native connection if the FIX connection was stateless. We already offer application sequencing for post trade reports and have been considering extending it to the order entry connection.
We did notice the wording in Volume 1 of the spec discouraging the use of application sequencing for order routing. What is the rationale for that? Application sequencing on an order routing connection seems like a reasonable option in a low latency environment.
I’d like to know if it’s FIX compliant to implement application sequencing in Order Routing systems?
For example we have a trading backend which has several partitions, and we need to send order into all the partitions. We are planning to initiate one FIX session for sending orders into all the backend partitions, while for each partition, we maintain a application sequence number.
I’m just wondeing if this design is FIX compliant?
This is quite an interesting topic, with a lot of history.
Technically, the language of the spec forbids use of application sequencing for transactional messaging, like executions in response to orders. Its intended use is for pre-trade market data, post-trade reports, drop copies, etc.
However, there has been an increasing need raised by a number of exchanges for the ability to use application sequencing for messages from exchanges to consumers.
The issue is that a lot of exchange systems not originally built around FIX embed a sort of recovery mechanism and sequencing in their core matching engines, which may or may not be partitioned, and they run this in parallel with their native API’s. And FIX has its own sequence numbers, which increment in each session. FIX recovery is based off of its own sequence numbers. So this means the exchange must expend the latency to commit FIX transactions to stable storage twice: once in the matching engine, and once at the FIX engine.
This has two negative consequences:
The FIX connection will probably be slower than the native protocol. Customers using the native protocol can use the matching engine’s sequence numbers for recovery. FIX messages must be committed to stable storage again, as FIX users will use FIX session recovery, not the native exchange sequence numbers and recovery protocol.
The FIX engine must be hardened against failure. If the FIX engine crashes, it must be able to recover its state, or it will result in an inability for the customer to recover state using in-band means.
I recall Kevin Houstoun had proposed the concept of “Stateless FIX” that would make use of application sequencing for recovery. My understanding is that in cases where “Stateless FIX” would be employed, FIX engines would no longer need to store message state to stable storage. If a FIX engine crashes, the message store is lost, but this is not in any way a problem. The customer will use the FIX application sequencing mechanism to ask for exchange messages for each partition. This gets translated into the matching engine’s native recovery protocol.
I don’t know if anything has happened regarding this. From what I understand, doing this would require:
FPL deciding to remove the language prohibiting application sequencing for transactional messages like order flow from exchange to client. Adding notes to the spec of how to use this in an exchage environment would be useful.
Adding the application sequencing fields on more messages, e.g. it is on Execution Report today (for drop copy purposes) but, I don’t think, on Order Cancel Reject. There is work underway to create the concept of a Business Application Header, which could be added to all business messages and would contain certain fields; the application sequencing fields would be likely candidates for this.
Hi, I am interested in this old discussion so that we can offer less statefull FIX application messages in conjunction with FIXT1.1 and FIX/P. I noticed that tag 5948 has never made it into the official standard and I wonder if this question of partitioning Application Sequences can be formalized ?
kind regards