Imported from previous forum
[ original email was from John Prewett - jprewett@lavatrading.com ]
I wondered if anyone has had some experience to share of using multiple FIX sessions between two corporations for fault tolerance (and load-balancing).
As far as I can see, FIX doesn’t enforce “same path” concepts whereby a NewOrderSingle sent on one FIX session mandates the response to be sent on the same FIX session.
If there were two sessions between a pair of entities, they could both agree to that requests and responses could be sent on either session, whichever was available. Of course the ClOrdIDs, ExecIDs etc would have to work appropriately when used on either session.
The only problem I can see to this approach would be the "well-ordered" enforcement about messages. If you use two sessions, it would be possible for a NewOrderSingle sent on one session to be "overtaken" by a CancelRequest to that order which was sent on the other session and happened to be processed first by the recipient.
A parallel to the use of multiple FIX sessions can be drawn by the usage of multiple X.25 SVCs when connecting to SIAC. SIAC will accept an order on one SVC and a cancel to that order on another SVC. SIAC will respond with ACKs, fills, OUTs etc,. on any available SVC which it considers "identical".
I have produced buy-side software in my past that could handle out-of-sequence events which arise from the use of multiple "sessions" with SIAC, why not adopt the same state change martix and use multiple FIX sessions?
The advantages are clear. You can multiply connect to another financial institution using different carriers, via different routers etc,. and offer a significant improvement in fault tolerance (and throughput if you adopt load-balancing strategies) versus a single FIX session.
Comments? Repudiations? Jokes?
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
You’ve brought up some pretty good points.
SIAC CMS doesn’t have the same-path constraint. However, CMS pretty much had to be engineered that way because it was based around 9600 baud SVCs; if you’re sending lots of order flow, you need lots of connections.
FIX is transport protocol-agnostic. But when FIX was developed, TCP/IP had emerged as the winner. TCP/IP bandwidth is cheap. Firms could run FIX over TCP/IP at whatever bandwidth they wished. So having pools of FIX connections between parties wasn’t really necessary.
While I can see the possibility of firms doing what you suggest using FIX, I don’t think that many actually do so. The problem of complexity increases enormously once rules regarding sequencing break. Even if you can do intelligent things like looking at CumQty to know that you should delay processing the ExecType=Cancelled message because it appears another partial fill hasn’t been received yet, the path from the buy-side to the sell-side still has problems. For example, a trader who places an order and immediately hits the cancel button doesn’t want his cancel routing down another session, getting to the other party first and being rejected, leaving the order live.
You can still get very good redundancy and recoverability with FIX assuming same-path constraints. You can establish N connections to your trading partner and round-robin order flow across them. If one connection fails, then:
A. You still have (N-1)/N of your capacity to initiate new order flow.
B. 1/N of your live orders are going to be temporarily inaccessible.
Individual FIX connections can be hardened against failure. If one session fails, an engine can try to connect to a backup machine that is a mirror of the primary.
It’s a tradeoff. Personally, I would be willing to live with having 1/N of my order flow inaccessible during very rare outage conditions until a standby becomes available if it means that I never have to worry about dealing with bizarre out-of-sequence conditions that will happen all the time and only get worse as trading volume rises.
[ original email was from Joseph Horowitz - joey@aegisoft.com ]
> B. 1/N of your live orders are going to be temporarily inaccessible.
I agree with Ryan, that the more significant problem with running multiple lines is the issue of fail-over and recovery - the ability to continue working an order or resending a request that may not have been received or acknowledged by SIAC on another session after the original connection has failed.
I believe that the correct use of the PosResend tag (97) can help solve this problem. As long as all the other message details are the same SIAC should detect duplicates correctly. Not that I’m suggesting that this is easy, there are typically application logic changes needed to support this flag, in both directions. An alternate technical solution we’ve seen to this problem is the Euronext MsgID, custom tag 9262.
> The only problem I can see to this approach would be the "well-ordered" enforcement about messages.
> If you use two sessions, it would be possible for a NewOrderSingle sent on one session to be "overtaken"
> by a CancelRequest to that order which was sent on the other session and happened to be processed first
> by the recipient.
Again I agree with Ryan, making sure that other requests for a specific order are sent down the same physical connection (at least) can avoid this problem - what was termed the "same-path constraint."
As a side note, we’ve noticed on some FIX implementations that sending a Cancel immediately after an Order may result in the Cancel being processed first (FEP may be giving it a higher priority!?) resulting in the order being accepted - so multiple sessions are not the exclusive domain for this concern.
> The problem of complexity increases enormously once rules regarding sequencing break.
> Even if you can do intelligent things like looking at CumQty to know that you should delay processing
> the ExecType=Cancelled message because it appears another partial fill hasn’t been received …
The elected use of CMS Activity ID Numbers can help solve many of the complex issues when handling the reception of out-of-sequence events.
I hope this was helpful.
Joseph Horowitz
Aegis Software, Inc.
joey@aegisoft.com
www.aegisoft.com
[ original email was from John Prewett - jprewett@lavatrading.com ]
Ryan is suggesting that using multiple sessions with the “same-path” constraint will give you the ability to enter new orders even if one session(line?) goes down. He indicates that not using “same-path” for modifies and cancels can cause some undesirable timing circumstances such as when an order is cancelled before it exists (from the sell-side’s perspective).
Always mandating "same-path" (the unofficial status quo today) would mean that orders previously sent over a dead session become unmanageable for the duration that the session remains dead (potentially a considerable period of time).
I think a compromise of "mainly same-path" could be beneficial.
Order sending rules:
- All new orders are sent in a load balanced method on all available "equivalent" sessions.
- Modify requests must always be sent on the same session as the original order.
- Cancel requests should be sent on the same session as the original order, if it is still available. If the original session is unavailable, any other "equivalent" session may be used. This would permit emergency cancellation of orders even if the session that originated them is down.
- Status requests should be sent on the same session as the original order, if it is still available. If the original session is unavailable, any other "equivalent" session may be used. This would permit emergency status requests on orders even if the session that originated them is down.
Execution report sending rules:
- Execution reports should always be sent on the same session where the newOrder/modify/cancel/status request originated from. If this session is unavailable, any other "equivalent" session may be used.
The above proposal would give good fault-tolerance and load balancing. It would also tend to minimize problems with out-of-sequence event processing as everything usually follows same-path.
The situation requiring effort would be to handle messages that were lost "in transit" while a session is down. While FIX will recover the lost messages automatically when the session is restarted, the outage could last for a significant time. The buy-side could always adopt a simple strategy of sending cancel requests on an "equivalent" session for all open orders. A more thorough (less drastic) approach could be to automate the recovery of the lost information by using status reports on an "equivalent" session during this outage.