Imported from previous forum
[ original email was from Garry Ryan - garry_p_ryan@hotmail.com ]
Firstly apologies I first submitted this to the discussion forum in error. Please respond to this posting.
Has any thought been given to extending the FIX protocol to support fault tolerant pairs of communicating engines?
I have worked on developing the FIX interface for NASDAQ Europe and am now involved in writing a system that will connect a trading system to Euronext. Both of these have hit practical limitations with the FIX protocol that has been solved by the introduction of a new field holding a business message id to all messages from the exchange.
Both exchanges follow a similar recovery sequence. When a session is reconnected, the session message sequence number is always reset and the (new) business message id is passed to the exchange. The exchange then transmits all messages from the last received by the client ensuring no messages are lost.
The issue they are trying to solve is this. The financial institutions that connect to the exchanges typically require that they have fault tolerant systems. This means running a primary and backup system.
When the primary system receives a FIX message it needs to:
a) Pass the message to the business engine.
b) Secure the message to persistent storage.
c) Pass the message to the backup engine.
(No ordering implied). These steps are each atomic and difficult to write into a single transaction. This means that in the event of a failure the systems can be out of synch. This together with the FIX optimistic delivery protocol makes it difficult to use the FIX session to recover. There are other issues such as the form of the communication between the FIX engine and the business engine (not always reliable) and the ability for a primary and standby system to communicate with each other.
The introduction of a business message sequence id allows the FIX engine to become stateless between connections and passes the responsibility for intra session recovery to the business engine. This clears up all of the problems and guarantees reliable message delivery.
Cheers.
[ original email was from John Prewett - jprewett@lavatrading.com ]
An excellent subject for discussion, Garry.
It is easy for two FIX engines to re-synchronize a FIX session, all you have to do is follow the protocol which several(;-)) FIX engines manage to do. It is much more difficult to re-synchronize the business applications that use the FIX engines, particularly when you have a pair of FIX engines co-owning one end of a FIX session operating in some sort of fault-tolerant mode.
Your suggestion of a new "business sequence number" field & "BusinessResendRequest message" is quite good as the FIX admin messages would then be ignored from a business perspective and more simple business<–>business resynchronization could be achieved.
If fails on two points:
- Reject messages generated by the FIX engine itself would be ignored. These are needed by the business application.
- It would imply a fairly major change to the FIX standard or using a non-standard version. I think a major change to the standard is unlikely at this juncture. I don’t like doing non-standard things as it causes all my counterparts to have to change (assuming that they can).
As an alternative, I prefer attempting to guarantee the integrity of the message transfer between your pair of fault tolerant FIX engines and your business application. Easy to state with a simple sentence, needs slightly more code to be implemented 
This suggestion directly implies that you have the power to change your FIX engine’s implementation, which I something I would recommend.
Good luck.
> Firstly apologies I first submitted this to the discussion forum in error. Please respond to this posting.
>
> Has any thought been given to extending the FIX protocol to support fault tolerant pairs of communicating engines?
>
> I have worked on developing the FIX interface for NASDAQ Europe and am now involved in writing a system that will connect a trading system to Euronext. Both of these have hit practical limitations with the FIX protocol that has been solved by the introduction of a new field holding a business message id to all messages from the exchange.
>
> Both exchanges follow a similar recovery sequence. When a session is reconnected, the session message sequence number is always reset and the (new) business message id is passed to the exchange. The exchange then transmits all messages from the last received by the client ensuring no messages are lost.
>
> The issue they are trying to solve is this. The financial institutions that connect to the exchanges typically require that they have fault tolerant systems. This means running a primary and backup system.
> When the primary system receives a FIX message it needs to:
> a) Pass the message to the business engine.
> b) Secure the message to persistent storage.
> c) Pass the message to the backup engine.
> (No ordering implied). These steps are each atomic and difficult to write into a single transaction. This means that in the event of a failure the systems can be out of synch. This together with the FIX optimistic delivery protocol makes it difficult to use the FIX session to recover. There are other issues such as the form of the communication between the FIX engine and the business engine (not always reliable) and the ability for a primary and standby system to communicate with each other.
>
> The introduction of a business message sequence id allows the FIX engine to become stateless between connections and passes the responsibility for intra session recovery to the business engine. This clears up all of the problems and guarantees reliable message delivery.
>
> Cheers.
>
>
[ original email was from Garry Ryan - garry_p_ryan@hotmail.com ]
Thanks for the comments, much appreciated.
> An excellent subject for discussion, Garry.
>
> It is easy for two FIX engines to re-synchronize a FIX session, all you have to do is follow the protocol which several(;-)) FIX engines manage to do. It is much more difficult to re-synchronize the business applications that use the FIX engines, particularly when you have a pair of FIX engines co-owning one end of a FIX session operating in some sort of fault-tolerant mode.
>
> Your suggestion of a new "business sequence number" field & "BusinessResendRequest message" is quite good as the FIX admin messages would then be ignored from a business perspective and more simple business<–>business resynchronization could be achieved.
>
> If fails on two points:
> 1. Reject messages generated by the FIX engine itself would be ignored. These are needed by the business application.
That’s a good point. As you suggest, the business engine would need the reject messages. This would require a protocol change. However, IMHO there are very few occasions when the reject message is generated; the chance of a network spike causing an issue would be negligible. The generation of a reject message for me indicates that one side either has a bug or is incompatible with the other. This indicates a serious error that should be addressed, so could I think be justifiably dealt with by terminating the connection, rather like getting a duplicated sequence number.
Implications of this to the community would be an interesting discussion to have (for me at least 
> 2. It would imply a fairly major change to the FIX standard or using a non-standard version. I think a major change to the standard is unlikely at this juncture. I don’t like doing non-standard things as it causes all my counterparts to have to change (assuming that they can).
>
I think you need:
a) An additional field in the Logon message for the last received business message. Can only be set when the session is being reset as well.
b) A modification that reject messages cause a reset.
c) Possibly an addition to the header, the business message id.
With the FIX engines I have written this is a minor change. Though my experience is limited…
> As an alternative, I prefer attempting to guarantee the integrity of the message transfer between your pair of fault tolerant FIX engines and your business application. Easy to state with a simple sentence, needs slightly more code to be implemented This suggestion directly implies that you have the power to change your FIX engine’s implementation, which I something I would recommend.
I do have the power to change the FIX engine implementation, but not much of the environment it runs in:
- Commercially clients run the engines at different sites and I have to watch the bandwidth between the primary and standby. Note also long round trips between the primary and standby for communications the latency interferes with overall performance.
- The comms to the business engine (which I cant change) works on a request reply basis. Which means I have another transactional issue which I dont think I can reliably solve.
Both make the business message id solution more appealing in my environment.
>
> Good luck.
>
> > Firstly apologies I first submitted this to the discussion forum in error. Please respond to this posting.
> >
> > Has any thought been given to extending the FIX protocol to support fault tolerant pairs of communicating engines?
> >
> > I have worked on developing the FIX interface for NASDAQ Europe and am now involved in writing a system that will connect a trading system to Euronext. Both of these have hit practical limitations with the FIX protocol that has been solved by the introduction of a new field holding a business message id to all messages from the exchange.
> >
> > Both exchanges follow a similar recovery sequence. When a session is reconnected, the session message sequence number is always reset and the (new) business message id is passed to the exchange. The exchange then transmits all messages from the last received by the client ensuring no messages are lost.
> >
> > The issue they are trying to solve is this. The financial institutions that connect to the exchanges typically require that they have fault tolerant systems. This means running a primary and backup system.
> > When the primary system receives a FIX message it needs to:
> > a) Pass the message to the business engine.
> > b) Secure the message to persistent storage.
> > c) Pass the message to the backup engine.
> > (No ordering implied). These steps are each atomic and difficult to write into a single transaction. This means that in the event of a failure the systems can be out of synch. This together with the FIX optimistic delivery protocol makes it difficult to use the FIX session to recover. There are other issues such as the form of the communication between the FIX engine and the business engine (not always reliable) and the ability for a primary and standby system to communicate with each other.
> >
> > The introduction of a business message sequence id allows the FIX engine to become stateless between connections and passes the responsibility for intra session recovery to the business engine. This clears up all of the problems and guarantees reliable message delivery.
> >
> > Cheers.
> >
> >
>
> > An excellent subject for discussion, Garry.
I’d second that sentiment.
>>It is much more difficult to re-synchronize the business applications
>> that use the FIX engines, particularly when you have a pair of FIX
>> engines co-owning one end of a FIX session operating in some
>> sort of fault-tolerant mode.
You also have to assume that in general the other end of the connection is operating a distributed hot standby!
This causes a problem for the unique business message ID concept. Having source-generated sequence numbers (or some ID) is absolutely fundamental to being able to achieve a truly reliable hot standby failover. [ I’ll just claim that for now, but I’m happy to discuss it if anybody wishes ]. For many organizations, and exchanges are usually good examples, their messages are actually sourced from a number of different systems and it’s then an extremely hard problem to generate the sequence numbers for a combined output message stream (because of possible time-ordering races on the A and B side systems).
In those circumstances, generating true sequence numbers is pretty much never done because it implies global system synchronization which has huge performance penalties. The closest approximation is to feed all the source streams through a merging, sequencing engine (again with a synchronized A and B) but that causes performance problems of its own and still leaves the possibility of some loss of sequencing on failover.
So the best option, IMHO, is to generate individual sequence number streams at each genuine source, and this is pretty much what FIX does/allows with its different namespaces for IDs (ClOrdID, OrderID, ExecID etc). It does make the code more messy but the result has at least the possibility of being bulletproof!
> 1) Commercially clients run the engines at different sites
> and I have to watch the bandwidth between the primary and standby.
You need to make efficient use of the bandwidth, which may rule out some solutions, but you need to have multiple independent network links between the sites in any case to make the failover work. The business apps almost certainly need more bandwidth between the A and B sites than the FIX connections. It’s just a cost of doing business.
> Note also long round trips between the primary and standby for
> communications the latency interferes with overall performance.
This is a red herring if suitable comms are chosen and the software is correctly designed. Network latencies are pretty small over metropolitan areas or even between cities and can certainly be hidden by the disk latency of the persistent storage, for example. (in fact, you can use the network to actually speed up performance by exploiting this)
> 2) The comms to the business engine (which I cant change) works
> on a request reply basis. Which means I have another transactional
> issue which I dont think I can reliably solve.
As long as the request-reply mechanism itself is sufficiently transactional (in the sense that either the A or B business engine can obtain an uncorrupted sequence from either of the FIX engines even in the presence of failures) there shouldn’t be a problem. It can just keep coming back to the persistent store until it’s done. If it can’t do this, then you have to change those comms if you want to solve the problem.
Ryan said:
> If a message isn’t in stable storage, the sequence numbers likely
> aren’t updated either, so when the engine restarts after failure the
> engine will send a Resend Request to the client and obtain the
> missing message. (Or the client will choose not to resend it
> and will Gap Fill over it, in which case the missing message
> isn’t important.)
The difficulty with this is that in a hot standby system the standby already has a live connection, probably to a different engine at the other end, and wants to continue business over that session. If you’re waiting for an engine to restart or a connection to re-establish, it isn’t hot standby. And who knows where or what the failure was, it may not be possible to re-establish the original session at all.
Cheers, Dave
[ original email was from Garry Ryan - garry_p_ryan@hotmail.com ]
I have received a lot of interesting responses to my original posting which I think suggests this would be a valid topic for discussion in a forum on where FIX goes in the future. The basic question I guess is: "Can the FIX protocol be extended to ease business to business message synchronisation either with or without fault tolerant servers".
My main intention with the posting wasn’t to get help with my forthcoming work on Euronext they have already defined the way their FIX implementation works and I can write something to communicate reliably with their system. It was to highlight an issue I have seen repeated a couple of times.
For my work on NASDAQ.
a) Clients have small pipes between the exchange and the exchange server. This implied that the FIX engine running on the client box couldnt communicate with a backup engine via the exchange link.
b) Many clients required a fault tolerant route to the exchange including dual site support be provided. Firewalls etc meant it was not possible for the primary and standby FIX engines to communicate.
These two implied that the FIX engine had to be stateless between sessions. Hence we used a version of the protocol which used business message id’s for inter session recovery and which reset session sequence numbers at the start of every new connection. Euronext have gone for a similar approach, I dont know what their design requirements were. The protocol did not need to be changed too much to achive this.
From the looks of the postings people have put a lot of work into using business identifiers like ClOrdId etc to solve this problem. Most solutions mentioned have included reference to clusters or implied relatively large scale implementations. I think it would be preferable if the protocol could be extended to help avoid this. Now the gotcha, reading between the lines on the postings and direct emails it looks like there are a lot of different requirements placed upon the engines out there (which I for one hadnt thought of) and the final solution should ideally be generic enough to cover all.
Is there any desire for a next step on this?
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> I have worked on developing the FIX interface for NASDAQ Europe and am now involved in writing a system that will connect a trading system to Euronext. Both of these have hit practical limitations with the FIX protocol that has been solved by the introduction of a new field holding a business message id to all messages from the exchange.
One can write high-availability, fault-tolerant engines, including hot standby engines, with the FIX protocol as it exists today. Every FIX application message has a unique business ID, which is extremely useful in recovery, as it makes the messages idempotent.
If a message isn’t in stable storage, the sequence numbers likely aren’t updated either, so when the engine restarts after failure the engine will send a Resend Request to the client and obtain the missing message. (Or the client will choose not to resend it and will Gap Fill over it, in which case the missing message isn’t important.)
If a message is in stable storage, but the engine is uncertain whether the business layer has seen it, the engine can always resend it after a failure to the business layer with the FIX PossResend=Y or an out of band flag indicating the same thing. The business layer can use ExecID if buy-side, or ClOrdID if sell-side, to determine if they have seen a message before, and if so, safely ignore it.
Likewise, on the outbound path, the business layer can push data it isn’t certain the other party has seen to the other party as long as it sets the FIX field PossResend=Y. The other party bears the responsibility of sorting through the data, processing what hasn’t been processed, and ignoring what has been processed.
Now I must warn about something rather unfortunate. Some firms do not properly implement PossResend. Either they ignore all PossResend messages without checking IDs, or they accept PossResend messages without checking IDs.