Imported from previous forum
We are using an order management system for the buy side that has a FIX engine built into the product. Due to the fact that the product is largely a black box, we were unable to conduct session level tests. However, after using the engine, it has become apparent that product has severe limitations.
Namely, it will not disconnect a FIX session after heartbeat transmissions cease from our counterparty. The only thing that will prompt our system to drop the session would be a closed socket. I am trying to get our vendor to fix this as I believe that this is grossly inadequate for a product that claims to be FIX compliant. However, the vendor has stated that this is a low level bug and that the transmission of heartbeats is not explicit in the FIX spec. Can someone please tell me that they are feeding us a line?
My interpretation was that if the heartbeat interval specified in the log on message was not maintained, that the session should be dropped after a certain number of attempts.
We would like to accuse them of being non-compliant with the specs in hopes of moving them towards a fix for this.
Thanks.
[ original email was from Bob Lamoureux - blam@bridge.com ]
Unfortunately, they are technically compliant. As one of the authors of the FIX protocol, this issue has been near and dear to me. The philosophy that I pushed forward was that the absence of heartbeats “could” be deemed to indicate a problem on the remote side. But it was left to the application level software to make the determination as to the action that should be taken. Since I envisioned vendors providing FIX engines, and that they may not have business level knowledge, I thought it would be best to not have FIX engines making harsh decisions such as this during the course of a session. Perhaps you wanted to be tolerant of a counterparty’s system that you knew to be slow or for some other reason you did not want to tear the session down. (remember that FIX was meant to ride on many transports, so session establishment can be expensive in other non-TCP scenarios)
That all being said, I do find it annoying that these FIX engines do not allow the application to get involved in any way with the "lower level" stuff. At a minimum, they should at least provide notification that heartbeats are not arriving in the prescribed time.
In the end, lack of heartbeats does not mean you "must" disconnect, the protocol merely recommends that as one possible solution.
[ original email was from Steven Glickstein - sglickstein@warburg.com ]
This sounds like a huge problem to me. If this is what the spec says, then it should be changed.
If an application doesn’t notify a trader of a dropped FIX connection, then buy side traders will be sending orders, think they are executed and then BOOM, discover they have problem.
This is exactly the scenario that causes firm’s loose lots of money. It also puts in question the ability to use FIX at all.
I’d like to know how many people are comfortable assuming this type operational risk? Sounds like someone better think through the liability issue here.
By the way, there are TOMS on the market that alert the trader to these types of issues.
> Unfortunately, they are technically compliant. As one of the authors of the FIX protocol, this issue has been near and dear to me. The philosophy that I pushed forward was that the absence of heartbeats “could” be deemed to indicate a problem on the remote side. But it was left to the application level software to make the determination as to the action that should be taken. Since I envisioned vendors providing FIX engines, and that they may not have business level knowledge, I thought it would be best to not have FIX engines making harsh decisions such as this during the course of a session. Perhaps you wanted to be tolerant of a counterparty’s system that you knew to be slow or for some other reason you did not want to tear the session down. (remember that FIX was meant to ride on many transports, so session establishment can be expensive in other non-TCP scenarios)
>
> That all being said, I do find it annoying that these FIX engines do not allow the application to get involved in any way with the “lower level” stuff. At a minimum, they should at least provide notification that heartbeats are not arriving in the prescribed time.
>
> In the end, lack of heartbeats does not mean you “must” disconnect, the protocol merely recommends that as one possible solution.
>
>
From the 4.0 spec, page 13:
"If there is still no Heartbeat message received after (HeartBtInt + "some reasonable transmission time") seconds then the connection should be considered lost and corrective action be initiated."
If the fix engine does not at least notify the
application that the condition has occurred, then
no corrective action can be initiated.
It seems to me that a fix engine which ignores
heartbeat interval timeouts is not compiant with
the above.
> > Unfortunately, they are technically compliant. As one of the authors of the FIX protocol, this issue has been near and dear to me. The philosophy that I pushed forward was that the absence of heartbeats “could” be deemed to indicate a problem on the remote side. But it was left to the application level software to make the determination as to the action that should be taken. Since I envisioned vendors providing FIX engines, and that they may not have business level knowledge, I thought it would be best to not have FIX engines making harsh decisions such as this during the course of a session. Perhaps you wanted to be tolerant of a counterparty’s system that you knew to be slow or for some other reason you did not want to tear the session down. (remember that FIX was meant to ride on many transports, so session establishment can be expensive in other non-TCP scenarios)
> >
> > That all being said, I do find it annoying that these FIX engines do not allow the application to get involved in any way with the “lower level” stuff. At a minimum, they should at least provide notification that heartbeats are not arriving in the prescribed time.
> >
> > In the end, lack of heartbeats does not mean you “must” disconnect, the protocol merely recommends that as one possible solution.
> >
> >
>
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
First, I’d like to clarify that lack of heartbeats is NOT a problem so long as other valid messages are being sent. Heartbeats are only needed during periods of inactivity. One should only be worried when one stops receiving ANY messages in a period longer than the heartbeat interval + a reasonable transmission time, and responses to Test Request(s) are not sent.
Two significant things can cause this to happen.
-
Loss of network connectivity. TCP can not detect connection loss instantly because TCP has its own retransmission logic that it uses to get the data across reliably. So a firm can be pumping orders into the ether(net) which will never go live, and in some cases the idle interval can be a faster indication that something is wrong than waiting for the TCP session to close.
-
Computer systems can occasionally get into an overloaded state and take a while to process their message queues. The TCP connection is often perfectly happy in this case. Orders could be submitted which never go live, or, worse, they could get stuck in a buffer and be processed several minutes later when market conditions have changed.
I agree with Bob that FIX shouldn’t dictate business policy in this regard. For instance, another perfectly valid response is to keep the session open but block any new orders from being sent over the session, and notify operations staff.
While it might technically be compliant, I personally wouldn’t want to trust any FIX engine that wouldn’t break the session on inactivity or give my application a signal that the session has gone idle.
When doing testing with counterparties, we recently added a test mode that forces our FIX engine to suspend sending heartbeats or responding to test requests. We can then check to make sure the counterparty sends test requests and disconnects, and should the client fail to do so, we can inform them of the inherent risks. I believe Scott talked about adding "niceness" tests to the FIX certification process, which if failed would not deny certification but would generate a warning. I consider this a good candidate for such a test.