Imported from previous forum
Guys,
Anyone out there who could explain to me why test requests and heartbeats needs to increase sequence and complicate the gap recovery event model?
Thanks in advance.
Regards,
H
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
It wouldn’t avoid gap fills due to resend requests for example. And if any sequence number is set in a resend request, you won’t be able to detect that you have missed it (if you miss it)…
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
That is absolutely correct and that’s my bad for not indicating that earlier.
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
Still wondering why FIX hasn’t updated their session protocol to reduce certain inefficiencies.
It wouldn’t avoid gap fills due to resend requests for example. And if any sequence number is set in a resend request, you won’t be able to detect that you have missed it (if you miss it)…
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
Since FIX 4.4, there is the tag 789 (NextExpectedMsgSeqNum) in the logon message which is an alternative to the resend request.
Since FIX 5.0, there is also the possibility of managing an applicative sequence number at business level. But it only works for messages from the exchange to the client. You can set 1181 (ApplSeqNum) in your execution reports. The client may request them again by sending an ApplicationMessageRequest (35=BW).
That is absolutely correct and that’s my bad for not indicating that earlier.
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
Still wondering why FIX hasn’t updated their session protocol to reduce certain inefficiencies.
It wouldn’t avoid gap fills due to resend requests for example. And if any sequence number is set in a resend request, you won’t be able to detect that you have missed it (if you miss it)…
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
Thanks Xavier.
My attempt is mainly to try and reduce the permutations on the session level and depend on the quality of the present networks for message delivery.
Lets hope version after FIXT 1.1 would try something new.
Thanks,
Heshan.
Since FIX 4.4, there is the tag 789 (NextExpectedMsgSeqNum) in the logon message which is an alternative to the resend request.
Since FIX 5.0, there is also the possibility of managing an applicative sequence number at business level. But it only works for messages from the exchange to the client. You can set 1181 (ApplSeqNum) in your execution reports. The client may request them again by sending an ApplicationMessageRequest (35=BW).
That is absolutely correct and that’s my bad for not indicating that earlier.
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
Still wondering why FIX hasn’t updated their session protocol to reduce certain inefficiencies.
It wouldn’t avoid gap fills due to resend requests for example. And if any sequence number is set in a resend request, you won’t be able to detect that you have missed it (if you miss it)…
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
Heshan,
FIX 2.7 and 3.0 (early 1990s) had a session layer where the administrative messages did not increment. That changed with FIX 4.0 and since then only very minor changes were made until FIXT.1.1.
I can’t tell you the history or the reasons for the change, perhaps someone involved in that decision making process can. I have dealt with FIX 2.7 and 3.0 implementations - was session recovery better in 2.7/3.0 than in 4.x? Not necessarily better or worse. From what I remember the 4.x session made it clearer what to expect in the different recovery testing scenarios.
Your comments is a bit worrisome:
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
By implementing customization to the FIX Session as you indicated, wouldn’t you be making it difficult for your counterparties that uses off the shelf FIX engines to connect to the exchange?
And what would be the “something new” you would try after FIXT.1.1? Share your thoughts.
Thanks Xavier.
My attempt is mainly to try and reduce the permutations on the session level and depend on the quality of the present networks for message delivery.
Lets hope version after FIXT 1.1 would try something new.
Thanks,
Heshan.Since FIX 4.4, there is the tag 789 (NextExpectedMsgSeqNum) in the logon message which is an alternative to the resend request.
Since FIX 5.0, there is also the possibility of managing an applicative sequence number at business level. But it only works for messages from the exchange to the client. You can set 1181 (ApplSeqNum) in your execution reports. The client may request them again by sending an ApplicationMessageRequest (35=BW).
That is absolutely correct and that’s my bad for not indicating that earlier.
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
Still wondering why FIX hasn’t updated their session protocol to reduce certain inefficiencies.
It wouldn’t avoid gap fills due to resend requests for example. And if any sequence number is set in a resend request, you won’t be able to detect that you have missed it (if you miss it)…
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
Thanks Lisa for your response; specially for the insight into what FIX was pre-4.0.
To answer your question on the existing FIX engines, the protocol I’m referring to would co-exist with the standard FIX 4.2 and 5.0 SP2 offerings that we have hence it shouldn’t be an issue. We’re expecting the brave to use this flavor.
Well honestly I’m hoping to hear that from FIX. But to point out 2 areas:
- Session level simplification steps
- FIX messaging latency and throughput performance improvement attempts
Wishful thinking you think?
Heshan,
FIX 2.7 and 3.0 (early 1990s) had a session layer where the administrative messages did not increment. That changed with FIX 4.0 and since then only very minor changes were made until FIXT.1.1.
I can’t tell you the history or the reasons for the change, perhaps someone involved in that decision making process can. I have dealt with FIX 2.7 and 3.0 implementations - was session recovery better in 2.7/3.0 than in 4.x? Not necessarily better or worse. From what I remember the 4.x session made it clearer what to expect in the different recovery testing scenarios.
Your comments is a bit worrisome:
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
By implementing customization to the FIX Session as you indicated, wouldn’t you be making it difficult for your counterparties that uses off the shelf FIX engines to connect to the exchange?
And what would be the “something new” you would try after FIXT.1.1? Share your thoughts.
Thanks Xavier.
My attempt is mainly to try and reduce the permutations on the session level and depend on the quality of the present networks for message delivery.
Lets hope version after FIXT 1.1 would try something new.
Thanks,
Heshan.Since FIX 4.4, there is the tag 789 (NextExpectedMsgSeqNum) in the logon message which is an alternative to the resend request.
Since FIX 5.0, there is also the possibility of managing an applicative sequence number at business level. But it only works for messages from the exchange to the client. You can set 1181 (ApplSeqNum) in your execution reports. The client may request them again by sending an ApplicationMessageRequest (35=BW).
That is absolutely correct and that’s my bad for not indicating that earlier.
We’re trying an implementation where Logon, Logout, Test Request and Resend Request messages are sequence-less.
Further we’re adding a Last Received Sequence to the Logon through which we’re trying to avoid a Resend Request usage completely.
Still wondering why FIX hasn’t updated their session protocol to reduce certain inefficiencies.
It wouldn’t avoid gap fills due to resend requests for example. And if any sequence number is set in a resend request, you won’t be able to detect that you have missed it (if you miss it)…
With no sequence for the heartbeats and test requests, we can avoid gap fills fully assuming other session messages such as session rejects are replayed on a resend request.
This discussion is dredging up old and foggy memories from the late 90’s…
First, I’d like to challenge the assumption that having session level messages consume sequence numbers necessarily complicates things.
I can see the desire to have an immutable sequence of business-level messages, with perfect monotonically incrementing sequence numbers. That may be possible from the exchange to the client. But from the client to the exchange is a different story entirely.
Let’s say the client sends a limit order, right as the connection crashes. While the connection is down, the client’s customer wants to cancel the limit order. When the connection comes back up again, say, a half hour later, the exchange asks for a Resend Request for a range of messages including that order. So it is uncertain whether the exchange has received this order.
The smart thing to do, at this point, is to do a Sequence Reset - Gap Fill to skip over the order, and hope that it was never processed by the exchange. This minimizes the risk of giving the customer a bad fill.
For that matter, such logic could be extended based on business needs. E.g. no orders should be retransmitted. Cancels for orders should always be retransmitted. Modification to orders should only be retransmitted if the customer has not attempted to cancel the order, and only then the last modification should be retransmitted, possibly taking into account whether the exchange acknowledged the order in the first place.
So it follows that the immutable, monotonic sequence of messages is a fallacy, and the session protocol has to support the ability to skip over messages that shouldn’t be retransmitted. So once that capability is there, why is it any more difficult to give application level messages their own sequence number?
This discussion is dredging up old and foggy memories from the late 90’s…
It’s worse than that. Back in the day (when Jim Lehman was at SB), FIX was contemplated waaay before the internet – and before even TCPIP was commonplace. As such, FIX was designed as a communications protocol in addition to being a transactional protocol (hence the need to request/resend messages).
I remember telling Jim that they really should switch to a procedural interface (e.g. CORBA) and get rid of the communications nonsense. But nobody listens to me …
M
Thanks Ryan your comments have been very helpful.
This discussion is dredging up old and foggy memories from the late 90’s…
First, I’d like to challenge the assumption that having session level messages consume sequence numbers necessarily complicates things.
I can see the desire to have an immutable sequence of business-level messages, with perfect monotonically incrementing sequence numbers. That may be possible from the exchange to the client. But from the client to the exchange is a different story entirely.
Let’s say the client sends a limit order, right as the connection crashes. While the connection is down, the client’s customer wants to cancel the limit order. When the connection comes back up again, say, a half hour later, the exchange asks for a Resend Request for a range of messages including that order. So it is uncertain whether the exchange has received this order.
The smart thing to do, at this point, is to do a Sequence Reset - Gap Fill to skip over the order, and hope that it was never processed by the exchange. This minimizes the risk of giving the customer a bad fill.
For that matter, such logic could be extended based on business needs. E.g. no orders should be retransmitted. Cancels for orders should always be retransmitted. Modification to orders should only be retransmitted if the customer has not attempted to cancel the order, and only then the last modification should be retransmitted, possibly taking into account whether the exchange acknowledged the order in the first place.
So it follows that the immutable, monotonic sequence of messages is a fallacy, and the session protocol has to support the ability to skip over messages that shouldn’t be retransmitted. So once that capability is there, why is it any more difficult to give application level messages their own sequence number?
Some additional thoughts:
Let’s say Heartbeats and Test Request messages don’t get their own sequence numbers. Let’s say I send:
10 Order
11 Order
12 Order
12 Heartbeat
12 Test Request
12 Heartbeat
What happens if the Order with sequence number 12 gets dropped? The engine would see 10, 11, and 12, and think all is in sequence, at least unless it has custom logic that says “Hey, wait a minute, nobody sent a business-level message for 12, so there’s a gap.” Giving everything its own sequence number would make the engine logic simpler in this case.
What happens if the Test Request got dropped? The engine couldn’t tell. It would see, for sequence number 12, an order and two heartbeats only. It would not see a gap, and wouldn’t respond to the Test Request. Depending on the implementation, this could cause the session to drop.
Your presumption is similar to mine.
What I’m trying to do here is to find out why we do certain things and whether they make sense at present with the advancements of networks and taking into consideration the level of sophistication of clients.
Thanks again Ryan appreciate your replies immensely.
Some additional thoughts:
Let’s say Heartbeats and Test Request messages don’t get their own sequence numbers. Let’s say I send:
10 Order
11 Order
12 Order
12 Heartbeat
12 Test Request
12 HeartbeatWhat happens if the Order with sequence number 12 gets dropped? The engine would see 10, 11, and 12, and think all is in sequence, at least unless it has custom logic that says “Hey, wait a minute, nobody sent a business-level message for 12, so there’s a gap.” Giving everything its own sequence number would make the engine logic simpler in this case.
What happens if the Test Request got dropped? The engine couldn’t tell. It would see, for sequence number 12, an order and two heartbeats only. It would not see a gap, and wouldn’t respond to the Test Request. Depending on the implementation, this could cause the session to drop.