Imported from previous forum
I have a question regarding sequence reset messages (synching 2 FIX engines after a sequence number gap has been detected). From what I understand from the FIX protocol, there are 3 ways that an engine can respond to a resend request:
1)By sending the requested messages with the PossDup flag.
2) By filling the gap via a gap fill message.
3) By forcing a resynchronization via a sequence reset message.
I have a disagreement with a counterparty regarding the sequence reset message. For arguments sake, let’s say that a counterparty asks my engine for missed sequence numbers 10-20. Instead of resending these messages (scenario 1 above) or by sending them 10 and then filling the rest of the gap via a gap fill message which then tells them to reset to 21, my engine prefers to send them sequence number 21, telling them to expect 22 next (tag 36 NewSeqNo). According to the FIX protocol, this is valid. The counterparty is suppossed to IGNORE 34=n in a SeqReset-Reset message. The only sequence number they should care about is the NewSeqNum (tag 36). They should then reset the sequence number that they expect from my engine to this value.
Please confirm that this is correct. The counterparty is claiming that all sequence numbers should be accounted for. I think that they feel that a SeqReset-Reset message ignores the fact that they are asking us for missed messages. They feel that the reset message ignores the gap. Thus, a violation of the sequence of messages.
Please advise of the proper way to use a SeqReset-Reset message to respond to a resend request. The FIX protocol is a little unclear in explaining this scenario.
Thanks,
Bryan Harkins
Tradescape
[ original email was from Witold Sames - witold@javtech.com ]
I believe Ryan Pierce at ARCA / Townsend had a very good write-up, among others that have been posted to this site. I attach the text here.
" Could someone offer some tips on how to decide between a Sequence Reset - Gapfill versus a Sequence Reset - Reset. Some counterparties seem to be more comfortable with Gapfill rather than Reset. But Reset seems to be easier (looser in the sense that the sequence number of the message itself is ignored). Also looks like Gapfill was not available till FIX 4.0, which led me to believe that older engines would prefer Reset. But that doesn’t seem to be the case."
You are right that Gap Fill wasn’t around before 4.0. So if you have the misfortune of communicating with a 3.0 engine, you have no choice in this matter.
Seq Reset - Gap Fill is a much better way of handling problems than a Seq Reset - Reset. The reason is that a Gap Fill explicitly states exactly what messages should be skipped over. It is possible with a Seq Reset - Reset that one may lose messages.
For instance, assume a link that randomly drops messages. You are expecting 10. I send you 10 Execution Report, 11 Heartbeat, and 12 Execution Report. 10 and 11 are dropped. You get 12 and send a Resend Request of 10-999999. I send 10 Poss Dupe Execution Report, 11 Poss Dupe Seq Reset - Reset to 12, and 12 Poss Dupe Execution Report. The line is lossy, and 10 gets dropped again but 11 goes through. Because Seq Reset - Reset is processed without respect to sequence number, you are required to accept it, increment your next inbound sequence number to 12, and you may likely never know you missed 10 until the next morning when the trades don’t compare.
Now lets say when responding to your Resend Request I send 11 Poss Dupe Seq Reset - Gap Fill to 12, and 10 is dropped as before. Gap Fills are required to be processed in sequence, so your receiving 11 will trigger you to send another Resend Request from 10-999999, and eventually message 10 will be recovered.
In all FIX 4.0 and later implementations I’ve developed, I employ the rule of thumb that when responding to a Resend Request, I will ALWAYS use Seq Reset - Gap Fill and NEVER Seq Reset - Reset. I believe the only good use of Seq Reset - Reset is when a session becomes out of control and an operator needs to trigger it manually in an attempt to recover. In this case, the two parties can compare logs and see if anything important was missed.
> I have a question regarding sequence reset messages (synching 2 FIX engines after a sequence number gap has been detected). From what I understand from the FIX protocol, there are 3 ways that an engine can respond to a resend request:
>
> 1)By sending the requested messages with the PossDup flag.
> 2) By filling the gap via a gap fill message.
> 3) By forcing a resynchronization via a sequence reset message.
>
> I have a disagreement with a counterparty regarding the sequence reset message. For arguments sake, let’s say that a counterparty asks my engine for missed sequence numbers 10-20. Instead of resending these messages (scenario 1 above) or by sending them 10 and then filling the rest of the gap via a gap fill message which then tells them to reset to 21, my engine prefers to send them sequence number 21, telling them to expect 22 next (tag 36 NewSeqNo). According to the FIX protocol, this is valid. The counterparty is suppossed to IGNORE 34=n in a SeqReset-Reset message. The only sequence number they should care about is the NewSeqNum (tag 36). They should then reset the sequence number that they expect from my engine to this value.
>
> Please confirm that this is correct. The counterparty is claiming that all sequence numbers should be accounted for. I think that they feel that a SeqReset-Reset message ignores the fact that they are asking us for missed messages. They feel that the reset message ignores the gap. Thus, a violation of the sequence of messages.
>
> Please advise of the proper way to use a SeqReset-Reset message to respond to a resend request. The FIX protocol is a little unclear in explaining this scenario.
>
> Thanks,
> Bryan Harkins
> Tradescape
>
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
From what you have described, it seems that what you are doing is completely compliant with the FIX spec. The FIX spec does not require all sequence numbers to be accounted for with a Seq Reset - Reset.
However, using a Seq Reset - Reset as you do is a particularly dangerous business practice, the reason for which was quoted by Witold. A firm might make it a business requirement that one’s counterparties use only Seq Reset - Gap Fills. This business requirement would NOT stem from the FIX spec, as both approaches are equally compliant, but from business practice requirements to eliminate the possibility of message loss.
The severity of message loss differs, based on whether one is doing pre-trade or order flow (the severity of dropping an IOI geneally isn’t that great when compared to order flow messages) and whether one is on the buy side or sell side. Buy side messages all have explicit application confirms that should happen almost immediately, so a lost message is generally quickly detected. Sell side messages (i.e. trades) do not have explicit application confirms, so the loss of a trade may not make itself apparent until the parties try to reconcile their trades the following morning.
I encounted a situation where a 3rd party (using FIX 4.1) is disallowing any-and-all use of the SeqReset message (35=4) when I am responding to their ResendRequest. Does this happen very often, or is this an isolated instance? I am looking at getting around this by using heartbeat messages to fill in for any administrative and application messages that should not be resent. I normally would have used SeqReset-Gap fill to deal with msgs not normally resent.
Thanks!
Jonathan Butler
Automated Trading Desk
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> I encounted a situation where a 3rd party (using FIX 4.1) is disallowing any-and-all use of the SeqReset message (35=4) when I am responding to their ResendRequest. Does this happen very often, or is this an isolated instance? I am looking at getting around this by using heartbeat messages to fill in for any administrative and application messages that should not be resent. I normally would have used SeqReset-Gap fill to deal with msgs not normally resent.
Disallowing all SeqReset messages is grossly non-compliant. Sending Gap Fills is the right thing to do in FIX 4.0 and higher when you must skip over admin messages.