Imported from previous forum
[ original email was from Scott Zionic - Scott.Zionic@petersweb.com ]
Consider the following FIX session:
PartyA>
Sends Logon, MsgSeqNum = 8
PartyB>
(expected MsgSeqNum=1)
Sends Logon
Sends ResendRequest, BeginSeqNo = 1, EndSeqNo=7
PartyA>
(messages 1-7 were admin messages)
Sends SequenceReset-GapFill, MsgSeqNo=1, NewSeqNum=8, PossDupFlag=Y
Is this correct? Should PartyA send 8 as the NewSeqNum and duplicate the sequence number used for the logon message with the next message? Or should PartyA send NewSeqNum = 9?
TIA
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> Consider the following FIX session:
>
> PartyA>
> Sends Logon, MsgSeqNum = 8
>
> PartyB>
> (expected MsgSeqNum=1)
> Sends Logon
> Sends ResendRequest, BeginSeqNo = 1, EndSeqNo=7
>
> PartyA>
> (messages 1-7 were admin messages)
> Sends SequenceReset-GapFill, MsgSeqNo=1, NewSeqNum=8, PossDupFlag=Y
>
>
>
> Is this correct? Should PartyA send 8 as the NewSeqNum and duplicate the sequence number used for the logon message with the next message? Or should PartyA send NewSeqNum = 9?
I believe Party A responded correctly to the request. B asked for messages 1-7, and A replied that 1-7 are admins, hence the next message after that is 8. B did not ask about 8, so A has no reason to examine 8 to see if it is an admin.
Now B might realize that 1-7 are now accounted for, and 8 is known, so B may next expect 9. This will result in proper behavior.
Or B might just be expecting 8. This causes a problem because when A sends 9, B will need to ask for at least 8. One would need to be careful to avoid "leapfrog" situations where B is just sending admin messages, and A keeps finding itself just a little bit behind.
I’ve seen other implementations that in the same situation will have B request 1-8 instead of 1-7. Since logons are clearly something which should never be retransmitted, A will send:
SequenceReset-GapFill, MsgSeqNo=1, NewSeqNum=9, PossDupFlag=Y
which will result in correct behavior in this one case.
My feeling is that the best thing to do would be for B to ask for 1-999999. Then A will send:
SequenceReset-GapFill, MsgSeqNo=1, NewSeqNum=9, PossDupFlag=Y
And all will be well. This solution is more robust because often the sequence gaps happen with both sides at once seeing gaps. This often results in both sides sending messages back to back which the other side believes is out of sequence. This creates another type of "leapfrog" situation. Requests from 1-999999 will cause the session to converge much faster and prevent a lot of unnecessary leapfrogging.
For instance, a person confronted with multiple messages out of sequence (i.e. 10, 11, 12) may send a flood of ResendRequests, i.e. A ResendRequest’s 1-10, 1-11, 1-12. Then B will Poss Dupe Gap Fill 1-11, 1-12, 1-13. A will receive the first Gap Fill, set inbound sequence number to 11, receive the 2nd Gap Fill and discard it because 1 < 11 and the message is PossDupe, and the same thing goes for 1-13. A is expecting 11 when B is sending 13. Repeat ad nauseum.
Had A requested 1-999999, 1-999999, 1-999999 then B will PossDupe Gap Fill 1-13, 1-13, 1-13 and A will properly expect 13.
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> Is this correct? Should PartyA send 8 as the NewSeqNum and duplicate the sequence number used for the logon message with the next message? Or should PartyA send NewSeqNum = 9?
This question may be simpler than I made it out to be.
Yes, I believe A should send 8 as the NewSeqNum because B asked for 1-7. But the next message A should normally be sending is 9. The only exception is if B asks for a ResendRequest of 8, and then 8 becomes a SeqReset-Gap Fill because Logons are never resent.
If A were to send, say, a Heartbeat or an application message with sequence number 8 after already using it for a Logon, then I think A is doing something wrong.
[ original email was from Bob Lamoureux - blamoureux@bridge.com ]
There are only two possibilities for A.
SOLUTION #1
Since A has only requested 1-7, A MUST hold on to 8 and insert it back into the message stream once 1-7 have been retransmitted and received. Party B must respond with a SeqReset to MSG 9 because that is where B is currently at after all admin messages have been taken into account.
SOLUTION #2
Party A accepts the LOGON, but asks for 1-8 in the ResendReq. This will effectively place message number 8 back on the end of any retransmission stream so that <A> doesn’t have to do any compicated queueing/insertions. When B responds with SeqReset to 9, all is back in sync
- Bob Lamoureux
> Consider the following FIX session:
>
> PartyA>
> Sends Logon, MsgSeqNum = 8
>
> PartyB>
> (expected MsgSeqNum=1)
> Sends Logon
> Sends ResendRequest, BeginSeqNo = 1, EndSeqNo=7
>
> PartyA>
> (messages 1-7 were admin messages)
> Sends SequenceReset-GapFill, MsgSeqNo=1, NewSeqNum=8, PossDupFlag=Y
>
>
>
> Is this correct? Should PartyA send 8 as the NewSeqNum and duplicate the sequence number used for the logon message with the next message? Or should PartyA send NewSeqNum = 9?
>
> TIA
>
>
>
On page 15 of the FIX 4.0 spec, in the description of Sequence Reset (Gap Fill) messages it states that…
"The message is used to reset the value of the next sequence number to be transmitted."
The "to be transmitted" means that regardless of what messages have been requested in the ResendRequest and regardless of what messages have already been sent, the MsgSeqNum of the NEXT MESSAGE THAT IS SENT will match the value in tag #36 (NewSeqNum).
So in the scenario described by Scott Zionic, Party A must have a value of "9" in the NewSeqNum field.
When a ResendRequest is made for a specific set of messages then messages outside that gap must be expected. If a FIX engine receives 1-100 and then requests 10-20, the next message it receives after the gap fill will be 101, not 21. In that case the NewSeqNum value in the SequenceReset-GapFill message will be "101".
Keith, I disagree. In your example, you run a very high risk of telling your counterparty to skip messages 10-100 vs. 10-20 on your gap fill and business messages could exist between 20-100. You are assuming that since you think you sent 100 messages your counterparty has received them all and queued them on the side. I, personally, do not think you should attempt to gap fill out of the range (i.e. NewSeqNum > (EndSeqNum + 1) ). I agree with Ryan Pierce’s posting that using infinity on the ResendRequest is a superior approach. This warrants further discussion.
> On page 15 of the FIX 4.0 spec, in the description of Sequence Reset (Gap Fill) messages it states that…
>
> "The message is used to reset the value of the next sequence number to be transmitted."
>
> The "to be transmitted" means that regardless of what messages have been requested in the ResendRequest and regardless of what messages have already been sent, the MsgSeqNum of the NEXT MESSAGE THAT IS SENT will match the value in tag #36 (NewSeqNum).
>
> So in the scenario described by Scott Zionic, Party A must have a value of "9" in the NewSeqNum field.
>
> When a ResendRequest is made for a specific set of messages then messages outside that gap must be expected. If a FIX engine receives 1-100 and then requests 10-20, the next message it receives after the gap fill will be 101, not 21. In that case the NewSeqNum value in the SequenceReset-GapFill message will be "101".
>
Hi Scott,
I’m sorry but the protocol actually seems pretty clear in this case. How can the “to be transmitted” on page 15 be interpreted to mean anything other than the outgoing MsgSeqNum value? And if the Initiating FIX engine has sent MsgSeqNum=8 and then performs a GapFill for 1-7, then the next outgoing (to be transmitted) MsgSeqNum must equal “9”. You cannot reset the value to “8” because then the other FIX engine will receive TWO messages with MsgSeqNum=8 and shutdown the connection. You can set the value to “10” but why bother? You would just be changing the value of outgoing sequence numbers on your side as well as what they expect. That just puts an empty spot where “9” would have been.
Remember that in Scott Zionic’s example, the GapFill was for a specific set of messages.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
In responding to a GapFill Request, I don’t see how the protocol could be intrepreted in any way other than I have described. The very fact that the FIX engine transmitted a ResendRequest for a specific set of messages directly implies that THAT FIX engine is HOLDING any message outside that range. In Scott Zionic’s example, the requesting engine got MsgSeqNum=8 but only requested 1-7. Since these seven messages were Admins, the responding FIX engine must transmit a SequenceReset-GapFill message to skip these messages. And since the responding engine has already transmitted MsgSeqNum=8, then the MsgSeqNum of the next message it transmits must be “9”.
Maybe it would be wiser for any FIX engine to request X-999999, but it is entirely possible that the FIX engine is smart enough to hold MsgSeqNum’s 8+ until it receives a satisfactory resolution on 1-7. Meaning that if the Initiating engine fires out messages 8 (Logon) through 100, the accepting engine may be capable of queueing 8-100 until it receives the GapFill for 1-7.
In fact, building a FIX engine capable of handling specific GapFills (as opposed to open-ended GapFills) would cut down on the bandwidth problem. In my example above, only 7 messages would get re-transmitted. Not 100.
> Keith, I disagree. In your example, you run a very high risk of telling your counterparty to skip messages 10-100 vs. 10-20 on your gap fill and business messages could exist between 20-100. You are assuming that since you think you sent 100 messages your counterparty has received them all and queued them on the side. I, personally, do not think you should attempt to gap fill out of the range (i.e. NewSeqNum > (EndSeqNum + 1) ). I agree with Ryan Pierce’s posting that using infinity on the ResendRequest is a superior approach. This warrants further discussion.
>
> > On page 15 of the FIX 4.0 spec, in the description of Sequence Reset (Gap Fill) messages it states that…
> >
> > “The message is used to reset the value of the next sequence number to be transmitted.”
> >
> > The “to be transmitted” means that regardless of what messages have been requested in the ResendRequest and regardless of what messages have already been sent, the MsgSeqNum of the NEXT MESSAGE THAT IS SENT will match the value in tag #36 (NewSeqNum).
> >
> > So in the scenario described by Scott Zionic, Party A must have a value of “9” in the NewSeqNum field.
> >
> > When a ResendRequest is made for a specific set of messages then messages outside that gap must be expected. If a FIX engine receives 1-100 and then requests 10-20, the next message it receives after the gap fill will be 101, not 21. In that case the NewSeqNum value in the SequenceReset-GapFill message will be “101”.
> >
>
Doh!
I want to apologize if my prior post came across as harsher than it intended. In re-reading Scott Atwell’s response to my post, I can see that Scott was disagreeing with my belief that a closed ResendRequest was better than an open-ended one.
I had thought Scott’s disagreement was with the FIX-correct value for the NewSeqNum tag! Oops!
In rethinking the situation, I definitely see what Scott is saying with the danger of skipping messages 21-100 in the example I use! So in light of this scenario, I have to agree with Scott and everyone else who recommends that ResendRequests take the form of X to 999999. The cost of bandwidth and the hassle of message duplication is outweighed by the potential business ramifications of lost messages.
But if an engine does make a closed ResendRequest, would it be appropriate to go ahead and Resend all messages (PossDupFlag=Y) from the beginning of the requested GapFill to the most recently sent message? The other engine should just drop the duplicates right? And that way you would be more certain that all messages had been received? It may seem redundant but it would conform to the protocol, right?!?!
Is there any talk of moving to this one and only method of Gap Recovery in future versions of the protocol?
> Hi Scott,
>
> I’m sorry but the protocol actually seems pretty clear in this case. How can the “to be transmitted” on page 15 be interpreted to mean anything other than the outgoing MsgSeqNum value? And if the Initiating FIX engine has sent MsgSeqNum=8 and then performs a GapFill for 1-7, then the next outgoing (to be transmitted) MsgSeqNum must equal “9”. You cannot reset the value to “8” because then the other FIX engine will receive TWO messages with MsgSeqNum=8 and shutdown the connection. You can set the value to “10” but why bother? You would just be changing the value of outgoing sequence numbers on your side as well as what they expect. That just puts an empty spot where “9” would have been.
>
> Remember that in Scott Zionic’s example, the GapFill was for a specific set of messages.
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
>
> In responding to a GapFill Request, I don’t see how the protocol could be intrepreted in any way other than I have described. The very fact that the FIX engine transmitted a ResendRequest for a specific set of messages directly implies that THAT FIX engine is HOLDING any message outside that range. In Scott Zionic’s example, the requesting engine got MsgSeqNum=8 but only requested 1-7. Since these seven messages were Admins, the responding FIX engine must transmit a SequenceReset-GapFill message to skip these messages. And since the responding engine has already transmitted MsgSeqNum=8, then the MsgSeqNum of the next message it transmits must be “9”.
>
> Maybe it would be wiser for any FIX engine to request X-999999, but it is entirely possible that the FIX engine is smart enough to hold MsgSeqNum’s 8+ until it receives a satisfactory resolution on 1-7. Meaning that if the Initiating engine fires out messages 8 (Logon) through 100, the accepting engine may be capable of queueing 8-100 until it receives the GapFill for 1-7.
>
> In fact, building a FIX engine capable of handling specific GapFills (as opposed to open-ended GapFills) would cut down on the bandwidth problem. In my example above, only 7 messages would get re-transmitted. Not 100.
>
>
>
> > Keith, I disagree. In your example, you run a very high risk of telling your counterparty to skip messages 10-100 vs. 10-20 on your gap fill and business messages could exist between 20-100. You are assuming that since you think you sent 100 messages your counterparty has received them all and queued them on the side. I, personally, do not think you should attempt to gap fill out of the range (i.e. NewSeqNum > (EndSeqNum + 1) ). I agree with Ryan Pierce’s posting that using infinity on the ResendRequest is a superior approach. This warrants further discussion.
> >
> > > On page 15 of the FIX 4.0 spec, in the description of Sequence Reset (Gap Fill) messages it states that…
> > >
> > > “The message is used to reset the value of the next sequence number to be transmitted.”
> > >
> > > The “to be transmitted” means that regardless of what messages have been requested in the ResendRequest and regardless of what messages have already been sent, the MsgSeqNum of the NEXT MESSAGE THAT IS SENT will match the value in tag #36 (NewSeqNum).
> > >
> > > So in the scenario described by Scott Zionic, Party A must have a value of “9” in the NewSeqNum field.
> > >
> > > When a ResendRequest is made for a specific set of messages then messages outside that gap must be expected. If a FIX engine receives 1-100 and then requests 10-20, the next message it receives after the gap fill will be 101, not 21. In that case the NewSeqNum value in the SequenceReset-GapFill message will be “101”.
> > >
> >
>
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> I’m sorry but the protocol actually seems pretty clear in this case. How can the “to be transmitted” on page 15 be interpreted to mean anything other than the outgoing MsgSeqNum value?
I see your point. This seems to be an error in the spec that need to be addressed by errata, as it seems that the spec is requiring the behavior you describe in the case of a Resend Request from 10-20 resulting in a gap fill to 101, which quite easily would result in lost business messages.
The NewSeqNo is not the next message to be transmitted, but rather, the next message in sequence following the gap. In this case of a Resend Request from 1-7, NewSeqNo is 8 even though the next message to be transmitted is 9. Even in the case of an open-ended Resend Request, NewSeqNo isn’t necessarily the next message to be transmitted. For instance, if the other side sends, in a row, two “Resend Request 1-999999” the proper response is: “1 Poss Dupe Gap Fill, NewSeqNo=9” “1 Poss Dupe Gap Fill, NewSeqNo=9” so even though NewSeqNo=9, the next message transmitted after the first gap fill was 1.