Imported from previous forum
I was wondering if anyone has run into the following scenario and what was the resolution:
FIX Engine A missed 5 messages and sends a Resend-Request to FIX Engine B for 10 - 0
FIX B queries the database for the requested messages
While the query is in progress the timer runs out and FIX B needs to send a heartbeat to maintain connection
What sequence number should the Heartbeat go on?
Sending the Heartbeat on 10 will interfere with the expected contents of the resend.
Sending the Heartbeat on 15 will trigger another Resend-Request and cause an infinite loop resend scenario.
Are there any other options?
I would appreciate any help you can provide.
Greg Ginzburg
Assuming the lost messages had sequence numbers 10 thru 14, you cannot send the heartbeat with sequence number 10. You should send the heartbeat with sequence number. Every message you generate should have a unique sequence number (in versions 4.0 and greater anyway).
Sending the heartbeat with sequence number 15 should not cause an infinite loop. You may get several resend requests before you catch up, but it should not be an infinite loop. If you find yourself generating more than one heartbeat before you are able to fill the resend request (for 5 messages), either the HeartBtInt is too small or you have a problem with the database.
> I was wondering if anyone has run into the following scenario and what was the resolution:
>
> FIX Engine A missed 5 messages and sends a Resend-Request to FIX Engine B for 10 - 0
> FIX B queries the database for the requested messages
> While the query is in progress the timer runs out and FIX B needs to send a heartbeat to maintain connection
>
> What sequence number should the Heartbeat go on?
>
> Sending the Heartbeat on 10 will interfere with the expected contents of the resend.
> Sending the Heartbeat on 15 will trigger another Resend-Request and cause an infinite loop resend scenario.
>
> Are there any other options?
>
> I would appreciate any help you can provide.
>
> Greg Ginzburg
>
>
Note also that the "timer" related to sending heartbeat messages should be triggered after a period of "inactivity". If you resend a message in response to Resend Request, that constitutes activity and thus your timer would be reset again. So I agree with Edward, that the scenario you have presented should not create an infinite loop and that the MsgSeqNum of your heartbeat in that example should be 15.
> Assuming the lost messages had sequence numbers 10 thru 14, you cannot send the heartbeat with sequence number 10. You should send the heartbeat with sequence number. Every message you generate should have a unique sequence number (in versions 4.0 and greater anyway).
>
> Sending the heartbeat with sequence number 15 should not cause an infinite loop. You may get several resend requests before you catch up, but it should not be an infinite loop. If you find yourself generating more than one heartbeat before you are able to fill the resend request (for 5 messages), either the HeartBtInt is too small or you have a problem with the database.
>
> > I was wondering if anyone has run into the following scenario and what was the resolution:
> >
> > FIX Engine A missed 5 messages and sends a Resend-Request to FIX Engine B for 10 - 0
> > FIX B queries the database for the requested messages
> > While the query is in progress the timer runs out and FIX B needs to send a heartbeat to maintain connection
> >
> > What sequence number should the Heartbeat go on?
> >
> > Sending the Heartbeat on 10 will interfere with the expected contents of the resend.
> > Sending the Heartbeat on 15 will trigger another Resend-Request and cause an infinite loop resend scenario.
> >
> > Are there any other options?
> >
> > I would appreciate any help you can provide.
> >
> > Greg Ginzburg
> >
> >
>
I don’t think that the situation is so simple. Edward and Scott rely on general considerations. For example, HeartBtInt can be set very small intentionally or DB access can take a while just because of any reason.
There could be other messages that are sent before reply to Resend-Request also. For example, existing lines are not of speed of light, and messages are not received immediately as they are sent. It means that messages after gap can come in portions of 10-20. So if every message will cause Resend-Request the connection will be flooded with unnecessary requests and PossDup replies.
In my mind FIX server should be robust enough to deal with all these situations. E.g. our FIX server never requests messages that are already came (upper boundary is never set 0 or 999999).
If it receives message after gap while gap is not filled yet and one Resend-Request has already sent recently, it does not send one more Resend-Request ("recentness" is parametrized).
Again, if FIX server receives one more Resend-Request while previous one is not completed yet, they merged (even they are not continuous). And so on.
> Note also that the "timer" related to sending heartbeat messages should be triggered after a period of "inactivity". If you resend a message in response to Resend Request, that constitutes activity and thus your timer would be reset again. So I agree with Edward, that the scenario you have presented should not create an infinite loop and that the MsgSeqNum of your heartbeat in that example should be 15.
>
> > Assuming the lost messages had sequence numbers 10 thru 14, you cannot send the heartbeat with sequence number 10. You should send the heartbeat with sequence number. Every message you generate should have a unique sequence number (in versions 4.0 and greater anyway).
> >
> > Sending the heartbeat with sequence number 15 should not cause an infinite loop. You may get several resend requests before you catch up, but it should not be an infinite loop. If you find yourself generating more than one heartbeat before you are able to fill the resend request (for 5 messages), either the HeartBtInt is too small or you have a problem with the database.
> >
> > > I was wondering if anyone has run into the following scenario and what was the resolution:
> > >
> > > FIX Engine A missed 5 messages and sends a Resend-Request to FIX Engine B for 10 - 0
> > > FIX B queries the database for the requested messages
> > > While the query is in progress the timer runs out and FIX B needs to send a heartbeat to maintain connection
> > >
> > > What sequence number should the Heartbeat go on?
> > >
> > > Sending the Heartbeat on 10 will interfere with the expected contents of the resend.
> > > Sending the Heartbeat on 15 will trigger another Resend-Request and cause an infinite loop resend scenario.
> > >
> > > Are there any other options?
> > >
> > > I would appreciate any help you can provide.
> > >
> > > Greg Ginzburg
> > >
> > >
> >
>
I agree with what you have stated, the spec recommends that you queue messages which are too high during resend request processing to avoid flooding and infinite looping, etc. However, the question that was asked was quite specific (narrower than your comment) and was suggesting that hitting a heartbeat interval or sending a Heartbeat could trigger an infinite loop which is not the case.
> I don’t think that the situation is so simple. Edward and Scott rely on general considerations. For example, HeartBtInt can be set very small intentionally or DB access can take a while just because of any reason.
> There could be other messages that are sent before reply to Resend-Request also. For example, existing lines are not of speed of light, and messages are not received immediately as they are sent. It means that messages after gap can come in portions of 10-20. So if every message will cause Resend-Request the connection will be flooded with unnecessary requests and PossDup replies.
>
> In my mind FIX server should be robust enough to deal with all these situations. E.g. our FIX server never requests messages that are already came (upper boundary is never set 0 or 999999).
> If it receives message after gap while gap is not filled yet and one Resend-Request has already sent recently, it does not send one more Resend-Request (“recentness” is parametrized).
> Again, if FIX server receives one more Resend-Request while previous one is not completed yet, they merged (even they are not continuous). And so on.
>
> > Note also that the “timer” related to sending heartbeat messages should be triggered after a period of “inactivity”. If you resend a message in response to Resend Request, that constitutes activity and thus your timer would be reset again. So I agree with Edward, that the scenario you have presented should not create an infinite loop and that the MsgSeqNum of your heartbeat in that example should be 15.
> >
> > > Assuming the lost messages had sequence numbers 10 thru 14, you cannot send the heartbeat with sequence number 10. You should send the heartbeat with sequence number. Every message you generate should have a unique sequence number (in versions 4.0 and greater anyway).
> > >
> > > Sending the heartbeat with sequence number 15 should not cause an infinite loop. You may get several resend requests before you catch up, but it should not be an infinite loop. If you find yourself generating more than one heartbeat before you are able to fill the resend request (for 5 messages), either the HeartBtInt is too small or you have a problem with the database.
> > >
> > > > I was wondering if anyone has run into the following scenario and what was the resolution:
> > > >
> > > > FIX Engine A missed 5 messages and sends a Resend-Request to FIX Engine B for 10 - 0
> > > > FIX B queries the database for the requested messages
> > > > While the query is in progress the timer runs out and FIX B needs to send a heartbeat to maintain connection
> > > >
> > > > What sequence number should the Heartbeat go on?
> > > >
> > > > Sending the Heartbeat on 10 will interfere with the expected contents of the resend.
> > > > Sending the Heartbeat on 15 will trigger another Resend-Request and cause an infinite loop resend scenario.
> > > >
> > > > Are there any other options?
> > > >
> > > > I would appreciate any help you can provide.
> > > >
> > > > Greg Ginzburg
> > > >
> > > >
> > >
> >
>
Ok, let us return to the HeartBeat case. First of all, it concerns FIX server logic rather than protocol itself. The problem is in what protocol recommends but not obliges, giving too much freedom to server creators.
Now the situation described by Greg. Server A received messages 1-9 and awaited message 10. Instead, it receives message 14. As implied by Greg, it disregards this message and requests retransmission all messages after #10, i.e. 10-14. Suppose there is no more travelling messages from B to A in the line and server B receives Resend-Request immediately (these are unusual conditions, but let them be). Suppose also that on the side B sent messages are not stored in the memory for whatever reason (small memory, faster operations, etc. - these reasons are very common) and heartbeat interval is set rather small because of the line quality (it is also usual). So small, that it is smaller then the period of time needed to extract sent messages from outer source (database, file, other computer in internal network, etc.). Can it be possible? Yes, it is likely to be possible nowadays.
BTW, imagine that server B has more application messages to send. Protocol does not prohibit server B from sending those messages with higher sequence numbers during recovery process mixing them with poss-dup messages. It can do so considering other party (i.e. server A) to be smart enough to hold these messages until gap filling process ends. And the reason to do so is to use standing line while messages for resend are not ready.
But imagine there are no such messages, but server B needs to send Heartbeat. Server B sends a Heartbeat to prevent connection from dropping by server A (or even it receives TestRequest from A and must do it). Certainly, it should send it in time and with the next number 15.
Now server A receives Heartbeat. As earlier, according its implied logic it disregards this Heartbeat and sends Resend-Request back to server B. Hopefully, it counts this message as network activity and does not send Test-Request 
What happens at the side B? Finally messages are extracted from outer source and ready to send and being sent to server A. But next is Resend-Request again! And now server B needs to send actually messages 10-15 (remember Heartbeat!). And now situation is repeating (see the beginning) - waiting for extracting messages from outer source, sending Heartbeat, resending messages and… receiving Resend-Request again. Definite loop. I mean, indefinite loop
Moreover, imagine, that we have gaps in both directions (that is quite usual if we have problems on the line). Resend-Request from B to A can cause the same problems and even worse because there is no time to wait - it is sent immediately upon receiving out-of-order message, that is counterpart Resend-Request. And it is disregarded because it is sent with higher number! We encountered this situation with one of our customers even using our beautiful, perfect, bla-bla-bla FIX server Hopefully, there were fading oscillations, but unfortunately floody and useless.
What are factors that influences the situation? Heartbeat interval, time to extract messages from outer source, line quality and, of course, server logic. The second and the third factors are beyond FIX and there can be no way to improve it. Or it could be just because of temporary drawback or outage - nobody can guarantee you 100% (or it can be very, very expensive :-), and even in this case…). You say, heartbeat interval can be increased, but this interval is usually determined by network conditions rather then outer source speed and in some cases cannot serve both purposes. So what is the rest? Server logic can be significantly improved. But if servers are stupid enough the situation described above should follow any line error or database delay.
You can say, improve your lines, improve your computers, improve your database, etc. Yes, it is better to be rich and healthy than to be poor and ill. But be realistic. Protocol was introduced to increase reliability and stated to be proof against transfer problems. And in case of 24/7 lines, power, no hardware failures and software bugs it is just useless.
How can FIX protocol definition help? I believe protocol definition should force smart logic and discourage stupid implementations. What about prohibition of requesting all messages, that is with undefined (0 or 99999) upper border? What about prohibition of disregarding already received messages? About merging consecutive Resend-Requests if they are not accomplished yet?
I understand, that some of these suggestions are not for protocol but for implementation notes. However FIX protocol description already contains some of them, why not to add a little more?
Best regards,
Yury
> I agree with what you have stated, the spec recommends that you queue messages which are too high during resend request processing to avoid flooding and infinite looping, etc. However, the question that was asked was quite specific (narrower than your comment) and was suggesting that hitting a heartbeat interval or sending a Heartbeat could trigger an infinite loop which is not the case.
As per our Fix Server, we first process the resend Request, and then send the Heartbeat, because even though we may send a heartbeat with sequence number 15 (our outgoing seqNum) (while the other side is waiting for 10, and has asked for a gapfill from 10-0), they will ignore it as the sequence number is greater than what they are expecting.
Also, every Message after the gap is detected, should not cause a resend request to be generated. For eg. In our FIX Server, once a gap is detected say with seqnum 10, msgs with seqNum 11-20 (if in buffer ) will not cause a resend request to be generated. We wait for a time interval & if no response to resend request is recvd, only then another request is sent. This prevents the connection to be flooded with unnecessary requests.
>
Assuming the lost messages had sequence numbers 10 thru 14, you cannot send the heartbeat with sequence number 10. You should send the heartbeat with sequence number. Every message you generate should have a unique sequence number (in versions 4.0 and greater anyway).
>
> Sending the heartbeat with sequence number 15 should not cause an infinite loop. You may get several resend requests before you catch up, but it should not be an infinite loop. If you find yourself generating more than one heartbeat before you are able to fill the resend request (for 5 messages), either the HeartBtInt is too small or you have a problem with the database.
>
> > I was wondering if anyone has run into the following scenario and what was the resolution:
> >
> > FIX Engine A missed 5 messages and sends a Resend-Request to FIX Engine B for 10 - 0
> > FIX B queries the database for the requested messages
> > While the query is in progress the timer runs out and FIX B needs to send a heartbeat to maintain connection
> >
> > What sequence number should the Heartbeat go on?
> >
> > Sending the Heartbeat on 10 will interfere with the expected contents of the resend.
> > Sending the Heartbeat on 15 will trigger another Resend-Request and cause an infinite loop resend scenario.
> >
> > Are there any other options?
> >
> > I would appreciate any help you can provide.
> >
> > Greg Ginzburg
> >
> >
>
> As per our Fix Server, we first process the resend Request, and then send the Heartbeat, because even though we may send a heartbeat with sequence number 15 (our outgoing seqNum) (while the other side is waiting for 10, and has asked for a gapfill from 10-0), they will ignore it as the sequence number is greater than what they are expecting.
We reset our timer whenever a valid Fix message is received, whether we process it or not. In the above case, Heartbeat 15 would reset the timer.
Paul van Diepen
> As per our Fix Server, we first process the resend Request, and then send the Heartbeat, because even though we may send a heartbeat with sequence number 15 (our outgoing seqNum) (while the other side is waiting for 10, and has asked for a gapfill from 10-0), they will ignore it as the sequence number is greater than what they are expecting.
We reset our timer whenever a valid Fix message is received, whether we process it or not. In the above case, Heartbeat 15 would reset the timer.
Paul van Diepen