Received ResendRequest for huge number of messages and missed Heartbeat

, ,
1

Imported from previous forum

2

[ original email was from Olexiy Getmanchuk - olexiyg@in4reach.com ]
Hello,

Is there anyone who knows how both sides should handle situation when handling of ResendRequest takes much more time then Heartbeat interval?

Assuming such situation:

  1. Side A has sent huge number of executions (34=[5-9999999])
  2. Side B by some reasons lost all of them and requested Side A to resend them all (35=2, 7=5, 16=0) while having nothing to send to side B.
  3. Side A start resending messages. Because of huge amount of messages that could take more then heartbeat interval to resend them.
  4. Side B have sent Heartbeat but it didn’t make to side B. Let’s assume that happened between side A was resending messages 34=1000 and 34=1001.
  5. What should side A do?

Please, should if you decided to answer, provide an example with sequence numbers or use one above.

Thanks in advance,
–Olexiy.

3

[ original email was from Daniel Graham - danny@pelynch.com ]
From the Fix 4.2 spec, and from the general purpose of a heartbeat,
B shouldn’t send the heartbeat … the heartbeat is used when NO DATA
has been sent for a number of seconds, in your case, you would be
sending/receiving lots of data so the FIX connection can be assumed to
be ok, and hence you don’t need to send the heartbeat.

D

-----8<-----
When either end of a FIX connection has not sent any data for [HeartBtInt] seconds, it will transmit a
Heartbeat message. When either end of the connection has not received any data for (HeartBtInt +
“some reasonable transmission time”) seconds, it will transmit a Test Request message. If there is still
no Heartbeat message received after (HeartBtInt + “some reasonable transmission time”) seconds then
the connection should be considered lost and corrective action be initiated. If HeartBtInt is set to zero
then no regular heartbeat messages will be generated. Note that a test request message can still be sent
independent of the value of the HeartBtInt, which will force a Heartbeat message.
Heartbeats issued as the result of Test Request must contain the TestReqID transmitted in the Test
Request message. This is useful to verify that the Heartbeat is the result of the Test Request and not as
the result of a regular timeout.

Hello,

Is there anyone who knows how both sides should handle situation when
handling of ResendRequest takes much more time then Heartbeat interval?

Assuming such situation:

  1. Side A has sent huge number of executions (34=[5-9999999])
  2. Side B by some reasons lost all of them and requested Side A to
    resend them all (35=2, 7=5, 16=0) while having nothing to send
    to side B.
  3. Side A start resending messages. Because of huge amount of messages
    that could take more then heartbeat interval to resend them.
  4. Side B have sent Heartbeat but it didn’t make to side B. Let’s
    assume that happened between side A was resending messages 34=1000
    and 34=1001.
  5. What should side A do?

Please, should if you decided to answer, provide an example with
sequence numbers or use one above.

Thanks in advance, --Olexiy.

4

[ original email was from Olexiy Getmanchuk - olexiyg@in4reach.com ]
Thanks, Daniel, for your attention but you probably misunderstood my question. In my example side A will be resending messages while side B will send nothing at all. In such case, according to FIX Spec, side B should send Heartbeat msg to A. My question was what should side A do if that Heartbeat message was lost in transmission. According to FIX Spec side A should send TestRequest. But in my example the problem is that that TestRequest should be send in the middle of response to ResendRequest (when you cannot issue new message with new seqNum) and if you send ResendRequest with new seqNum (in my example it would be 9999999+1=10000000) then it should be processed according to its seqNum and it’ll be late so side A should close socket (again according to FIX spec).

That’s why I’m asking this question here with hope that someone from FIX committee can clarify how it should be handled properly.

Thanks,
–Olexiy.

From the Fix 4.2 spec, and from the general purpose of a heartbeat, B
shouldn’t send the heartbeat … the heartbeat is used when NO DATA has
been sent for a number of seconds, in your case, you would be
sending/receiving lots of data so the FIX connection can be assumed to
be ok, and hence you don’t need to send the heartbeat.

D
…

Hello,

Is there anyone who knows how both sides should handle situation when
handling of ResendRequest takes much more time then Heartbeat
interval?

Assuming such situation:

  1. Side A has sent huge number of executions (34=[5-9999999])
  2. Side B by some reasons lost all of them and requested Side A to
    resend them all (35=2, 7=5, 16=0) while having nothing to send to
    side B.
  3. Side A start resending messages. Because of huge amount of messages
    that could take more then heartbeat interval to resend them.
  4. Side B have sent Heartbeat but it didn’t make to side B. Let’s
    assume that happened between side A was resending messages 34=1000
    and 34=1001.
  5. What should side A do?

Please, should if you decided to answer, provide an example with
sequence numbers or use one above.

Thanks in advance, --Olexiy.

5

[ original email was from Dennis Wiatzka - dennis.wiatzka@computershare.com ]
This scenario is pretty extreme in its assumptions but here is how most production systems will resolve it.

  • Side A disconnects after heartbeat period + some reasonable time expires after the test request was sent (per the spec I believe)
  • Side A and B re-establish the connection
  • Side B will have received sequence numbers 5 through n during the retransmission processing before disconnection (assume n = 5000000); n will always be greater than 5 of course, since messages will be retransmitted and processed during the time the heartbeat interval elapsed and during the time after the test request was sent while Side A times out waiting
  • after successful logon Side B requests Side A to resend all messages after sequence number 5000000 (35=2, 7=5000000, 16=0)

This scenario repeats every time there is a disconnection caused by the unserviced test request but eventually the two sides are back in sync since each time through the amount of messages requiring retransmission becomes smaller until it reaches zero.

Regards, Dennis

Thanks, Daniel, for your attention but you probably misunderstood my
question. In my example side A will be resending messages while side B
will send nothing at all. In such case, according to FIX Spec, side B
should send Heartbeat msg to A. My question was what should side A do
if that Heartbeat message was lost in transmission. According to FIX
Spec side A should send TestRequest. But in my example the problem is
that that TestRequest should be send in the middle of response to
ResendRequest (when you cannot issue new message with new seqNum) and
if you send ResendRequest with new seqNum (in my example it would be
9999999+1=10000000) then it should be processed according to its
seqNum and it’ll be late so side A should close socket (again
according to FIX spec).

That’s why I’m asking this question here with hope that someone from FIX
committee can clarify how it should be handled properly.

Thanks, --Olexiy.

From the Fix 4.2 spec, and from the general purpose of a heartbeat, B
shouldn’t send the heartbeat … the heartbeat is used when NO DATA has
been sent for a number of seconds, in your case, you would be
sending/receiving lots of data so the FIX connection can be assumed to
be ok, and hence you don’t need to send the heartbeat.

D
…

Hello,

Is there anyone who knows how both sides should handle situation
when handling of ResendRequest takes much more time then Heartbeat
interval?

Assuming such situation:

  1. Side A has sent huge number of executions (34=[5-9999999])
  2. Side B by some reasons lost all of them and requested Side A to
    resend them all (35=2, 7=5, 16=0) while having nothing to send to
    side B.
  3. Side A start resending messages. Because of huge amount of
    messages that could take more then heartbeat interval to resend
    them.
  4. Side B have sent Heartbeat but it didn’t make to side B. Let’s
    assume that happened between side A was resending messages
    34=1000 and 34=1001.
  5. What should side A do?

Please, should if you decided to answer, provide an example with
sequence numbers or use one above.

Thanks in advance, --Olexiy.

6

[ original email was from Olexiy Getmanchuk - olexiyg@in4reach.com ]
Thank you Dennis!
Finally someone got my point!

I assume that in your scenario before the very first step you mentioned side A had sent TestRequest with [34=10000000] in the middle of resending messages [5…9999999].

Am I right?

Thanks,
–Olexiy.

This scenario is pretty extreme in its assumptions but here is how most
production systems will resolve it.

  • Side A disconnects after heartbeat period + some reasonable time
    expires after the test request was sent (per the spec I believe)
  • Side A and B re-establish the connection
  • Side B will have received sequence numbers 5 through n during the
    retransmission processing before disconnection (assume n = 5000000); n
    will always be greater than 5 of course, since messages will be
    retransmitted and processed during the time the heartbeat interval
    elapsed and during the time after the test request was sent while Side
    A times out waiting
  • after successful logon Side B requests Side A to resend all messages
    after sequence number 5000000 (35=2, 7=5000000, 16=0)

This scenario repeats every time there is a disconnection caused by the
unserviced test request but eventually the two sides are back in sync
since each time through the amount of messages requiring retransmission
becomes smaller until it reaches zero.

Regards, Dennis

7

[ original email was from Dennis Wiatzka - dennis.wiatzka@computershare.com ]
Yes, that is correct. TestRequest has [34=10000000] and is sent because (per your scenario) the heart beat sent by Side B was lost before step 1 in my posting.

In most cases production systems are using TCP (not UDP, for example) over sufficient bandwidth in which cases such message loss (e.g. the heartbeat) is not going to occur - the connection will simply be dropped by the network.

If the two parties are using a communication medium which loses messages without dropping the connection, then perrhaps they should rethink their choice of medium.

Thank you Dennis! Finally someone got my point!

I assume that in your scenario before the very first step you mentioned
side A had sent TestRequest with [34=10000000] in the middle of
resending messages [5…9999999].

Am I right?

Thanks, --Olexiy.

This scenario is pretty extreme in its assumptions but here is how
most production systems will resolve it.

  • Side A disconnects after heartbeat period + some reasonable time
    expires after the test request was sent (per the spec I believe)
  • Side A and B re-establish the connection
  • Side B will have received sequence numbers 5 through n during the
    retransmission processing before disconnection (assume n = 5000000);
    n will always be greater than 5 of course, since messages will be
    retransmitted and processed during the time the heartbeat interval
    elapsed and during the time after the test request was sent while
    Side A times out waiting
  • after successful logon Side B requests Side A to resend all messages
    after sequence number 5000000 (35=2, 7=5000000, 16=0)

This scenario repeats every time there is a disconnection caused by
the unserviced test request but eventually the two sides are back in
sync since each time through the amount of messages requiring
retransmission becomes smaller until it reaches zero.

Regards, Dennis

8

[ original email was from Olexiy Getmanchuk - olexiyg@in4reach.com ]
Thanks for sharing your experience.

I just simplified the case saying that Heartbeat was lost due to the network. It might happen that side B had some problem on their side (e.g. time synchronization which caused sending that Heartbeat later), or high CPU load, or - whatever other reason caused time delay.

Although it sounds more like a work around of a problem rather then well thought protection against message loss (as people would expect from messaging protocol) it sounds like a working solution.

This scenario is pretty extreme in its assumptions but here is how
most production systems will resolve it.

If most systems implemented that way, should not it be considered as de-facto standard?
If yes, do you think FIX committee is going to include this use case in the list of Test cases?

BTW, do you know is there any other special forum dedicated especially for FIX session protocol (handling ResetRequests/TestRequests/SeqNum mismatch)? I cannot find anything in FIX spec related to handling other extreme cases, like ResendRequest message loss, or reply to ResendRequest loss. Since FIX spec claims to be neutral to physical layer I would expect them to cover such cases.