Imported from previous forum
Hi,
I’m wondering if it’s legal to assume there will be no more ResendRequests for messages that have been “acknowledged” by a TestRequest response.
Basically what I want is to implement a sliding window of outbound messages. At regular intervals I send a TestRequest, wait for the response and then remove all messages prior to the TestRequest from the send-window, because they are confirmed to be received and processed by the counterparty (so they won’t need retransmitting in the future).
Is this approach valid and will a proper FIX engine guarantee that all preceeding messages have been received when it responds to a TestRequest? And is it true that this counterparty is guaranteed never to ask for a retransmission for such acknowledged messages?
regards,
Erik
[ original email was from Dean Kauffman - dean.kauffman@tradeweb.com ]
Although your idea sounds reasonable implementing it and enforcing it will have some complications. There are FIX engines out there that do not bother to safe-store protocol-level messages (heartbeats, test-requests, resend-requests, etc.) and thus burn a range of incoming sequence numbers that are forgotten on session recovery. Recently we received a resend request 4-0 for an idle session that was up to sequence 1479. The size of the range raised some concerns from personnel monitoring the network, but it was a “normal” occurrence.
Hi,
I’m wondering if it’s legal to assume there will be no more
ResendRequests for messages that have been “acknowledged” by a
TestRequest response.Basically what I want is to implement a sliding window of outbound
messages. At regular intervals I send a TestRequest, wait for the
response and then remove all messages prior to the TestRequest from the
send-window, because they are confirmed to be received and processed by
the counterparty (so they won’t need retransmitting in the future).Is this approach valid and will a proper FIX engine guarantee that all
preceeding messages have been received when it responds to a
TestRequest? And is it true that this counterparty is guaranteed never
to ask for a retransmission for such acknowledged messages?regards, Erik
Although your idea sounds reasonable implementing it and enforcing it
will have some complications. There are FIX engines out there that do
not bother to safe-store protocol-level messages (heartbeats, test-
requests, resend-requests, etc.) and thus burn a range of incoming
sequence numbers that are forgotten on session recovery. Recently we
received a resend request 4-0 for an idle session that was up to
sequence 1479. The size of the range raised some concerns from personnel
monitoring the network, but it was a “normal” occurrence.
Gee… you happen to know which product the counterparty was using?
Anyway, you think it’d be safe to still purge outbound messages after an online sequence reset (using Logon to gracefully reset the seqno’s back to 1)? Since the seqno’s are back to 1, answering retransmissions for older messages would seem ambiguous. You think your counterparty would behave differently after such online sequence reset?
Hi,
I’m wondering if it’s legal to assume there will be no more
ResendRequests for messages that have been “acknowledged” by a
TestRequest response.Basically what I want is to implement a sliding window of outbound
messages. At regular intervals I send a TestRequest, wait for the
response and then remove all messages prior to the TestRequest from
the send-window, because they are confirmed to be received and
processed by the counterparty (so they won’t need retransmitting in
the future).Is this approach valid and will a proper FIX engine guarantee that all
preceeding messages have been received when it responds to a
TestRequest? And is it true that this counterparty is guaranteed never
to ask for a retransmission for such acknowledged messages?regards, Erik
Hi,
in theory, though unorthodox that sounds ok, but there may be times when an opposite engine (somehow) loses its messages and asks for a resend. Then you would be stuck…
my 2p
Hi,
I’m wondering if it’s legal to assume there will be no more
ResendRequests for messages that have been “acknowledged” by a
TestRequest response.Basically what I want is to implement a sliding window of outbound
messages. At regular intervals I send a TestRequest, wait for the
response and then remove all messages prior to the TestRequest from the
send-window, because they are confirmed to be received and processed by
the counterparty (so they won’t need retransmitting in the future).Is this approach valid and will a proper FIX engine guarantee that all
preceeding messages have been received when it responds to a
TestRequest? And is it true that this counterparty is guaranteed never
to ask for a retransmission for such acknowledged messages?regards, Erik
[ original email was from Matt Simpson - msimpson@cme.com ]
Erik -
It’s an interesting concept but the specified use for Test Request as defined in the 4.4 spec is to elicit a Heartbeat from the counterparty - nothing else. I would be relecutant to see the use of Test Request expanded to cover Resend Request mitigation unless it had been reviewed by the GTC for inclusion in the spec. We’d be interested to look into this further if you would like to engage us.
Hi,
I’m wondering if it’s legal to assume there will be no more
ResendRequests for messages that have been “acknowledged” by a
TestRequest response.Basically what I want is to implement a sliding window of outbound
messages. At regular intervals I send a TestRequest, wait for the
response and then remove all messages prior to the TestRequest from the
send-window, because they are confirmed to be received and processed by
the counterparty (so they won’t need retransmitting in the future).Is this approach valid and will a proper FIX engine guarantee that all
preceeding messages have been received when it responds to a
TestRequest? And is it true that this counterparty is guaranteed never
to ask for a retransmission for such acknowledged messages?regards, Erik
Matt,
It would seem premature to ask for inclusion in the spec. However, it would seem useful to have a means for discovering just how far the counterparty is behind.
For example, in his response to this message, Dean Kauffman explained how his counterparty asked for a retransmission of a significant part of the trading day. Effectively that means that he can’t be sure about when his old outbound messages safely made it across.
Worse, assuming Dean’s counterparty really had received and processed the messages, but ‘forgot’ to remember that when it sent its ResendRequest, Dean’s logic could decide to send GapFills instead of orders, leaving them in an inconsistent state.
Generally I think it’d be useful for FIX to provide a reliable feedback mechanism of this kind and I was hoping a TestRequest response (as well as LastMsgSeqNumProcessed in 4.4) offered that.
I’ll abandon my sliding window, but do you think it would still be ok if I purged the outbound window after a resetting sequence numbers back to 1?
-Erik
Erik -
It’s an interesting concept but the specified use for Test Request as
defined in the 4.4 spec is to elicit a Heartbeat from the counterparty -
nothing else. I would be relecutant to see the use of Test Request
expanded to cover Resend Request mitigation unless it had been reviewed
by the GTC for inclusion in the spec. We’d be interested to look into
this further if you would like to engage us.Hi,
I’m wondering if it’s legal to assume there will be no more
ResendRequests for messages that have been “acknowledged” by a
TestRequest response.Basically what I want is to implement a sliding window of outbound
messages. At regular intervals I send a TestRequest, wait for the
response and then remove all messages prior to the TestRequest from
the send-window, because they are confirmed to be received and
processed by the counterparty (so they won’t need retransmitting in
the future).Is this approach valid and will a proper FIX engine guarantee that all
preceeding messages have been received when it responds to a
TestRequest? And is it true that this counterparty is guaranteed never
to ask for a retransmission for such acknowledged messages?regards, Erik
I’ll abandon my sliding window, but do you think it would still be ok
if I purged the outbound window after a resetting sequence numbers
back to 1?
Erik, consistent with ‘abandoning your sliding window’, I would advise against purging sent messages simply because you know the counterparty received them. There is nothing to prevent, and in a disaster recovery scenario there is a benefit to, a counterparty issuing a Resend Request for them. Effectively the “recovery window” becomes those messages since MsgSeqNum of 1. I recall discussion (thought it was captured in the spec but didn’t find it during my cursory re-review) back in the '90s around using the FIX message recovery mechanism to recover messages to an alternate system in the event of a disaster. Firms often take this recovery window into consideration when determining the optimal time to reset back to 1 each day.
Having a log of the messages you have sent may also be useful to resolve a dispute or for audit purposes.
I do know systems that purge their message queues when MsgSeqNum are reset back to 1. (we actually retain a few days worth)
The gist is that LastMsgSeqNumProcessed and other mechanisms to determine if the counterparty has properly received messages up to a given point are valuable in terms of knowing that they got the message, but I would not recommend using that to drive deleting one’s sent message log history for the day as they could still ask for it again.