Identifying Duplicate messages

, ,
1

Imported from previous forum

2

[ original email was from Sreedhar Tulluri - sreedhar_tulluri@ml.com ]
We encountered a strange sceneraio ,where a counterparty sent back messages with tag 43=Y , responding to a resend request from a buy-side.
These messages are not duplicate but tagged as PossibleDuplicate.

Here is the message sequence:

1 - Buy Side lost FIX session
2 - Sell side queued up all the messages
3 - Buy side reconnected and sent "Resend Request"
4 - Sell side sent all queued up messages with 43=Y

Is this a correct behaviour as per FIX4.2 ?

How can we identify a true duplicate message from a possible duplicate msg?

  • Sreedhar Tulluri
3

[ original email was from John Prewett - jprewett@lavatrading.com ]
Hi Sreedhar,

This is absolutely correct behavior.

I would hope that the FIX session layer (typically the FIX engine) on the buy side would de-duplicate messages that arrive with PossDupFlag=Y as necessary.

Messages received over a FIX session should only be processed by the application layer in ascending message sequence number order. Duplicate messages arriving with PossDupFlag=Y will have the same MsgSeqNum as when they were received on previous occasions. Thus, if your session layer is doing its job correctly, it should only be passing the messages to the application layer in ascending sequence without duplicates.

After all is said and done, and your receiving process hasn’t been implemented in a layered session/application model (unfortunate IMHO), then ExecReport messages can still easily be de-duplicated because their ExecID (tag 17) fields are obligated to be unique.

I hope this helps.

JohnP

4

[ original email was from Sreedhar Tulluri - sreedhar_tulluri@ml.com ]
John,
In this particular case, the sequence number is not duplicated(not received on previous occasion). Messages came in with a unique seq number and 43=Y and were delivered to FIX Client by buy-side engine.
At this point, there is no way to know if its a duplicate message unless FIX client keeps track of MsgSeqNum’s for each session and look for duplicate MsgSeqNum.

How can we clearly distinguish between a possible duplicate and true duplicate by looking at the tags? (43 & 97 ).

Hi Sreedhar,

This is absolutely correct behavior.

I would hope that the FIX session layer (typically the FIX engine) on
the buy side would de-duplicate messages that arrive with PossDupFlag=Y
as necessary.

Messages received over a FIX session should only be processed by the
application layer in ascending message sequence number order. Duplicate
messages arriving with PossDupFlag=Y will have the same MsgSeqNum as
when they were received on previous occasions. Thus, if your session
layer is doing its job correctly, it should only be passing the messages
to the application layer in ascending sequence without duplicates.

After all is said and done, and your receiving process hasn’t been
implemented in a layered session/application model (unfortunate IMHO),
then ExecReport messages can still easily be de-duplicated because their
ExecID (tag 17) fields are obligated to be unique.

I hope this helps.

JohnP

5

[ original email was from John Prewett - jprewett@lavatrading.com ]
If the sequence number isn’t a duplicate, then PossDupFlag can be safely ignored as this message isn’t a duplicate at all.

PossResend is a totally different matter.
The FIX engine/session layer cannot de-dup messages with PossResend=Y simply because they do indeed have a different sequence number from the (possibly received) original transmission.
If you receive a message with PossResend=Y, the application layer is responsible for doing the de-dup by using whatever unique application field identifies the reason for the message flow.
As I said previously, if you receive an ExecReport message, it can be de-dup’d by using the ExecID tag which is obligated to be unique. Any ExecReport which is resent with a different sequence number and PossResend=Y must still contain the same ExecID as before.

JohnP