Hi everyone, I just wanted to start this thread to get a sense on how to properly handle a scenario where a counterparty tries to resend a Fill message to their customer via FIX after the original fill is DK’ed.
Scenario:
Buy side client sends order to their Broker via a routing hub.
While the order is being worked: The FIX session between the routing hub and the counterparty drops.
The order is still live and fills are being queued up by the counterparty so when the session between the broker and routing hub reconnects all missed fills were sent over the FIX line. However, at this point the routing hub failed to send several of these fills back to the buy side connection.
This caused a fill qty mismatch between client and broker because at some point the broker sent a complete fill on the order in question with OrdStatus = 2 | ExecType = 2, but from the client’s perspective since several fill messages were not received, the order status is still partially filled so the Buy side system sent a DK message of the full fill message back to the counterparty with DK reason in the text filed stating “The complete fill does not fully fill the order”. which was correct behavior.
The counterparty then tried to reconcile the missing fills by replaying all their executions after the close which allowed the buyside customer to process the missed fills during the outage EXCEPT for the fully filled message, thus leaving the state of the order in partially filled state with missing fill shares.
So my questions here are:
Since this was a resend of a fill with same ExecID should the fill message in this particular scenario been honored after the buy side DK’ed the original fill?
It appears to me that once a DK message of a fill is instituted that execution goes into a “terminal state”. Thus making any attempt to resend that fill should get discarded / ignored. Do you all think this statement is accurate?
Is there another action that the broker / routing hub could’ve taken in this scenario?
The routing hub failing to send fills back is a fairly egregious problem. It should be possible to reprocess these on the routing hub to ensure they are in fact sent to the client.
The right semi-automated solution is probably an OrderStatusRequest (35=H) from the client system, responded to by the broker with a 35=8 with the correct state of the order. Many client systems and brokers may not support this and usually this sort of problem is resolved out-of-band to be honest.
Even when they do support it, sending it is generally a manual action to reconcile the state.
Missing fills are a core issue and it’s why hubs have an utmost responsibility to be able to ensure they forward reports and can handle outages and bad reports.
DKs and rejected/dropped 35=8s should be red flags in any monitoring.
Fully agree with @philipwhiuk that this scenario is not acceptable from a buy-side perspective, i.e. that the routing hub sends new messages before having made sure that all previous messages were sent to the buy-side. The routing hub must recognise this situation and, if it is unable to resend the fills, terminate the connection to the buy-side to allow an out-of-band resolution.
The buy-side should also have detected the gaps in the messages from the routing hub, unless the routing hub simply dropped messages and presented a gapless sequence to the buy-side. Sounds like the routing hub does things in memory and is unable to recover properly from an outage.
Hi Hanno, the routing hub still hasn’t provided any updates on why they failed to send the proper messages. From the client’s point of view I know the trading system is configured to get visual alerts on open orders in the market where they haven’t been acted on in a long period of time. Do you think from a FIX perspective the DK message was applied correctly and if there should be some check in place to have allowed the original fill once the counterparty resent every execution?
@mparente2 the DK message should not have been sent by the application. That fill message should not have even made it to the application layer. Instead, the gap in the message sequence number should have been detected by the underlying FIX engine, followed by the attempt to recover the missing messages. A failure to recover them should eventually lead to the termination of the connection.
The above is based on the information you provided. It sounds hard to believe that a FIX engine would ignore gaps and continue to pass business level messages to the application for processing.
Hi Hanno, apologies if I didn’t state the scenario correctly:
In our case we have 2 sets of FIX sessions:
Session 1 is the buyside client connecting to the routing hub
Session 2: is the routing hub’s outbound connection to a sell side broker.
Here, the FIX session disconnect happened between the routing hub to outbound broker (session 2). The FIX session between the buy side engine and hub (session 1) was up and heartbeating and receiving fills from other counterparties who are connected to this hub.
Once the session between hub engine to outbound broker was re-established the broker sent all fills that were queued up in their engine to the routing hub. The break occurred when the RH failed to send several partial fills by the broker back to the hub to buy side connection.
The DK of the full fill message was sent from buyside to routing hub (session 1) and then RH to the Sell Side (session 2)
Understand, probably goes too far for this forum to analyse the architecture of the routing hub and how the two sessions are connected. Maybe both sessions write into an internal database and the hub has to pick messages up from there and forward them to the other side.
Fact seems to be that the architecture does not prevent an inconsistent state between buy-side and sell-side. That is not a FIX problem to synchronise sessions that are independent. The FIX StandardHeader lets you explicitly link buy-side to sell-side via a hub. Please see https://www.fixtrading.org/standards/fix-session-layer-online/#fix-message-routing.
BTW, the first partial fill that is out of sync should already be recognised by the buy-side because LeavesQty and CumQty do not match expectations, requiring manual intervention.
@mparente2 I worked on a FIX routing hub system for 7 years in the 1990s. From what you described there seems to be a problem with the hub’s architecture in how it handles recovery when itself failed (did I understand correctly there was another system fail here?) during processing of the recovered messages from the broker. This would require some serious internal analysis of the sequence of events to assess what is the bug in the hub itself.
If my understanding above is incorrect that there was no hub system fail when it processed the recovered sell-side msgs, then you should be able to try to reproduce this scenario in your QA environment to see if this can be replicated in order to figure out where the bug is. The hub should have processed the recovered (resent) mesgs from the sell-side in sequence and even after having stored it in its database be able to send them back out to the buy-side in sequence.
In terms of FIX state model, if I were the buy-side I wouldn’t have DK the full-fill msg (to me this is incorrect use of DK, if the ER has the correct the ClOrdID that the buy-side knows about then it shouldn’t have been DK’d). That full-fill state (even if they didn’t receive all the partials) is a higher precedence state and it tells me that the order was indeed fully filled even if i didn’t get the partials. For the buy-side to get the information of the missing partials they would likely have to do that out-of-band with the broker.