Imported from previous forum
[ original email was from Eric Berg - eberg@rblt.com ]
We’ve determined that it may happen that NYSE will send execution reports out of order at times. This is due to the multithreaded nature of their internal systems leading to executions arriving at CMS, and subsequently being rerouted back to the sending parties out of order.
Has anyone else had problems with this? Is there a thread about this that I’m missing?
Thanks.
-Eric.
[ original email was from Joseph Horowitz - joey@aegisoft.com ]
Eric,
The following thread discusses this issue as it relates to FIX; “Multiple ‘indentical’ sessions & fault tolerance”:
http://www.fixprotocol.org/cgi-bin/BBS.cgi?menu=710&board=1&message=2396&thread=2395
-Joey
> We’ve determined that it may happen that NYSE will send execution reports out of order at times. This is due to the multithreaded nature of their internal systems leading to executions arriving at CMS, and subsequently being rerouted back to the sending parties out of order.
>
> Has anyone else had problems with this? Is there a thread about this that I’m missing?
>
> Thanks.
> -Eric.
>
[ original email was from Eric Berg - eberg@rblt.com ]
I’m not sure that it does address this issue, Joey. My primary
concern is that I’m receiving executions out of sequence from NYSE and
passing them on to my customers.
Some clients have processing problems when I pass on a fill, THEN a
partial fill against the same order.
I understand that there’s been some discussion of this NYSE-specific
issue on the list, but I am unable to find it.
-Eric.
>
> The following thread discusses this issue as it relates to FIX; “Multiple ‘indentical’ sessions & fault tolerance”:
>
> http://www.fixprotocol.org/cgi-bin/BBS.cgi?menu=710&board=1&message=2396&thread=2395
>
> -Joey
>
> > We’ve determined that it may happen that NYSE will send execution reports out of order at times. This is due to the multithreaded nature of their internal systems leading to executions arriving at CMS, and subsequently being rerouted back to the sending parties out of order.
> >
> > Has anyone else had problems with this? Is there a thread about this that I’m missing?
> >
> > Thanks.
> > -Eric.
> >
>
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> I’m not sure that it does address this issue, Joey. My primary
> concern is that I’m receiving executions out of sequence from NYSE and
> passing them on to my customers.
>
> Some clients have processing problems when I pass on a fill, THEN a
> partial fill against the same order.
In a CMS environment, even though a given CMS session does sequence its messages, I was under the impression that reordered messages were a way of life once one goes beyond more than one 9600 baud X.25 SVC. CMS messages generally carry enough state around so that firms can use some relatively intricate logic to reassemble messages at an application level if reordered.
FIX has very well-defined state transitions in “Appendix D.” The FIX session requires ordered message processing, so within a session, messages can’t be reordered. Since FIX was never tied to 9600 baud X.25 SVCs, and is generally routed over TCP/IP, firms generally will size one session to handle all of their order flow, or will partition their order flow among multiple sessions but expect a correlation between the session on which they sent the order and the session on which they send cancels and receive reports. As a result, many (most?) FIX systems don’t have advanced application-level reordering logic to compensate for these types of problems.
If what you’re saying is correct, it sounds like NYSE is not FIX compliant with respect to the message order stated in Appendix D. Assuming your customers can’t handle out-of-order messages, I can think of three solutions, ordered by what I believe to be most preferable first:
#1. Ask NYSE to be FIX compliant with respect to message ordering.
#2. Assuming NYSE properly populates CumQty and LeavesQty, you can build in custom code that checks it. This code will hold onto a NYSE FIX message if it anticipates that the message is out of order, and re-inserts it once it believes the time is right.
For a 1000 share order, if you believe only 400 shares have traded and you receive from NYSE a fill for 100 shares with CumQty=1000, LeavesQty=0, stash the message away and don’t tell the client. If you then see a partial fill for 200 with LeavesQty=100, CumQty=900, stash it away as well. Then when you get a partial fill for 300 with CumQty=700, LeavesQty=300, report it to the client, followed by the stashed partial fill for 200, followed by the stashed fill for 100.
This has a failure mode where a lost message causes further messages for that order to be suspended. To work around this, you may want to set some form of watchdog timer. If a message is stashed for more than X seconds, you could report it to the client regardless of sequencing, page an operator, or both.
Another area of concern would be cancels. If you receive OrdStatus=Cancelled but CumQty doesn’t match what you expect, this could indicate that a partial fill and a UR OUT have been reordered, in which case you may want to wait for the partial fill before reporting to the client that the order is dead.
#3 You could distribute the messages immediately in the order NYSE sends them, but recompute key state fields so that the resulting ordering makes sense. For instance, in the fill then partial fill case, you can adjust OrdStatus, CumQty, LeavesQty, and AvgPx so that the first "fill" you receive is sent to the client immediately as a partial fill, and the second "partial fill" is sent to the client immediately as a fill.
In the case of a cancel sent before a trade, FIX 4.2 defines a “partial decline”. For a 1000 share order with 400 traded, and you receive OrdStatus=Cancelled with CumQty=600, you know that a fill for 200 shares is likely missing. So you send ExecType=Restated with ExecRestatementReason=5 (Partial decline of OrderQty) and OrderQty=600, LeavesQty=200. This informs the client that 400 shares are dead and 200 shares are still live. As was said, this requires FIX 4.2, and I’d imagine that not all 4.2 implementations understand the concept of restatement for a partial decline.
I am thinking that the easiest way to solve the problem is to utilize Tag 9440 (ERCReferenceNumber), one issue with this is that it would require that the receiver of the execution reports "sit" on out of order messages before they are processed. I am thinking that this might be a non-issue since when an out of sequence message is received, it is likely the case that the next message to arrive is the message that will correct the order of these execution reports, and the reason the out-of-sequencing occurs is because of a synchronization race on multiple paths in CMS, and thus the total time to be sitting on this out of sequence message (which comes in not so frequently) would be at most X milliseconds, where X is the time for the message to be transported from CMS to sell-side engine.
This solution should require no state tracking except for an expected ERCReferenceNumber for each OrderID (37). When a message with an ERCReferenceNumber is received that is not what is expected, the engine could store it in a queue until the message(s) in the queue satisfy the requirement that the ERCReferenceNumber-containing messages will only be processed if its ERCReferenceNumber is that which is expected.
A hiccup is that certain messages coming back from CMS increment the ERCReferenceNumber, but don’t tell you that they do by setting ERCReferenceNumber in the message, PendingCancel messages are an example of this, so these messages need to update the expected ERCReferenceNumber table, even though they do not have one set in the message.
mohan
> Eric,
>
> The following thread discusses this issue as it relates to FIX; “Multiple ‘indentical’ sessions & fault tolerance”:
>
> http://www.fixprotocol.org/cgi-bin/BBS.cgi?menu=710&board=1&message=2396&thread=2395
>
> -Joey
[ original email was from Sam Matthews - sam@ucs.net ]
> We’ve determined that it may happen that NYSE will send execution reports out of order at times. This is due to the multithreaded nature of their internal systems leading to executions arriving at CMS, and subsequently being rerouted back to the sending parties out of order.
>
> Has anyone else had problems with this? Is there a thread about this that I’m missing?
>
> Thanks.
> -Eric.
>
We have. No real answer from SIAC about it. It basically forces us to ingnore their "Leaves" quantity. First you will get Leaves=0, and then Leaves=500, etc.
Sam Matthews
Global Brokerage Concepts