Imported from previous forum
[ original email was from Tim Yates - tyates@patsystems.com ]
I am trying to implement the servicing of market data requests. I am having one or two problems interpreting the FIX 4.2 specification.
Firstly, how are blank market data items indicated? For example, say a snapshot request asks only for the best bid price, but there are no bids in the market. Should the snapshot response attempt to specify a blank best bid price (how?), or should NoMDEntries be zero? The latter seems more sensible to me, but the FIX engine I am using crashes on receipt of a repeating group with no members. What is the correct behaviour?
I have also implemented ‘snapshot plus updates’ (incremental refreshes). I am unsure of the required behaviour when the market data request specifies a limited market depth (i.e. MarketDepth is not equal to 0).
For example, say there are two bids in the market and the request asked for a market depth of 2. Assume now that a new bid is added at the top of the market. This implies that the second best bid now becomes the third best, and therefore goes beyond the required depth. Should the incremental refresh: a) specify deletion of the previously 2nd best bid (and its associated MDEntryID), b) assume that the recipient implicitly deletes this bid or c) only notify the new best bid. If we assume c), then the recipient may have to keep track of many market data MDEntryID’s that correspond to bids that are outside of the specified market depth. What is the correct behaviour?
Finally, since it is quite expensive to persist the MDEntryID map, would it be too unreasonable to specify that all market data subscriptions should be assumed deactivated on session logout?
I have not seen any other correspondence in this discussion group about FIX Market Data support. Is this functionality rarely used?
Tim Yates.
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> Firstly, how are blank market data items indicated? For example, say a snapshot request asks only for the best bid price, but there are no bids in the market. Should the snapshot response attempt to specify a blank best bid price (how?), or should NoMDEntries be zero? The latter seems more sensible to me, but the FIX engine I am using crashes on receipt of a repeating group with no members. What is the correct behaviour?
First, a required repeating group with no data is not valid FIX; the engine shouldn’t accept it.
For a snapshot + updates, don’t send anything. Only if a bid later appears for that symbol do you send something.
I don’t believe the behavior in this scenario was ever explicitly defined for a snapshot without updates. I see a couple approaches to implementing it.
-
If the request was for the bid and offer of one stock, and there’s an offer but no bid, a Market Data - Snapshot / Full Refresh containing the symbol and just the offer will convey that there’s no bid.
-
If the request was for the bids of several symbols, you could respond with the bids for symbols that have them, and not respond with bids for symbols that don’t have them. The other party could then assume that there is no bid for that symbol at that time. This may or may not be acceptable.
-
You could send a bid with a price and volume of 0. Note that this really should be an option ONLY in the snapshot case without updates.
-
If the request is for the bid of one symbol that has no bid, or bids for several symbols, none of which have a bid, you could send a Market Data Request Reject with a Text field describing that no data exists.
[NOTE TO TECH COMMITTEE: Please track a proposed change to add "Requested data does not currently exist" to MDReqRejReason to make this case more explicit in the future.]
> I have also implemented ‘snapshot plus updates’ (incremental refreshes). I am unsure of the required behaviour when the market data request specifies a limited market depth (i.e. MarketDepth is not equal to 0).
>
> For example, say there are two bids in the market and the request asked for a market depth of 2. Assume now that a new bid is added at the top of the market. This implies that the second best bid now becomes the third best, and therefore goes beyond the required depth. Should the incremental refresh: a) specify deletion of the previously 2nd best bid (and its associated MDEntryID), b) assume that the recipient implicitly deletes this bid or c) only notify the new best bid. If we assume c), then the recipient may have to keep track of many market data MDEntryID’s that correspond to bids that are outside of the specified market depth. What is the correct behaviour?
First, not everyone must support every aspect of Market Data Request. You can certainly decide that you’ll only support limited values of this field, i.e. 0 or 1 (top of book), and not bother with handling truncated books (>=2). There’s even an MDReqRejReason to denote an unsupported MarketDepth field. Given the complexity of truncating the book, as I’ll illustrate below, it may be better to offer the full book or top of book only, and, if the client wants the top N tiers, make the client do the filtering on their side.
One way to support truncated books in the Incremental Update mode which eliminates most deletes is to pre-assign IDs for all slots and then simply change them. I.e. for a book with only the top two entries showing for MSFT, I’d create two IDs:
MSFT-BID-A (set to 50.30)
MSFT-BID-B (set to 50.20)
If a quote comes along that bids MSFT at 50.40, I’d change MSFT-BID-B to 50.40. If the 50.30 bid gets cancelled, that makes 50.20 the 2nd best, so I’d change MSFT-BID-A to 50.20. Now if the 50.20 bid gets cancelled and there’s only one price tier on the bid side for MSFT, I’d then delete MSFT-BID-A. If a new bid emerges again, I’d have to add MSFT-BID-A back again.
Now if one insists on correlating MDEntryIDs with, say, order IDs, then the method above likely won’t work, and changes to the top N tiers require adds and deletes. FIX 4.3 added a new field called MDImplicitDelete (547) which is used in this truncated book case.
In your example, if this field is N, then you must delete the 3rd best bid and add the new 1st best. If this field is Y, then you must add the 1st best bid, and it is the client’s responsibility to drop the 3rd best bid.
If the 1st best bid gets deleted, in either mode, you can’t assume the client will still remember the 3rd best bid and be able to make it the 2nd best. So that delete of the #1 slot must be accompanied by the addition of the #2 slot again, in either mode.
> Finally, since it is quite expensive to persist the MDEntryID map, would it be too unreasonable to specify that all market data subscriptions should be assumed deactivated on session logout?
This isn’t specified in the spec, but it certainly is a reasonable approach. If someone wanted to persist this across sessions, that would be reasonable, too.
> I have not seen any other correspondence in this discussion group about FIX Market Data support. Is this functionality rarely used?
I’m not sure about the level of support for FIX Market Data messages. It is a relatively recent addition to FIX, first appearing in 4.2.
I know that Archipelago uses it to disseminate its own book, as well as producing a combined feed that contains the books of REDI and ISLD. With ARCA’s implementation, Market Data Request is not supported. The client connects and immediately is fed a snapshot + updates of one or more of the available books, which is communicated offline.
I also know that NYSE was the other major contributor to the Market Data spec. I believe they are using these messages for their Institutional XPress feed, and doing so over the IP Multicast extensions to the FIX protocol. Like ARCA, they’re doing an “unsolicited” feed, i.e. not supporting Market Data Request.
I’d be interested in knowing who else has implemented the Market Data messages.
[ original email was from Tim Yates - tyates@patsystems.com ]
Thanks for the useful and detailed response.
I think my first issue (relating to empty snapshots) is a real problem. There are a few possible workarounds, but none of these is ideal. In the case when no data at all is available, one could reject the request. However, there is no obviously relevant MDReqRejReason. Also this workaround only applies to one special case; when a snapshot request relates to multiple instruments it won’t work.
The idea of sending nothing in response to a snapshot request also seems dubious. If the snapshot request specifies n instruments, the reply should surely consist of n snapshot messages. Otherwise, how can the originator know when the full reply has been received? What happens if the originating application needs to block waiting for the reply?
I can see that the problem lessens for an unsolicited feed, since, in that case if there is no data, you just don’t send a message at all for the relevant instrument. However, how do you blank all prices in this case? Or don’t you ever do this? Are there certain prices (e.g. settlement price) which are always available?
On further reflection, I don’t think the issue with incremental updates and restricted market depth is a problem after all. For example, say the market data request has specified a maximum depth of 10 and there are currently 10 bids in the market. 10 MDEntryIDs are therefore active for these bids. If a new best bid becomes available, I can merely send a single ‘Change’ update to change the price and (probably) the size of what was previously the 10th best bid to the price and size of the new best bid. It is not necessary to allocate MDEntryIDs by entry position; in the scenario described, this would result in the replace of all top 10 bid entries. This would somewhat defeat the object of incremental updates.
> > Firstly, how are blank market data items indicated? For example, say a snapshot request asks only for the best bid price, but there are no bids in the market. Should the snapshot response attempt to specify a blank best bid price (how?), or should NoMDEntries be zero? The latter seems more sensible to me, but the FIX engine I am using crashes on receipt of a repeating group with no members. What is the correct behaviour?
>
> First, a required repeating group with no data is not valid FIX; the engine shouldn’t accept it.
>
> For a snapshot + updates, don’t send anything. Only if a bid later appears for that symbol do you send something.
>
> I don’t believe the behavior in this scenario was ever explicitly defined for a snapshot without updates. I see a couple approaches to implementing it.
>
> 1. If the request was for the bid and offer of one stock, and there’s an offer but no bid, a Market Data - Snapshot / Full Refresh containing the symbol and just the offer will convey that there’s no bid.
>
> 2. If the request was for the bids of several symbols, you could respond with the bids for symbols that have them, and not respond with bids for symbols that don’t have them. The other party could then assume that there is no bid for that symbol at that time. This may or may not be acceptable.
>
> 3. You could send a bid with a price and volume of 0. Note that this really should be an option ONLY in the snapshot case without updates.
>
> 4. If the request is for the bid of one symbol that has no bid, or bids for several symbols, none of which have a bid, you could send a Market Data Request Reject with a Text field describing that no data exists.
>
> [NOTE TO TECH COMMITTEE: Please track a proposed change to add “Requested data does not currently exist” to MDReqRejReason to make this case more explicit in the future.]
>
> > I have also implemented ‘snapshot plus updates’ (incremental refreshes). I am unsure of the required behaviour when the market data request specifies a limited market depth (i.e. MarketDepth is not equal to 0).
> >
> > For example, say there are two bids in the market and the request asked for a market depth of 2. Assume now that a new bid is added at the top of the market. This implies that the second best bid now becomes the third best, and therefore goes beyond the required depth. Should the incremental refresh: a) specify deletion of the previously 2nd best bid (and its associated MDEntryID), b) assume that the recipient implicitly deletes this bid or c) only notify the new best bid. If we assume c), then the recipient may have to keep track of many market data MDEntryID’s that correspond to bids that are outside of the specified market depth. What is the correct behaviour?
>
> First, not everyone must support every aspect of Market Data Request. You can certainly decide that you’ll only support limited values of this field, i.e. 0 or 1 (top of book), and not bother with handling truncated books (>=2). There’s even an MDReqRejReason to denote an unsupported MarketDepth field. Given the complexity of truncating the book, as I’ll illustrate below, it may be better to offer the full book or top of book only, and, if the client wants the top N tiers, make the client do the filtering on their side.
>
> One way to support truncated books in the Incremental Update mode which eliminates most deletes is to pre-assign IDs for all slots and then simply change them. I.e. for a book with only the top two entries showing for MSFT, I’d create two IDs:
>
> MSFT-BID-A (set to 50.30)
> MSFT-BID-B (set to 50.20)
>
> If a quote comes along that bids MSFT at 50.40, I’d change MSFT-BID-B to 50.40. If the 50.30 bid gets cancelled, that makes 50.20 the 2nd best, so I’d change MSFT-BID-A to 50.20. Now if the 50.20 bid gets cancelled and there’s only one price tier on the bid side for MSFT, I’d then delete MSFT-BID-A. If a new bid emerges again, I’d have to add MSFT-BID-A back again.
>
> Now if one insists on correlating MDEntryIDs with, say, order IDs, then the method above likely won’t work, and changes to the top N tiers require adds and deletes. FIX 4.3 added a new field called MDImplicitDelete (547) which is used in this truncated book case.
>
> In your example, if this field is N, then you must delete the 3rd best bid and add the new 1st best. If this field is Y, then you must add the 1st best bid, and it is the client’s responsibility to drop the 3rd best bid.
>
> If the 1st best bid gets deleted, in either mode, you can’t assume the client will still remember the 3rd best bid and be able to make it the 2nd best. So that delete of the #1 slot must be accompanied by the addition of the #2 slot again, in either mode.
>
> > Finally, since it is quite expensive to persist the MDEntryID map, would it be too unreasonable to specify that all market data subscriptions should be assumed deactivated on session logout?
>
> This isn’t specified in the spec, but it certainly is a reasonable approach. If someone wanted to persist this across sessions, that would be reasonable, too.
>
> > I have not seen any other correspondence in this discussion group about FIX Market Data support. Is this functionality rarely used?
>
> I’m not sure about the level of support for FIX Market Data messages. It is a relatively recent addition to FIX, first appearing in 4.2.
>
> I know that Archipelago uses it to disseminate its own book, as well as producing a combined feed that contains the books of REDI and ISLD. With ARCA’s implementation, Market Data Request is not supported. The client connects and immediately is fed a snapshot + updates of one or more of the available books, which is communicated offline.
>
> I also know that NYSE was the other major contributor to the Market Data spec. I believe they are using these messages for their Institutional XPress feed, and doing so over the IP Multicast extensions to the FIX protocol. Like ARCA, they’re doing an “unsolicited” feed, i.e. not supporting Market Data Request.
>
> I’d be interested in knowing who else has implemented the Market Data messages.
>