Imported from previous forum
[ original email was from Michael Whealy - michael.whealy@asx.com.au ]
Hi,
I have 2 questions regarding sending market data incremental refresh messages for an aggregated book.
Scenario 1) A client sends a market data request for both an aggregated book and a non-aggregated book, in two separate market data requests for the same symbol.
When a client receives a market data incremental refresh, how does the client know which book it is intended for? The field 346 (NumberOfOrders) could be used, however it is not mandatory.
Scenario 2) A client sends a market data request with a depth of 5. According to the 4.2 spec, if you send a new market data entry with an MDEntryPosition = 3, then this should shift all the entries at position 3 and below down one position (ie 3 to 4, 4 to 5). If you then delete a market data entry with MDEntryPosition = 3, then every entry below 3, should shift up one position (ie 4 to 3, 5 to 4).
My question is what happens to the entry that should now occupy the 5th position. The client cannot be expected to move it from position 6 to 5, because it has only requested a depth of 5. Should the server send a new market data entry for the entry at position 5? If so, has anyone implemented this yet?
Thanks.
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
Sorry I forgot about answering this.
> I have 2 questions regarding sending market data incremental refresh messages for an aggregated book.
>
> Scenario 1) A client sends a market data request for both an aggregated book and a non-aggregated book, in two separate market data requests for the same symbol.
>
> When a client receives a market data incremental refresh, how does the client know which book it is intended for? The field 346 (NumberOfOrders) could be used, however it is not mandatory.
The client has to send two Market Data Request messages. They each have two unique MDReqID tags. Responses will then be done with different Market Data - Incremental Refresh messages, and the MDReqID on those messages will indicate the request for which the data is intended.
> Scenario 2) A client sends a market data request with a depth of 5. According to the 4.2 spec, if you send a new market data entry with an MDEntryPosition = 3, then this should shift all the entries at position 3 and below down one position (ie 3 to 4, 4 to 5). If you then delete a market data entry with MDEntryPosition = 3, then every entry below 3, should shift up one position (ie 4 to 3, 5 to 4).
>
> My question is what happens to the entry that should now occupy the 5th position. The client cannot be expected to move it from position 6 to 5, because it has only requested a depth of 5. Should the server send a new market data entry for the entry at position 5? If so, has anyone implemented this yet?
When doing this, the server, upon inserting a new #3, will need to send a delete as well for the old #6.
When the server deletes #3, it will need to send a new #5 that it had previously deleted. The server might choose to reuse the MDEntryID it had assigned to #5. Note that two things must be sent for one business action.
Hence truncating the book might be a bad idea, as it may increase, not reduce, bandwidth and quote rate. While it shields the client from activity happening far from the BBO, it does so at the cost of effectively doubling the bandwidth required for quotes being inserted and deleted near the BBO.
> > Scenario 2) A client sends a market data request with a depth of 5. According to the 4.2 spec, if you send a new market data entry with an MDEntryPosition = 3, then this should shift all the entries at position 3 and below down one position (ie 3 to 4, 4 to 5
> When doing this, the server, upon inserting a new #3, will need to send a delete as well for the old #6.
Is this defined in the spec?
Sending the delete for the entry that drops off the bottom of the watchlist appears to be unnecessary. The client knows that it needs to be deleted because s/he asked for a limited-depth aggregated book. The client is prepared to undertake the necessary bookkeeping, and is interested in saving bandwidth, since s/he requested incremental updates (or the server said that was all they support 
And the processing needed for limited-depth is already different to that needed for full depth because of the need to send and receive ‘new’ entries for existing order levels re-entering the bottom of the specified depth.
So it seems to me that there is no downside in making the delete an implicit result of the insert, and there is a saving of bandwidth, which is always an important consideration with market data.
I couldn’t spot an explicit statement in the spec (admittedly only a quick browse), and I wonder how many systems have been implemented, and with what semantics. Would it cause a lot of work in many systems?
I suggest making an implicit delete an explicit part of 4.3
Cheers, Dave
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> > When doing this, the server, upon inserting a new #3, will need to send a delete as well for the old #6.
>
> Is this defined in the spec?
No, I don’t believe it is defined explicitly.
> Sending the delete for the entry that drops off the bottom of the watchlist appears to be unnecessary. The client knows that it needs to be deleted because s/he asked for a limited-depth aggregated book. The client is prepared to undertake the necessary bookkeeping, and is interested in saving bandwidth, since s/he requested incremental updates (or the server said that was all they support And the processing needed for limited-depth is already different to that needed for full depth because of the need to send and receive ‘new’ entries for existing order levels re-entering the bottom of the specified depth.
>
> So it seems to me that there is no downside in making the delete an implicit result of the insert, and there is a saving of bandwidth, which is always an important consideration with market data.
I do see potential downsides here.
In the explicit delete case, the client need not manage keeping track of deleting quotes as they fall off the bottom of the book. If the server sends deletes, the client can use the same functionality for full book as they do for the truncated book. The book display will just be limited. (And it may extend from 5 to 6 temporarily between the addition of #3 and the deletion of the old #5 which became #6.)
It assumes the client has an accurate view of the book and has not lost any data. Otherwise only the server, not the client, is capable of saying which is the bottom quote that needs to be deleted.
Further, it could be confusing if the quote is implicitly deleted, as the sequence of events, looking just at the message stream, are 1. New quote, 2. implicit delete, 3. New quote with same MDEntryID. The two New messages without a delete in the middle could be confusing, especially if one does not have the context of knowing the book (hence being able to judge what was implicitly deleted.)
> I couldn’t spot an explicit statement in the spec (admittedly only a quick browse), and I wonder how many systems have been implemented, and with what semantics. Would it cause a lot of work in many systems?
I don’t know either, hence I’d like to throw this out for discussion.
Further, I’d argue that even with implicit deletes, one still might be better off not truncating the book.
For instance, assume that 80% of all quote activity is in the top 5 tiers. Assume that 50% of all activity results in the creation or deletion of a new quote, as opposed to the addition or subtraction of volume on an existing quote.
Take 100,000 quote updates on a stock. Only 80,000 are of interest if we truncate the book. But of that 80,000, 40,000 are adding or dropping off entries to the bottom of the truncated book. If each of these resulted in an additional update (the explicit delete model), then we’d have 80,000 + 40,000 = 120,000 updates, or 20% more than if we did no filtering at all. If the deletes are implicit, but the updates explicit, then we have 80,000 + 20,000 = 100,000 updates, or no gain whatsoever.
So in a market where half the quoting activity adds or deletes a new price level (and the move to decimals or smaller trading units generally raises this number!), if more than 80% of the quoting activity occurs in those 5 tiers, then you will be increasing the quoting activity by truncating. Now I made up the 50% and 80%, but I’d imagine the 80% number is low. Most of the activity is going to occur at or near the BBO, not far away from it.
In the spirit of open debate:
> > So it seems to me that there is no downside in making the delete an
implicit result of the insert, and there is a saving of bandwidth, which is
always an important consideration with market data.
>
> I do see potential downsides here.
>
> In the explicit delete case, the client need not manage keeping track of
deleting quotes as they fall off the bottom of the book. If the server sends
deletes, the client can use the same functionality for full book as they do
for the truncated book.
I agree that it is possible, but I doubt it’s an issue for the client. I
think there are two scenarios; either the client is receiving the data
stream on behalf of a networked system (multiple displays, or a program
trading system perhaps), or it is directly attached to a single display
station. In the former case, it will be transforming the updates into
whatever protocol it uses internally, and its logic will already be
sophisticated, so the authors won’t much care about the slight extra
programming effort. In the latter case, the display probably has some set
number of slots for price display (since it asked for truncated, aggregated
book), so will have to manage the issue explicitly to avoid messing up its
display, and maybe it won’t even have logic for full book display.
> It assumes the client has an accurate view of the book and has not lost
any data. Otherwise only the server, not the client, is capable of saying
which is the bottom quote that needs to be deleted.
Pretty much all market data systems make this assumption. If you don’t
assume a stateful client, and a reliable protocol, you end up with wasting a
lot of bandwidth. FIX is reliable and offers the client the possibility of
requesting full refresh if they don’t want to take that responsibility.
Since the server can omit fields that haven’t changed (such as the identity
of the security :-), it is already trusting the client to keep accurate
state.
> Further, it could be confusing if the quote is implicitly deleted, as the
sequence of events, looking just at the message stream, are 1. New quote, 2.
implicit delete, 3. New quote with same MDEntryID. The two New messages
without a delete in the middle could be confusing, especially if one does
not have the context of knowing the book (hence being able to judge what was
implicitly deleted.)
As long as the protocol is properly documented, I don’t see that this would
be confusing. Complicated enough to warrant a few sentences and examples
explaining it, certainly.
How could the client not know the book? That’s why there is a snapshot at
the start of a request, in order to get the state on both sides into sync.
Again, all efficient market data distribution systems place this requirement
on clients.
If the client somehow loses track of the book, then it’s fault-recovery time
which is going to involve recovery of a known good state, either by
rerequesting the data from the server, or failover to a backup client.
> Further, I’d argue that even with implicit deletes, one still might be
better off not truncating the book.
>
> <snip of the details of the example>
I would agree with you that there are cases where it would be beneficial not
to truncate the book. But I suggest that there are also probably cases where
it is. For example, most of the traffic is due to the most heavily traded
instruments, and these are more likely to have at least one bid at each
price level close to the best. So with an aggregated book, I’d suggest most
changes in the top few levels are likely to be volume changes.
So I think the best way to handle this would be for the data source
organization to do some traffic analysis and make recommendations to clients
as to the most efficient way to handle their particular traffic. They may
well pre-empt the whole issue in any case, by specifying that they are only
prepared to transmit particular data streams. They will have to do this if
they plan to multicast their market data.
Cheers, Dave
PS I too would like to hear of any impact on existing systems.
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> > It assumes the client has an accurate view of the book and has not lost
> any data. Otherwise only the server, not the client, is capable of saying
> which is the bottom quote that needs to be deleted.
>
> Pretty much all market data systems make this assumption. If you don’t
> assume a stateful client, and a reliable protocol, you end up with wasting a
> lot of bandwidth. FIX is reliable and offers the client the possibility of
> requesting full refresh if they don’t want to take that responsibility.
> Since the server can omit fields that haven’t changed (such as the identity
> of the security :-), it is already trusting the client to keep accurate
> state.
This assumption of reliable transport isn’t necessarily true. A multicast FIX session has been defined, specifically for things like Market Data. Multicast UDP is not reliable. While in a normal FIX session, the client must halt all business processing of messages if it detects a gap and wait for the gap to be filled, that isn’t necessarily true for multicast FIX sessions. It works similar to the SIAC CTS/CQS and NASDAQ NQDS model, where one notices a gap but still keeps running and processing data, calls in the gap, and the gap gets retransmitted at a later time.
In a system with implicit deletes, once a gap is detected, the system is effectively helpless because it doesn’t have an accurate image of the book to make the delete decisions. With explicit deletes, the system can keep running (although there may be stuck or missing quotes) and the gap can be filled in at a later time to make the book entirely accurate.
> > Further, it could be confusing if the quote is implicitly deleted, as the
> sequence of events, looking just at the message stream, are 1. New quote, 2.
> implicit delete, 3. New quote with same MDEntryID. The two New messages
> without a delete in the middle could be confusing, especially if one does
> not have the context of knowing the book (hence being able to judge what was
> implicitly deleted.)
>
> As long as the protocol is properly documented, I don’t see that this would
> be confusing. Complicated enough to warrant a few sentences and examples
> explaining it, certainly.
My point here is largely one of troubleshooting. If everything is explicit, there can be no confusion of which quote should be deleted. If deletion is implicit, then it is possible that the server might want the client to delete one entry, and the client deletes another. With implicit deletes, one cannot tell what is happening from the data stream along.
I think that we need to allow for the option of explicit deletes for these reason. If parties want to use implicit deletes, I see no problem with this as well. I suppose we could handle it either through bilateral agreement, or we could add a field to the Market Data Request message specifying whether, for truncated books, explicit or implicit deletes should be used. Would this be satisfactory?
> I would agree with you that there are cases where it would be beneficial not
> to truncate the book. But I suggest that there are also probably cases where
> it is. For example, most of the traffic is due to the most heavily traded
> instruments, and these are more likely to have at least one bid at each
> price level close to the best. So with an aggregated book, I’d suggest most
> changes in the top few levels are likely to be volume changes.
>
> So I think the best way to handle this would be for the data source
> organization to do some traffic analysis and make recommendations to clients
> as to the most efficient way to handle their particular traffic. They may
> well pre-empt the whole issue in any case, by specifying that they are only
> prepared to transmit particular data streams. They will have to do this if
> they plan to multicast their market data.
>
> Cheers, Dave
>
> PS I too would like to hear of any impact on existing systems.
ARCA disseminates aggregated books in FIX for several ECNs. At present, all ECN books have orders at the same price level aggregated into one entry, but only some of the ECN books are truncated to 10 price levels. This all happens before the quote data gets translated to FIX.
The data feed is unsolicited in nature, i.e. the client is not requesting a truncated book, hence we cannot assume the client can do implicit deletes. All our deletes are explicit.
I will run some benchmarks with the ECN books, comparing side-by-side the FIX data rate for truncated vs. non-truncated.
Note also that when I speak of truncated, I’m referring to books showing depth, i.e. at least 2 price levels. Aggregated books truncated to just 1 price level (top of book) do not suffer from these problems, as one can use one MDEntryID for the best bid, one MDEntryID for the best offer, and just keep sending updates. Nothing has to fall off the bottom of the book, hence extra deletes or adds are not needed.
> This assumption of reliable transport isn’t necessarily true. A multicast FIX session has been defined, specifically for things like Market Data. Multicast UDP is not reliable. While in a normal FIX session, the client must halt all business processing of messages if it detects a gap and wait for the gap to be filled, that isn’t necessarily true for multicast FIX sessions. It works similar to the SIAC CTS/CQS and NASDAQ NQDS model, where one notices a gap but still keeps running and processing data, calls in the gap, and the gap gets retransmitted at a later time.
All I can offer on this is my personal opinion and experience, which is that (1) the goal when designing MDD protocols must be to have immediate automatic error-correction and (2) the best place to start to build this in is at the bottom. So when using multicast, I’m a strong believer in putting the recovery in immediately above the UDP packet layer. Either by using one of the available reliable multicast products or by building directly on UDP if the requirements are sufficiently specialised.
> In a system with implicit deletes, once a gap is detected, the system is effectively helpless because it doesn’t have an accurate image of the book to make the delete decisions. With explicit deletes, the system can keep running (although there may be stuck or missing quotes) and the gap can be filled in at a later time to make the book entirely accurate.
I’m not sure I understand this. So let me set my stall out, and you can point out any rotten fruit.
Isn’t the explicit delete always going to be for level 6 (in a 5 level book), so if you miss the insert the client is going to discard the delete anyway because it applies to an element that doesn’t exist in its image (or do an implicit delete on #5
Also, as soon as you miss a message you’re dead (or very dubious). It could have been an insert for best, so you’re potentially off market.
And the insert and delete are quite likely to have been packaged together in the same multicast packet, using the normal packing algorithm, so you’re most likely to lose them both together.
> My point here is largely one of troubleshooting. If everything is explicit, there can be no confusion of which quote should be deleted.
I entirely agree that the explicit case is easier to debug. It’s just a question of whether the economic benefit of reduced bandwidth outweighs the economic cost of increased testing. Historically, telecoms costs here in Europe and also in Asia have been strong enough to tip that balance. And as ever, good documentation will reduce testing costs.
> I think that we need to allow for the option of explicit deletes for these reason. If parties want to use implicit deletes, I see no problem with this as well. I suppose we could handle it either through bilateral agreement, or we could add a field to the Market Data Request message specifying whether, for truncated books, explicit or implicit deletes should be used. Would this be satisfactory?
I personally don’t have a problem with that. As devil’s advocate I’d point out that it increases the complexity and therefore the cost of implementation. (But, hey, if it was easy we wouldn’t get paid
Actually, the reason I don’t have a problem is that as long as the implicit deletes are allowed, a client will deal correctly with explicit deletes, and the cost on the server side is negligible.
> > PS I too would like to hear of any impact on existing systems.
>
> ARCA disseminates aggregated books in FIX <snip>
Thanks for the explanation.
> i.e. the client is not requesting a truncated book, hence we cannot assume the client can do implicit deletes.
And I understand why you need to do explicit deletes in this case. I expect it would remain unchanged even if the ground rules were changed.
> Note also that when I speak of truncated, I’m referring to books showing depth, i.e. at least 2 price levels. Aggregated books truncated to just 1 price level (top of book) do not suffer from these problems
Agreed.
Cheers, Dave
[ original email was from Michael Whealy - michael.whealy@asx.com.au ]
> > Scenario 1) A client sends a market data request for both an aggregated book and a non-aggregated book, in two separate market data requests for the same symbol.
> >
> > When a client receives a market data incremental refresh, how does the client know which book it is intended for? The field 346 (NumberOfOrders) could be used, however it is not mandatory.
>
> The client has to send two Market Data Request messages. They each have two unique MDReqID tags. Responses will then be done with different Market Data - Incremental Refresh messages, and the MDReqID on those messages will indicate the request for which the data is intended.
>
MDReqID is not a mandatory field (except for this first response to a snapshot + updates response). So can I assume that MDReqID will be populated? Are you saying that MDReqID is conditionally required if a client sends two or more MarketDataRequests for the same symbol?
> MDReqID is not a mandatory field (except for this first response to a snapshot + updates response). So can I assume that MDReqID will be populated? Are you saying that MDReqID is conditionally required if a client sends two or more MarketDataRequests for the same symbol?
Which version of the spec are you looking at? In the "4.2 with errata 2001-05-01" version, MDReqID is mandatory in the Market Data Request, and conditionally required in both flavours of the Market Data message when responding to a request (unsolicited market data does not have an MDReqID).
Cheers, Dave