Imported from previous forum
The Global Technical Committee has reviewed and preliminarily approved the Throttling Proposal. The document now enters a public comment period in which public review and feedback is encouraged. Once the Public Comment period closes, the Global Technical Governance Board will meet to review public comments before final approval.
Please post feedback, comments, and questions as replies to this discussion thread.
A link to the proposal can be found at: http://www.fixprotocol.org/documents/4992/FIX%20Protocol%20Gap%20Analysis%20-%20Message%20Throttle%20v%200.10.doc
The public comment period ends on April 2, 2010.
Hi All,
From 3.1 of the document:
“At this point, session-level changes are outside the scope of this gap analysis.”
I’m just curious about something with respect to things like message throttling - isn’t that the whole point of having seperated the session layer from the application layer in FIX 5.0/TIF?
Why would any of this be implemented at the application layer in a FIX-based application? It just seems like this is so naturally suited to the session layer of the system - e.g. this is what a session layer is all about in the first place, why would this need to impact application layer messages?
I’m just curious in terms of the need for application layer message changes to support this kind of thing…
Thanks,
Russ
The public comment period ends on April 2, 2010.
[ original email was from Ryan Pierce (FPL Technical Director) - ryan.pierce@fixprotocol.org ]
> I’m just curious about something with respect to things like message
throttling - isn’t that the whole point of having seperated the session
layer from the application layer in FIX 5.0/TIF?
This is a very interesting point.
While the session/application split happened back in 2006, it is important to note that this does not dictate software architecture. Just because FPL says Message X is part of FIXT.1.1 and Message Y is part of FIX 5.0 does not mean that the function behind Message X must be implemented solely by FIX engines, and the function behind Message Y must be implemented solely by business applications. Further, the concept of FIX gateways, which sit between business applications and FIX engines, muddy the water. I think it important to distinguish between the component(s) implementing a concept (e.g. FIX engine or business application) and whether something is defined in the session or application layer.
Why would any of this be implemented at the application layer in a FIX-
based application? It just seems like this is so naturally suited to the
session layer of the system - e.g. this is what a session layer is all
about in the first place, why would this need to impact application
layer messages?
I would say throttling is an unusual use case that occupies an interesting position at the border between the two layers.
Throttling, clearly, can be a very low level concept. But there are nuances that affect higher layers. For example, the case of algo and retail order flow sharing a FIX session is mentioned in the Gap Analysis. A typical black box algo trading system considers any latency to be hazardous. If a throttle limit is exceeded, the last thing such a system would want is for the recipient to queue it (possibly until the next second) and wait before processing it. An immediate reject is far preferable. But a retail investor clicking a button on a web page probably doesn’t have the same latency requirement, and having the order rejected would be problematic. So the Gap Analysis identifies an application-level field (ThrottleInst) defined within specific application-level messages (e.g. New Order Single, Quote, etc.) that can be used to indicate the sender’s preference should a limit be exceeded.
Additionally, multiple firms have identified a requirement to identify throttle limits on a per-party (e.g. firm, trader, account, etc.) basis, which is a very high level business concept. This is not included in the current Gap Analysis because the proposed implementation involves using the Party Entitlements Report message, which itself is still in the Gap Analysis stage.
So I’d say the area of throttling would be a liminal space, or a grey area, between the functions of a FIX engine and a business application.
From a purely practical standpoint, it was implemented as part of the application layer, seeing as FPL can easily extend the application layer through the Extension Pack / Service Pack processes, while the session layer is far more difficult to modify. (Note that the “CME Enhancements for Trading System Identification” gap analysis, which was also posted for public comment, proposes that ultimately FPL should move in the direction of using Extension Packs for the session layer.)
It is also important to note that the method required in this Gap Analysis for rejecting messages in excess of the throttle limit is the Business Message Reject, while the current sole method to convey throttle limits between parties is the User Response message. Both of these are application level messages, but they are part of the “Infrastructure” group which is, arguably, the lowest level of the FIX application messages. For example, should a message be received that passes session level validation, but the business application that is supposed to process it is unavailable, the proper response is a Business Message Reject. This was done because the BMR is far simpler to generate than an Execution Report, and could, conceivably be generated by a FIX gateway or even a FIX engine. Also, the BMR is used as a generic reject for application level messages when application level rejects do not exist. Case in point: the IOI. If one party sends an IOI for an invalid symbol, the recipient’s business application may process it and may decide to send back a BMR. For these reasons, I consider the BMR in this same grey area; it definitely is an application level message, but it could conceivably be generated by a FIX engine, gateway, or business application.
Hi Ryan,
Thanks for taking the time to write such an illustrative post - I have a better understanding of the issues FPL is trying to address with this now, so that’s great.
Cheers,
Russ
We are working on an implementation for which the proposed new BusinessRejectReason (380) “Throttle limit exceeded” will be very useful.
On the assumption that this proposal is adopted, what is the likely value for this new BusinessRejectReason? Is 19 the likely value for “Throttle limit exceeded”?
The Global Technical Committee has reviewed and preliminarily approved
the Throttling Proposal. The document now enters a public comment period
in which public review and feedback is encouraged. Once the Public
Comment period closes, the Global Technical Governance Board will meet
to review public comments before final approval.Please post feedback, comments, and questions as replies to this
discussion thread.A link to the proposal can be found at://www.fixprotocol.org/documents/4992/FIX%20Protocol%20Gap%20Analysis%20-
%20Message%20Throttle%20v%200.10.docThe public comment period ends on April 2, 2010.
[ original email was from Ryan Pierce (FPL Technical Director) - ryan.pierce@fixprotocol.org ]
> We are working on an implementation for which the proposed new
BusinessRejectReason (380) “Throttle limit exceeded” will be very
useful.On the assumption that this proposal is adopted, what is the likely
value for this new BusinessRejectReason? Is 19 the likely value for
“Throttle limit exceeded”?
Please do not make any assumptions about what values FPL will assign.
This proposal is still in the public comment phase and will be until April 2. It has not been approved by GTC Governance Board vote. Due to public comments that may be received in the future, the proposal could be modified substantially before final approval.
Following the release of SP2, FPL has published several extension packs. We also have several more extension packs and proposals undergoing review in the pipeline. Trying to guess which enums a proposal will be assigned is extremely prone to error. I have seen several cases where firms assumed certain enums and began coding to them, only to discover that a prior EP took the enums in question.
Also, the enums for this field are very unusual. In SP2, they increment as 5, 6, 7, 18, leaving a large gap. We need to decide internally whether to continue assigning enums as 19, 20, 21… or fill in the gap with 8, 9, 10…
So please don’t assume what value will be assigned.
I would like to propose two minor additions in terms of valid values that should provide useful:
-
Additional ThrottleAction “4=Warning” to allow notification of the user before actual throttling takes place
-
Additional ThrottleType “2=Risk Limit” to allow throttling when exceeding risk limits
Regards,
Hanno.
I would like to propose two minor additions in terms of valid values
that should provide useful:
Additional ThrottleAction “4=Warning” to allow notification of the
user before actual throttling takes placeAdditional ThrottleType “2=Risk Limit” to allow throttling when
exceeding risk limitsRegards, Hanno.
Hi All,
Hanno’s suggestion, regarding congestion mitigation alerting/controls, brings another question to mind - something which I’ve been wondering a little about as I read the specification:
Seeing as this is really just a QoS implementation of sorts, with congestion relief (aka throttling) being just one aspect of QoS, is there an actual QoS standard that this proposal is modeled on? Considering how much work has been done on QoS and congestion relief/mitigation on existing network protocols, I was wondering if this proposal is based on one of those, or if it’s entirely ad-hoc?
If there are other QoS strategies and implementations that were drawn on in the development of this proposal, it might be nice to include references / citations to them in the proposal itself so that people can see where this proposal aligns with, or departs from, the existing bodies of work done in the QoS space.
[ original email was from Ryan Pierce (FPL Technical Director) - ryan.pierce@fixprotocol.org ]
> Seeing as this is really just a QoS implementation of sorts, with
congestion relief (aka throttling) being just one aspect of QoS, is
there an actual QoS standard that this proposal is modeled on?
I’m not sure I consider Throttling to be QoS. This proposal was modeled on requirements stated by exchanges participating in the Global Exchanges and Markets Committee meetings, not by any QoS standard.
I see QoS as much more dynamic. Throttling rules are largely static. An exchange tells its members at the start of day how many orders per second they can send, and what consequences they can expect if they exceed those limits. They are just as much business rules as a technical strategy to avoid congestion.
It is possible for these proposed Throttling messages to be sent intraday, but I haven’t heard much exchange interest in doing so. It might be useful in emergency situations caused by reduced exchange capacity or an increase in market activity. But I certainly don’t see an exchange sending one out every second to tell a member how many orders can be sent in that second.
The idea of using QoS protocols for order flow sounds very interesting. However, extensions to FIX generally result from business requirements presented to FPL. Are there any exchanges that use, or plan on using, a QoS protocol?
Hi Ryan,
Thanks for taking the time to reply.
I’m not sure I consider Throttling to be QoS.
By definition, at least from the old ITU standards, QoS refers to:
“A set of quality requirements on the collective behavior of one or more objects” (in a network).
Hanno’s question made me wonder if he was talking about a dynamic congestion relief mechanism (4=Warning) and if so, I was wondering which existing works on QoS might be serving as the underpinnings for the decisions being made in this proposal.
I guess one of the problems that I’ve always seen when reading a number of different FIX proposals, is that many of them seem to be devoid of references/citations to prior art in the industry, much of which can be very useful when creating new systems, since other people have already spent significant amounts of time solving the problem at hand - or at least one which is very similar…
This proposal was modeled
on requirements stated by exchanges participating in the Global
Exchanges and Markets Committee meetings, not by any QoS standard.
I guess my interest wasn’t in “which QoS standard does this implement” but rather “which QoS works were referenced or consulted in its design.” Not to say that an existing QoS standard should be implemented in toto, but rather to say that referencing existing works helps people understand that the decisions related to the design of something are grounded in successful implementations of the same thing from different parts of the industry.
That’s a good thing, since it allows those of us who look at something and try to implement it have more confidence that we’re not going to be looking at an errata in 3 months that says we have to make a bunch of changes because something doesn’t work right.
I’m not saying this will happen with this particular proposal, but I’m just pointing out that providing citations/references to existing bodies of work helps us implementers understand that all the bases have been covered and increases our degree of confidence that we can move forward with something.
I see QoS as much more dynamic. Throttling rules are largely static. An
exchange tells its members at the start of day how many orders per
second they can send, and what consequences they can expect if they
exceed those limits. They are just as much business rules as a technical
strategy to avoid congestion.
It is possible for these proposed Throttling messages to be sent
intraday, but I haven’t heard much exchange interest in doing so. It
might be useful in emergency situations caused by reduced exchange
capacity or an increase in market activity. But I certainly don’t see an
exchange sending one out every second to tell a member how many orders
can be sent in that second.
The idea of using QoS protocols for order flow sounds very interesting.
However, extensions to FIX generally result from business requirements
presented to FPL. Are there any exchanges that use, or plan on using, a
QoS protocol?
I guess this goes to my original concern - by definition, any exchange that implements aspects of this proposal is implementing an application level QoS protocol. If the only way to say this is not a QoS proposal is to redefine QoS, that just seems like a bad way to go.
Hanno’s question made me wonder if he was talking about a dynamic congestion relief mechanism (4=Warning)
No, I was not. “Warning” is meant for something other than high-speed throttles, for example, total number of Mass Quote messages allowed per day or Risk Limits being approached over time. I was not aiming at warnings when your inbound rate is limited to 1000 msg/sec and you reach 900. Our members want to know upfront what the max is and will immediately be sending at or around the max value. Warnings are of no value here, rejects are. Compare it to an OS warning when your hard disk fills up beyond 75%, not a warning whenever your drive access rate goes beyond 8ms.
“Congestion relief” sounds to me more like a traffic problem where you have a lot of cars driving around in a meshed network and you would like to optimize the overall flow so that none of the cars has to wait too long (or wait at all) at any crossing. The underlying assumption is that a car can take a number of routes to reach its destination. Our problem was much simpler and is rather a star network with a center node and single paths leading up to it. Each path gets a capacity assigned and can get predictable latency if it does not exceed this capacity.
[ original email was from Joseph Horowitz - jhorowitz@aegisoft.com ]
> Considering how much work has been done on QoS and congestion
relief/mitigation on existing network protocols, I was wondering if this
proposal is based on one of those, or if it’s entirely ad-hoc?
Speaking of network protocols, may I suggest a simpler scheme:
The Exchange-side sends a single new field in any message: this new field gives the number of additional orders and/or replaces that the exchange will accept at this time. Optionally, if the exchange-side needs to update this number, it can send an asynchronous (heartbeat or status) message with the new value.
Thanks!
-Joey
The proposal is based on specific requirements for throttles received from various parties that have implemented or need to implement it. The idea of explicit tokens being provided was discussed but discarded for high-speed environments where predictability of latency is key. Algos having to wait for explicit tokens before they can “continue” will not like it. They want to be in control and know upfront at what rate they can send requests to the marketplace without getting slowed down.
Considering how much work has been done on QoS and congestion
relief/mitigation on existing network protocols, I was wondering if
this proposal is based on one of those, or if it’s entirely ad-hoc?Speaking of network protocols, may I suggest a simpler scheme:
The Exchange-side sends a single new field in any message: this new
field gives the number of additional orders and/or replaces that the
exchange will accept at this time. Optionally, if the exchange-side
needs to update this number, it can send an asynchronous (heartbeat or
status) message with the new value.Thanks! -Joey
Having worked with less complex throttling rules as proposed here, I think Joey’s suggestion merit’s further thought. I suspect some may not appreciate the full scope of his suggestion, so I’ll elaborate as I understand it:
First, what Joey has suggested is a credit based scheme:
Let’s assume that we have a new Fix Tag 98765 “TAGOrderCredit”.
It would work like this:
On Fix Logon response, the bank adds the tag 98756=L where L is the max number of orders that the bank will accept from client at this moment.
Client is obligated to set local variable O = L.
When client has an order to send, it checks O. If O > 0, client sends order and decrements O. If before decrement O == 0, client either rejects order to trader, or queues order for later transmission
Bank is obligated to send execution report each time an order changes state. So, each execution report will now include the OrderCredit tag, and that tag’s value updates client’s current value of O.
When O is updated, client checks Queue and if O > 0 and client has queued order, sends it and decrements O.
The beauty of this is that it allows streaming of orders with no need to worry if client clock ticks at same rate as bank clock. If client is a bit too fast with clock tick, it might send an order ever so slightly sooner than the bank clock ticks, and then bank will reject the order. The time wasted to handle that reject adds considerable delay to time at which the order can be re-sent.
Moreover, it allows the bank to increase or decrease the throttle limits without need for clients to change code or parameters.
Finally, it keeps orders flowing at the maximum rate since executions are sent in parallel (full duplex) with new orders, so credit is updated on the fly. This is pipelining: keeping the order pipe filled with orders as much as possible.
I think that some will worry that they will see additional latency if they have to wait for a 0 credit state to be refreshed with an execution message. They are correct in that regard.
But, in practice, it will work that way for everybody, so no one should be at a disadvantage here, hence the mechanism is fair.
The proposal is based on specific requirements for throttles received
from various parties that have implemented or need to implement it. The
idea of explicit tokens being provided was discussed but discarded for
high-speed environments where predictability of latency is key. Algos
having to wait for explicit tokens before they can “continue” will not
like it. They want to be in control and know upfront at what rate they
can send requests to the marketplace without getting slowed down.Considering how much work has been done on QoS and congestion
relief/mitigation on existing network protocols, I was wondering if
this proposal is based on one of those, or if it’s entirely ad-hoc?Speaking of network protocols, may I suggest a simpler scheme:
The Exchange-side sends a single new field in any message: this new
field gives the number of additional orders and/or replaces that the
exchange will accept at this time. Optionally, if the exchange-side
needs to update this number, it can send an asynchronous (heartbeat or
status) message with the new value.Thanks! -Joey
[ original email was from Ryan Pierce (FPL Technical Director) - ryan.pierce@fixprotocol.org ]
> I would like to propose two minor additions in terms of valid values
that should provide useful:
- Additional ThrottleAction “4=Warning” to allow notification of the
user before actual throttling takes place
How would the warning take place? Out of band? An existing FIX message?
- Additional ThrottleType “2=Risk Limit” to allow throttling when
exceeding risk limits
I’m not sure I understand how this would work. I had thought risk limits were like a switch; if you exceed them, you can’t trade anymore.
Party Risk Limits Report, as defined in EP105, just states a limit, as well as optional textual warning level names and the percent of the limit at which they are issued.
In the case of integrating risk limits and throttling, I’m guessing you’d have a lower risk limit, at which point a customer is throttled, and a higher risk limit, at which point the customer can no longer trade. Or possibly there may be multiple levels of increasing risk limits, at which point the customer could be more aggressively throttled.
I also don’t see how ThrottleType = Risk Limit would work. It sounds like you’re saying, “If a risk limit is exceeded, then impose this throttle.” But the throttle to be imposed would itself need to indicate if it limited the inbound rate or the outstanding requests, which is what ThrottleType does. I’d like to consider extending Party Risk Limits Report to indicate that if a given risk limit is exceeded, then a given throttle will be turned on.
The warning can be issued in multiple ways and should be subject to bilateral agreement. It could be out-of-band, it could be a generic message such as UserNotification, it could be a more specific message informing about a utilization rate relative to the pre-defined limit. Warnings only make sense if you have time to react and avoid to exceed further limits that might lead to a slowdown, rejection or disconnect.
Plain vanilla risk limits are like a switch. This requires to be quite conservative, regardless of the market situation on that day. It is better to have multiple risk levels of which only the last one results in the inability to trade. Disconnect are not desirable in the context of risk limits as you want the member to be able to see and reduce his risk. You just don’t want him to be able to increase it.
The ThrottleType of RiskLimit says what is being used as a basis for the detection of a condition requiring throttling. It is not an inbound rate and not the number of outstanding requests being measured. The ThrottleAction (not the ThrottleType) defines the throttle to be imposed, e.g. queue inbound.
You are right that the Throttle component is something needed in the PartyRiskLimit messages to describe throttles per risk limit type. But that is outside of this proposal.
I would like to propose two minor additions in terms of valid values
that should provide useful:
- Additional ThrottleAction “4=Warning” to allow notification of the
user before actual throttling takes placeHow would the warning take place? Out of band? An existing FIX message?
- Additional ThrottleType “2=Risk Limit” to allow throttling when
exceeding risk limitsI’m not sure I understand how this would work. I had thought risk limits
were like a switch; if you exceed them, you can’t trade anymore.Party Risk Limits Report, as defined in EP105, just states a limit, as
well as optional textual warning level names and the percent of the
limit at which they are issued.In the case of integrating risk limits and throttling, I’m guessing
you’d have a lower risk limit, at which point a customer is throttled,
and a higher risk limit, at which point the customer can no longer
trade. Or possibly there may be multiple levels of increasing risk
limits, at which point the customer could be more aggressively
throttled.I also don’t see how ThrottleType = Risk Limit would work. It sounds
like you’re saying, “If a risk limit is exceeded, then impose this
throttle.” But the throttle to be imposed would itself need to indicate
if it limited the inbound rate or the outstanding requests, which is
what ThrottleType does. I’d like to consider extending Party Risk Limits
Report to indicate that if a given risk limit is exceeded, then a given
throttle will be turned on.
The public comment period ends on April 2, 2010.
Here are a couple of additional thoughts regarding this:
Considering the initial proposal, it seems that there should really be two component blocks introduced - one for session-establishment, and one for message-specific criteria, right?
The way the proposal reads right now, there’s one discreet component block which is used for session-establishment, but then it goes on to mention specific modifications to invidual messages. Wouldn’t it make more sense to introduce a second component block for the throttle criteria/hints being incorporated into those messages? According to the proposal, almost all of the message modification involve adding the same field (or fields) to individual messages, so would it make more sense to incorporate those fields into a separate component block and then apply that component block to those messages instead?
Thanks,
Russ
Not sure if I understand the exact issue(s) you are addressing.
The proposal defines a component block and proposes an extension of the User Response message to include this block. I do not see a reason why the block cannot be re-used as-is for other messages.
The message-specific criteria are part of the new component and they are optional. You can have more than one throttle and some could be specific to message types and others not.
Messages sent by an order submitter can only use ThrottleInst and would not justify a component. Messages received by the order submitter have three fields which could be put into a component. However, this would create an asymmetry between request and response. At this point I would rather wait, especially since components only affect a FIXML syntax.
The approach to put Throttle-Inst/Status/CountIndicator into the body and not the header has its drawbacks in general but I support the proposal as it stands now. We add them to the current messages but every new set of request/response messages should have them and one might forget to add them if the firm submitting the proposal does not need throttling.
The public comment period ends on April 2, 2010.
Here are a couple of additional thoughts regarding this:
Considering the initial proposal, it seems that there should really be
two component blocks introduced - one for session-establishment, and one
for message-specific criteria, right?The way the proposal reads right now, there’s one discreet component
block which is used for session-establishment, but then it goes on to
mention specific modifications to invidual messages. Wouldn’t it make
more sense to introduce a second component block for the throttle
criteria/hints being incorporated into those messages? According to the
proposal, almost all of the message modification involve adding the same
field (or fields) to individual messages, so would it make more sense to
incorporate those fields into a separate component block and then apply
that component block to those messages instead?Thanks,
Russ
Not sure if I understand the exact issue(s) you are addressing.
The proposal defines a component block and proposes an extension of the
User Response message to include this block. I do not see a reason why
the block cannot be re-used as-is for other messages.
Hi Hanno,
I think I’m just wondering why throttle-specific fields wouldn’t go in a throttle-specific component block - both in the repository and in the specification documents. Sure, if there’s never going to be more than one field added to a message to support throttling, then that doesn’t make much sense. However, what’s the level of certainty that this proposal nails throttling and we’ll never see another throttle-related field in a message? If it’s 100%, then that’s great - my question doesn’t make much sense at that point. But, if people start adding new throttle fields down the road, it might make sense to just use a reserved component block from the beginning, so that we have this:
(message name=“SomeAppMessage”)
(field name=“Field1” required=“N”)
(component name=“ThrottleParams” required=“N”)
(field name=“Field2” required=“Y”)
(/message)
instead of this:
(message name=“SomeAppMessage”)
(field name=“Field1” required=“N”)
(field name=“ThrottleInst” required=“N”)
(field name=“Field2” required=“Y”)
(field name=“Oh wait, here’s another throttle field” required=“N”)
(field name=“Field3” required=“N”)
(field name=“Oops, here’s another throttle field” required=“N”)
(/message)
I’m just thinking about this from the perspective of the specification documents and the repository layout, and don’t know what kind of impact a new component block would have on FIXML…
Thanks,
Russ