Imported from previous forum
There’s hardly any documentation describing the flow of messages received and sent or length of fields. This allows a great deal of freedom on the part of the implemention, which means that the standard will probably break down when talking between to different brokerages. For example, what is the response to an order message? Am I misinformed or what is the real state of FIXML at present? I’ve heard people are having problems due to the length fields not being defined and truncations happening on either end.
Just seems that a protocol should include the definition of whatever handshaking that needs to occur between communicating agents.
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> There’s hardly any documentation describing the flow of messages received and sent or length of fields. This allows a great deal of freedom on the part of the implemention, which means that the standard will probably break down when talking between to different brokerages. For example, what is the response to an order message? Am I misinformed or what is the real state of FIXML at present? I’ve heard people are having problems due to the length fields not being defined and truncations happening on either end.
>
> Just seems that a protocol should include the definition of whatever handshaking that needs to occur between communicating agents.
The data flow response for your example, order messages, is illustrated in quite a bit of detail in Appendix D. While I grant that the current FIX spec has limited high-level overviews, the presentations on the web site Scott mentioned do a good job of illustrating the business uses of FIX.
Your issue regarding field lengths is an interesting one, and I’d like to take this opportunity to share my opinions on the matter.
Protocols can take different philosophies in their level of specificity.
In general, lower-level protocols for interoperation between inexpensive plug-and-play components need to be very specific to ensure maximum interoperability. Examples include TCP/IP and HTTP. Nobody wants to worry whether their Microsoft TCP stack can talk with a Sun TCP stack and those of every other vendor.
Higher-level business protocols often have more of a choice in terms of specificity.
One approach is to be very strict. While this allows for maximum out-of-the-box interoperability, it also has the price tag of potentially forcing businesses to change their business models and systems to interoperate.
The other approach is to be very flexible. Firms generally don’t need to change their business practices, but bilateral agreement is needed on many things, and counterparty testing plays an important role.
Both of these approaches are valid. In an industry that already has rigorously defined standards, rigorous protocols are helpful. But doing so can limit that protocol’s entry into other industry sectors. FIX has positioned itself as a very dynamic protocol, and, in my view, benefits greatly from more flexibility.
FIX provides a session layer which tends towards the former approach (as low-level protocols benefit from rigorous definition), and a rigorous set of rules for message formatting. But the FIX application layer tends toward the latter approach. It provides a data dictionary and message catalog that is fairly flexible and can be extended by bilateral agreement, as well as state transitions for order flow and numerous apendices of examples.
This approach has proven highly successful in terms of market penetration of the FIX protocol, and I do not believe that if the spec had been made thoroughly rigid we would have this kind of success today.
For instance, maximum sequence numbers are a good example of how the FIX philosophy differs from other protocols. Older FIX protocol versions took the more rigid model and capped message sequence numbers at 99,999 messages. When this proved inadequate, this was later amended to 999,999 messages (FIX 4.0? Or is it 3.0?), and then the restriction was removed altogether (FIX 4.2).
Do you want to be the one to say to your traders, “Sorry, we’re using FIX, and limitations in the FIX protocol cap the maximum number of messages per session, and we’ve hit that, so we’ve got to close down shop and go home?”
Of course not! Trading volume is growing faster than Moore’s Law, so limits to maximum message sequence number that seem ridiculously large now (as the 3.0, 4.0 and 4.1 limits did when first written) can and will be hit.
Further, such artificial limits impede protocol expansion. FIX didn’t use to support Market Data. This changed in version 4.2, which, incidentally, was the first version to drop the message number cap. FIX could not have supported Market Data with a 99,999 message limit, or even a 999,999 message limit in the case of active trading systems.
Similarly, FIX used to have a 9999 byte message length limitation. Larger messages (i.e. Market Data, Mass Quote) and the encapsulation of XML in FIX would have been severely hampered by this, so the limitation was dropped.
This kind of flexibility allows FIX to expand readily into new business applications, industry sectors, and geographic regions.
Likewise, things like field length limits pose similar problems.
Assume that FIX declared that all OrderID and ClOrdID values were capped at X characters. Immediately, you have created a large conflict. Some factions will cry “But my native order IDs are longer than X characters! I’m not going to change my system! I’m not using this protocol!” If you increase X to accomodate these firms, others will cry “But X is too large! I can’t waste all this RAM and database space holding X characters per ClOrdID!” No matter what you do, you cannot assign an X where everyone is happy.
So there are two choices.
-
Declare an arbitrary value. A lot of people will be unhappy on both sides of the fence, and this can lead to poor adoption of the protocol.
-
Leave the length undefined and let counterparties decide amongst themselves what to implement. If one firm wants to send 150-character ClOrdIDs, they probably aren’t going to find that many people to trade with. If one firm cannot accept more than 6-character ClOrdIDs, they likely are not going to find many firms to trade with either. Hence firms can consider the trade-offs of how much data they need to encode in their ClOrdIDs, and how much they can afford to accept, and arrive at good decisions.
Further, flexibility allows the system to be future-proof. For instance, one particular market sector or geographic region might consider it standard practice to send very long ClOrdIDs. If FIX imposed an artificial cap, this would effectively hinder FIX from expanding into that industry sector or region. Without the cap, bilateral agreement allows this kind of business to be conducted.
In my opinion, FIX is a significant step forward from protocols with rigid fixed-length fields that cannot be extended. It has gained considerable market penetration. It has definitely done so at a cost of increased counterparty bilateral negotiation and testing. But even that is becoming less of an issue. Just as FIX engine vendors became a valuable niche market in the last few years and helped expand FIX’s industry penetration, FIX certification vendors are, in my opinion, already starting to take the next step by filling the role of helping FIX implementations play nicely together.
> So there are two choices.
>
> 1. Declare an arbitrary value. A lot of people will be unhappy on both sides of the fence, and this can lead to poor adoption of the protocol.
>
> 2. Leave the length undefined and let counterparties decide amongst themselves what to implement.
Fix has chosen the second approach. As a result, with each new counterparty we have to verify whether their Fix “standard” is compatible with our Fix “standard”. That’s not what commonly is understood as a standard.
The argument that a lot of people use Fix is not proving that Fix made the right choice. It just means that they did not have better alternatives. Most counterparties fill in the missing standards in a common sense way (like: an OrderID should be not too short and not too long, let’s take 20 characters). But it creates a lot of development overhead that could have been avoided when Fix would have filled in these common sense limits. In the latter case a company could implement the standard once (which indeed would perhaps require some adjustments to the companies system), but from that point on you could connect directly with any counterparty using the same standard.
A good thing that Fix has done for us is to design a more or less asynchronous interface with full recovery capabilities.
Paul