Imported from previous forum
All,
following up on today’s call;
An important point that was raised by Jim was the fact that FAST is already perceived as “too complex” and the added complexity that follows from the 1.2 extension proposals threaten to fragment FAST implementations and users.
My view is that FAST is simple, but as they say:
“Beauty is in the eye of the beholder” …
All joking aside, I’d like to get your help.
In general;
- What do we need to do to assist in understanding FAST?
- Do we need a 1.1/1.2 impl that demonstrates the full feature set?
About the 1.2 proposal;
- Should we skip one or more of the proposed extensions?
- Should we modify one or more of them?
- Should we add some other extension?
Thx,
Rolf
Simple:
-
People care about speed, simplicity and correctmess, and FAST is already slowing, complicating and messing up things. This is all a design flaw.
-
Demonstrating a full feature set goes completely against what every major exchange or ECN did. No one cares about it and does their own thing TM.
-
People lie in benchmarks.
-
You need to develop something that doesn’t rely on bits, simple as that. No hardware architecture in the world was designed for it, despite appereances, apart from maybe some ancient telephone systems. Someone should have foreseen that.
-
You should skip everything, and see the existing mess with major proponents and adopters (it is already mega huge).
-
Fixing FIX via FASTFix has a full-fidelity nail and hammer analogy.
-
The world economy, software included, does not need another unnecessary complication.
Regards,
Troll
Please be specific or be silent
/Rolf
Simple:
People care about speed, simplicity and correctmess, and FAST is
already slowing, complicating and messing up things. This is all a
design flaw.Demonstrating a full feature set goes completely against what
every major exchange or ECN did. No one cares about it and does
their own thing TM.People lie in benchmarks.
You need to develop something that doesn’t rely on bits, simple as
that. No hardware architecture in the world was designed for it,
despite appereances, apart from maybe some ancient telephone systems.
Someone should have foreseen that.You should skip everything, and see the existing mess with major
proponents and adopters (it is already mega huge).Fixing FIX via FASTFix has a full-fidelity nail and hammer analogy.
The world economy, software included, does not need another
unnecessary complication.Regards, Troll
You need me to be specific after all those answers below point to the same conclusion?
Okay, fine, provide a reference implementation that decodes sad 150,000 messages on a 3GHz core. Tangible one if possible.
For the record, that is a pathetic rate by any standards of late 1990s, and two orders of magnitude improvement is perfectly normal in this day and age.
Once again, bits are the wrong abstraction in all hardware architectures for the last 10 years. Moreover, equivalent bandwidth saving is achievable with less complexity and latency, try it out yourself (it is not my job to educate).
FAST design is going against that, period.
Point two, talk to the exchanges or vendors who hack according to target specifics rather than FAST protocol definition.
Taking the figures and the facts above, please do not be silent on the issue. Speaking up is the only way to move forward, so:
Why is this the case? And, why complicate with workaround on top of workaround?
You need me to be specific after all those answers below point to the
same conclusion?Okay, fine, provide a reference implementation that decodes sad 150,000
messages on a 3GHz core. Tangible one if possible.For the record, that is a pathetic rate by any standards of late
1990s, and two orders of magnitude improvement is perfectly normal in
this day and age.Once again, bits are the wrong abstraction in all hardware architectures
for the last 10 years. Moreover, equivalent bandwidth saving is
achievable with less complexity and latency, try it out yourself (it is
not my job to educate).FAST design is going against that, period.
I get the impression that you think a FAST software implementation cannot be be processed at a high rate on contemporary hardware. Is that correct?
The perception of complexity often comes from a lack of knowledge. Separate from the spec, a good set of usage guidelines and examples would help to understand the concepts. Some of the existing implementations have such examples in their docs which could be pulled together.
Guidelines need to cover the encoding aspects (e.g. how does a FAST message actually look like?) as well as best practices for template design (e.g. how do I decide which operator to use?).
FAST should become a commodity service for those that have to deal with the application level. The role of the database application programmer is different from a physical database designer who has to worry about primary keys, partitions etc. People that deal with the low level, more technical stuff will not mind the complexity as it is the only way to increase speed. Maybe the problem is simply that the wrong skill profile is trying to tackle FAST. Designing a FIX message and designing a FAST template for it are quite different things. The FAST guy does not really need to have any idea of what the underlying business functionality is. He needs information about the repetitive nature of the data, i.e. how does the data change from one message to the next and how often does which piece of data occur.
My view about extensions is that we should not avoid a good extension just because we cannot ensure that everybody implements it. People will implement what they want/need to use. Vendor solutions should be complete similar to a FIX engine. I am more worried about proprietary extensions that are not part of the standard. People will want to achieve further optimization and they need a way to do that within the standard. A new version 1.2 capturing those extensions is the right way forward I believe.
Regards,
Hanno.
All,
following up on today’s call;
An important point that was raised by Jim was the fact that FAST is
already perceived as “too complex” and the added complexity that follows
from the 1.2 extension proposals threaten to fragment FAST
implementations and users.My view is that FAST is simple, but as they say: “Beauty is in the eye
of the beholder” …All joking aside, I’d like to get your help.
In general;
- What do we need to do to assist in understanding FAST?
- Do we need a 1.1/1.2 impl that demonstrates the full feature set?
About the 1.2 proposal;
- Should we skip one or more of the proposed extensions?
- Should we modify one or more of them?
- Should we add some other extension?
Thx, Rolf
I agree with you Hanno in terms of complexity is a perception issue.
I also agree that we should not avoid a “good” feature. However, “good” is not a sufficient adjective. I am reminded of the previous HSBC advertising campaign where the same object is viewed both negatively and positively - depending upon observer.
I would prefer that we instead proactively attempt some form of decision making process where we include implementation complexity, backward compatibility, and the temporal and spacial quantitative benefits of an extension as part of the decision making process and carefully document and then test before standardizing.
For instance, is there some marginal value for putting field data within the PMAP? There may be - but I would argue it is a marginal benefit when compared to the costs to existing implementations and to breaking the model of “presence” map. From a conceptual perspective, from a processing perspective - including field data in the PMAP is an unnecessary complication that needs to be analyzed against quantitative bbenefits. At this point in the life of the FAST Protocol - I think FPL should underwrite engineering exercises and testing before we break the existing protocol and risk splintering FAST into multiple dialects. I don’t think the resulting fragmentation of the standard is worth this feature.
The other area in terms of complexity is the scaled decimal numbers and the issue of one or two bits in the presence map - but I think this complexity has already been absorbed by the industry - trying to address this probably is not justified from a business case perspective.
Other than that I believe you have identified a critical aspect of what FPL should be doing with member’s funding - I think one of the key things FPL should be doing is improving the quality of education material and implementation guides to the industry. This was a major driver behind us deciding to bring on board additional full time technical resources.
I agree proprietary extensions or improper implementations are more a threat than a new standardized feature (within reason of course).
The perception of complexity often comes from a lack of knowledge.
Separate from the spec, a good set of usage guidelines and examples
would help to understand the concepts. Some of the existing
implementations have such examples in their docs which could be
pulled together.Guidelines need to cover the encoding aspects (e.g. how does a FAST
message actually look like?) as well as best practices for template
design (e.g. how do I decide which operator to use?).FAST should become a commodity service for those that have to deal with
the application level. The role of the database application programmer
is different from a physical database designer who has to worry about
primary keys, partitions etc. People that deal with the low level, more
technical stuff will not mind the complexity as it is the only way to
increase speed. Maybe the problem is simply that the wrong skill profile
is trying to tackle FAST. Designing a FIX message and designing a FAST
template for it are quite different things. The FAST guy does not really
need to have any idea of what the underlying business functionality is.
He needs information about the repetitive nature of the data, i.e. how
does the data change from one message to the next and how often does
which piece of data occur.My view about extensions is that we should not avoid a good extension
just because we cannot ensure that everybody implements it. People will
implement what they want/need to use. Vendor solutions should be
complete similar to a FIX engine. I am more worried about proprietary
extensions that are not part of the standard. People will want to
achieve further optimization and they need a way to do that within the
standard. A new version 1.2 capturing those extensions is the right way
forward I believe.Regards, Hanno.
All,
following up on today’s call;
An important point that was raised by Jim was the fact that FAST is
already perceived as “too complex” and the added complexity that
follows from the 1.2 extension proposals threaten to fragment FAST
implementations and users.My view is that FAST is simple, but as they say: “Beauty is in the eye
of the beholder” …All joking aside, I’d like to get your help.
In general;
- What do we need to do to assist in understanding FAST?
- Do we need a 1.1/1.2 impl that demonstrates the full feature set?
About the 1.2 proposal;
- Should we skip one or more of the proposed extensions?
- Should we modify one or more of them?
- Should we add some other extension?
Thx, Rolf
[ original email was from Rich Shriver - shriver@jandj.com ]
I think we should remember that these extensions were primarily identified as a result of discussions that came from the work we have been doing for FIX over FAST and from other implementers of FAST that have asked for specific incremental improvements. There are some features which I beleive are very important to the efficiency of FIX over FAST, Perhaps this has been lost to some extent as we focus in on the concise details.
In one specific instance, a FAST implementer asked for a solution where a pmap bit could be used to represent one of two possible values for a field instead of burning up a pmap slot and a field. Thus the onset of mixing data within the metadata in the pmpap begins. I don’t believe this is a new concept and the logical extension is to provide a greater capability than inlining a single pmap bit for data.
Compared to ASCII FIX, TLV message formats FAST is complex. My view is that FAST was designed to solve a problem of bandwidth problem and it does that extremely well by using every trick in the book (for want of a better phrase). However this does make it a lot more complex than non-compressed self describing message structures.
Ideally FAST should become a commodity and one and an official fully implemented reference implementation would go some way to achieving that. People will always be able to differentiate their implementations on performance.
The proposed extensions look good to me, and plug some of the gaps to making FAST a generic message format.
Glenn
The perception of complexity often comes from a lack of knowledge.
Separate from the spec, a good set of usage guidelines and examples
would help to understand the concepts. Some of the existing
implementations have such examples in their docs which could be
pulled together.Guidelines need to cover the encoding aspects (e.g. how does a FAST
message actually look like?) as well as best practices for template
design (e.g. how do I decide which operator to use?).FAST should become a commodity service for those that have to deal with
the application level. The role of the database application programmer
is different from a physical database designer who has to worry about
primary keys, partitions etc. People that deal with the low level, more
technical stuff will not mind the complexity as it is the only way to
increase speed. Maybe the problem is simply that the wrong skill profile
is trying to tackle FAST. Designing a FIX message and designing a FAST
template for it are quite different things. The FAST guy does not really
need to have any idea of what the underlying business functionality is.
He needs information about the repetitive nature of the data, i.e. how
does the data change from one message to the next and how often does
which piece of data occur.My view about extensions is that we should not avoid a good extension
just because we cannot ensure that everybody implements it. People will
implement what they want/need to use. Vendor solutions should be
complete similar to a FIX engine. I am more worried about proprietary
extensions that are not part of the standard. People will want to
achieve further optimization and they need a way to do that within the
standard. A new version 1.2 capturing those extensions is the right way
forward I believe.Regards, Hanno.
Complexity isn’t in itself a definitive term. It’s arguably easier for a human to get a grasp of a tag/value rendition of a message than by looking at a hexdump of the same message encoded as FAST. FIXML would from that perspective look even less complex since you wouldn’t need to know the tag numbers by heart (though some would certainly be blinded by all the pointy brackets and verbosity :-).
However, if you’re about to implement an encoder or a decoder, a self describing format will probably give you a jump start. Being able to “see” the encoding helps, but in the longer run I’d say that there are other aspects that matter more, like for example consistency, predictability and clarity.
Complexity is also relative. Creating a naïve processor for either FAST or FIX tag/value isn’t a particularly great challenge. You’ll need to get more friendly with your hexdumper in the FAST case, but you’ll catch up when you get to the details.
On the other hand, creating a high-performing, robust and flexible processor is a complex task for all of these encodings. From my experience, most of the development efforts in such cases won’t be devoted to understanding the encodings per se, but to figure out how to waste the least number of cycles in the CPU while still maintaining an intuitive and flexible interface to the application. And this part gets easier the less wiggle room the encoding presents.
/David
Compared to ASCII FIX, TLV message formats FAST is complex. My view is
that FAST was designed to solve a problem of bandwidth problem and it
does that extremely well by using every trick in the book (for want of a
better phrase). However this does make it a lot more complex than non-
compressed self describing message structures.
Glenn,
thank you for your comments. I’d be very interested to hear if you have any suggestions on other specific extensions plug more of the gaps to a generic message format.
Best,
Rolf
Compared to ASCII FIX, TLV message formats FAST is complex.
My view is that FAST was designed to solve a problem of bandwidth
problem and it does that extremely well by using every trick in
the book (for want of a better phrase). However this does make it
a lot more complex than non-compressed self describing message
structures.Ideally FAST should become a commodity and one and an official
fully implemented reference implementation would go some way to
achieving that. People will always be able to differentiate their
implementations on performance.The proposed extensions look good to me, and plug some of the gaps
to making FAST a generic message format.Glenn
[ original email was from Kevin Houstoun - kevinh@altkb.com ]
My personal take on this is that it is not complexity but accessibility. I think FAST is simple but becoming harder to understand. I think for people, like me, without formal IT training, my academic background is engineering, some aspects of the documentation are initially hard to follow.
That said, other coders I work with, who have a formal IT background find the documentation, unambiguous and easy to follow. Perhaps an approach around this is to clearly state that examples are exactly that and that if they introduce ambiguity then the formal part of the documentation takes precedence.
I think we should seek to address this by providing documentation, more sample code etc, rather than by compromising the protocol.
Just a thought
Cheers
Kevin
All,
following up on today’s call;
An important point that was raised by Jim was the fact that FAST is
already perceived as “too complex” and the added complexity that follows
from the 1.2 extension proposals threaten to fragment FAST
implementations and users.My view is that FAST is simple, but as they say: “Beauty is in the eye
of the beholder” …All joking aside, I’d like to get your help.
In general;
- What do we need to do to assist in understanding FAST?
- Do we need a 1.1/1.2 impl that demonstrates the full feature set?
About the 1.2 proposal;
- Should we skip one or more of the proposed extensions?
- Should we modify one or more of them?
- Should we add some other extension?
Thx, Rolf
I guess maybe I can provide a (biased?) end user perspective. FAST is complicated relative to what it is replacing. Take two examples, CME FIX/FAST and ARCA. legacy formats, RLC for CME, ITCH for ARCA, are very simple, RLC is so simple that a jr programmer can have a half decent implementation in a week. Take same jr programmer, give him/her FIX/FAST specs, they are likely have no clue where to begin. So IMHO, it is a skillset and resource commitment issue that is causing the perceived FAST “complexity”.
In effect, by adopting FAST, the barrier to have a system up and running, has been raised, and the non-early-adopter User Firms, are “shocked” that their existing Tech staff does not have the skillset and resources (!) to upgrade their system to FAST. Combined with economic climate, i am certain IT’s request for resources are met with skepticism, with IT budget unlikely to grow in the next 12-18 months. obviously i am biased from my perspective, head of it for a broker, and now running a sys trading firm.
Therefore if anything the WG can do, is to make more accessible examples, more detailed “tutorial”, so that the proverbial “jr programmer”, can get over some “initial understanding hump” of FAST, rather than just being completely lost.
All,
following up on today’s call;
An important point that was raised by Jim was the fact that FAST is
already perceived as “too complex” and the added complexity that follows
I couldn’t agree more!
/David
Therefore if anything the WG can do, is to make more accessible
examples, more detailed “tutorial”, so that the proverbial “jr
programmer”, can get over some “initial understanding hump” of FAST,
rather than just being completely lost.
I think the complexity could be diminished if an Exchange encoding FAST data provided a standard decoder in the various popular implementation languages. When SpryWare did the initial ARCA FAST implementation, we provided a ‘C’ decoder that was free to use by ARCA customers. It gave a baseline to compare decoded data to the original data so if you did decide to do your own decoder implementation, you could at least “diff” your decoder output with the official Exchange provided decoder.
Just as not many people pick up and write their own zlib or jpeg decompressor, FAST needs to get to a point where there are good, reliable, high performance decoders available for free. Perhaps the business model is to sell encoders and give decoders away for free ? Adobe seems to be successful with this model, giving away the reader for free.
In regards to production FAST based feeds, I agree that RLC was a much easier feed to handle, and the CME FAST implementation is very, very complicated.
Here are my observations:
-
When ARCA and Opra went to FAST, it was a FAST only implementation. No FIX was involved. In the simplest sense you could take the supplied decoder and simply decode the FAST message back to the original, proprietary ARCA Book Binary message or Opra ASCII message. In some cases it was simply inserting a line of code to do the decoding, much like unzipping a zlib based data stream. Since both exchanges ran the old and the new feeds in parallel, you could compare the outputs and validate the data.
-
Once you had that up and running, most realized the conversion to the original ACSII or binary format was an unnecessary step, and moved to process the FAST message directly, but this was an option, not mandatory.
-
In both the Opra and ARCA implementations, the basic business logic involved in processing the feed did not change at all, it was purely for bandwidth (and CPU) reduction.
-
The CME decided to kill two birds with one stone, implement a replacement to RLC with a FIX based market data feed, and use FAST to encode/decode that feed. So the wire format changed, and the logical format, along with a new set of business rules to learn. Throw into that a mix of dozens of multicast groups, a separate “Instrument Definition” channel, a replay channel that needs to be synchronized with the live channel, and it takes a mound of resources to get your hands around it. An aggressive plan.
-
Along with the new CME FAST came the idea of templates and template management. The idea of a hard coded “state” machine became very complex because message structure was dynamic.
-
The only reference implementation was in Java, a C++ one surfaced then was marked unsupported. Our experience along with many others was the C++ implementation was buggy, over complicated (STL), and single threaded. It did not port well to 64-bit environments, and did not even compile under Win32.
-
So as most, we embarked on our own C/C++ decoder, found a reasonable xml parser to read the templates, and started producing “FIX like” messages from the FAST stream. But how do you know the “FIX like” message that you decoded and assembled is really correct ? There is no existing feed to compare it to as RLC does not contain the same FID’s, and there is no checksum or CRC that says “When you build a FIX message from this FAST message, it will CRC to this value…” The best you can do is pass CME certification and assume you have it all correct, building in basic sanity checks that test the business rules against the data.
-
Then you run it against production data at a customer site (the certification is against canned data) and realize that the test data only covered a fraction of the production data set, and you start decoding multicast frames by hand to figure out what is going wrong. Not for the junior programmer or faint at heart.
-
So while all of the complaints against the CME feed are falling on FAST, some are not FAST or FIX related, but rather the implementation of the business logic of the feed.
My long 2 cents.
Daniel May
SpryWare, LLC
Compare FAST with say ASN.1 with DER (disting encod) or PER (packed encod). I would say FAST is vastly simpler by comparison. The concept of complex is all relative.
As to discussion regarding performance. Please note that ALL managed network elements are SNMP capable, hey have embedded a compliant (but subset) of ASN.1. SNMP stress performance tests have shown to have Gigabit capabilities (obviously), not just enc/dec, end to end. with standard server hw, network monitor appss can easily integrate 100MB/s of SNMP traffic. With a simple spec, FAST theoretical limit would be at least at same level, if not even higher.
I will push the ASN.1 comparison even further. In early 90s, ASN.1 compliance was limited to expensive rule compilers, etc. But with SSLeay, a fully PKCS (a cryptography standard relies on ASN.1) compliant open source (well, the whole NSA export mess aside) package blew the industry wide open. Today, you will find very reliable, high performance ASN.1 implementations, whether it is Cisco MIB, Microsoft CryptAPI, or SSLeay (and its successor, OpenSSL), etc, in all browsers, secure network sessions, all shapes and sizes (standard PC, or embedded network devices, even setup boxes) by billions of ppl, and hundreds of millions of servers around the world.
15 yrs ago, I was that proverbial “jr programmer” that was handed a dolley full of freshly minted ITU documentations to write a PKCS compliant engine. After abt a week of “must find easier job where i won’t stare at docs that doesn’t seem to be in English”, want to torch RSA PKCS ref implementation (horribly written, inflexible, and unlike rest of RSA’s code base, full of bugs). I somehow found an alpha of Eric Young’s code base, contributed to that. It was like magic, in 3-4 weeks, I went from utterly clueless and fantasized abt superglue my office door so they can’t come in and fire me, to have majority of a fully compliant, working implementation.
To take complex comparison even further. Hats off to anyone that had to write a PostScript compliant display / print applications. PS was considered horribly complex, and definitely unnecessarily convoluted.
What can I say except yes - it it clearly way too complicated. Given it ignores one bit which is potenitally 20% of capacity I can’t even see that it is ultra compact. But anyway we have gone from a hugley verbose but easy to use format to a very compressed and ultra difficult format. I think best to junk it and start again! At the very least give up on new versions until the documents are perfect and the sample application are usable. The only ones who are going to profit from all of this is the software vendors selling fast fix convertors. But sending data should not be that difficult. Anyone who says this stuff is not difficult tell me how many lines of code for a complete project… I am still writing and looks like months of work…
William,
thank you for your comments. Just so I understand:
-
Which “20% of capacity” bit is ignored?
(for the next round of testing) -
If you would “junk it and start again”, what would you do?
(maybe we can benefit from your ideas) -
Which documents are you referring to?
(FAST specs, specs for a specific feed, …) -
Could you be more specific on the imperfections?
(would be useful for the next round of reviews) -
Which sample application are you referring to?
(I assume you are referring to a specific feed?) -
Which feed are you implementing for?
Let me know it there is any specific info that we can provide you with.
BTW, what’s your company affiliation?
Best,
Rolf
What can I say except yes - it it clearly way too complicated. Given it
ignores one bit which is potenitally 20% of capacity I can’t even see
that it is ultra compact. But anyway we have gone from a hugley verbose
but easy to use format to a very compressed and ultra difficult format.
I think best to junk it and start again! At the very least give up on
new versions until the documents are perfect and the sample application
are usable. The only ones who are going to profit from all of this is
the software vendors selling fast fix convertors. But sending data
should not be that difficult. Anyone who says this stuff is not
difficult tell me how many lines of code for a complete project… I am
still writing and looks like months of work…
20%… I am saying by not using bit 8 in the stream you have lost 1/8 of the data possibly… Sorry that is 12.5%
The Fast Fix documentation and sample code. For eample just to understand what the message structure is I have on page 44 a few words about “Transfer Encoding” with no diagrams etc.
The translation of FIX into Fast is something else again.
The documents are really really hard to get to grips with. Also the sample code is very limited. String Delta operation for example. Unless I am missing something you have not bothered with that in the c# sample code but it used all the time.
I used to get EBS LIVE via XML and the code was trivial - literally a few days work. I am trying to use their new Fast Fix feed but it’s really a nighmare. So far it has taken me about a week and I have still not logged on properly.
Even when i have done all this I see EBS LIVE sends strings like “SPOT.EUR/USD” down the line. It is all a mess.
What would I do instead? I don;t pretend to have an answer to that but for EBS LIVE they could have written a much easier format which saved a lot of space and was super easy to pass. Using Fast Fix was a nighmare and a waste of time unless you own a fast fix engine already. Why bother with mantissa and exponent when all their prices could be sent as integer? they have no text data to send just prices so I would have had no strings in the messages at all. Why send body length and checksums in the fix over fast stuff. Just more work. Why send string timestamps when it takes 40us to turn a time to a string when you could send an int64 tick time. By trying to do everything it just turn into a nightmare and is slow to boot.