Imported from previous forum
In a recent post on another thread, Hanno said:
I’m working together with the core mdowg group to improve documentation,
presentation and training materials to improve the learning curve for
new-comers, as well as encouraging existing implementers to be able to
reap the full benefit of FAST.We welcome any comments and suggestions on what to include in the work
that we currently do
As I developed QuickFAST [see my sig], I spent a lot of time with the FAST documentation. In general, I have found it clear and unambiguous as a specification, but not as a tutorial. This is a good thing. Reference manuals and tutorials should be separate.
The FAST protocol is moderately complex, but the complexity is obviously targeted at the goal achieving a high degree of practical (as opposed to theoretical) lossless data compression. The learning curve is steep but it makes sense once you get it.
The proposed extensions to encoding in FAST 1.2 continue this clarity. FAST 1.2 adds some useful new data types while supporting backward compatibility such that a properly written FAST 1.2 decoder should be able to handle a FAST 1.1 data stream without problems.
I had a minor concern about the FAST 1.2 bit map field. As originally proposed it did not allow representation of an empty set of bits. However when I expressed this concern and the response on this list was positive.
I also have some concerns about the proposed modifications to the XML template definition language, but they are cosmetic rather than functional. For example, rather than:
[field] [type name="abc"/][/field]
I would much prefer
[field type="abc"/]
However, that’s certainly not a show-stopper. (substitute angle brackets for where appropriate)
One thing I appreciate about the FAST specification is the separation of the payload-data-encoding-decoding issues from the packet framing and session management issues. This was a severe drawback to the FIX standard in which message framing, session management and application data issues were muddled together. FAST has avoided this by focusing completely on the transcoding/compression issues and trusting that other layers of the software will address other issues.
Unfortunately at the moment these “other layers” are left mostly unspecified. I say “mostly” because Session Control Protocol (SCP) attempts to address some of the session layer issues , and the FAST specification does mention the possibility of “blocking” the FAST encoded messages.
Blocking is optional and does not appear to have been accepted by the industry which seems to be developing a number of ad hoc and incompatible alternatives to handle transmission of FAST on a (logically) streaming medium (TCP) as opposed to a logically packetized medium(UDP and Multicast) . This to me is the most important issue that needs to be addressed – preferably by a solution that is clearly distinct from the encoding/decoding concerns of the FAST protocol.
Some concerns that could be addressed at the message framing level include:
– Resynchronization after “something goes wrong.”
Suppose for example, a fast decoder is delivering data to the application code on-the-fly as opposed to assembling a complete message before delivering it to the application (there are some obvious latency advantages in doing this.) If the application were to throw an exception, it would be very useful to have a way to find the next “clean point” from which decoding could continue.
There is more involved in this than just identifying the next message because the issue of resetting the data dictionaries needs to be addressed as well. On a multicast data feed this is simply a matter of restarting the decoder at the next packet. However this is an artifact of the lossy and packetized nature of multicast. On a reliable transport or a streaming transport the problem is unsolved.
– Providing “filtering” information at the framing level to allow an application to determine that this message is not of interest and skip forward to the next message in the stream of data with minimal decoding overhead. Again the issue of resetting the dictionaries would have to be considered. (CME’s FIX/FAST 2.0 attempts to address this.)
It might be argued that having a framing protocol that addresses these needs defeats some of the advantages of FAST encoding, however one thing to consider is there is already “framing” information inherent in the IP protocol. For every packet on the wire there are somewhere between 20 and 40 bytes of “overhead” information on the wire (IP and TCP and/or UDP headers, etc.). A well designed framing protocol that addresses the above concerns with, say, 8 to 12 bytes of overhead could easily be an overall win in effective, usable, throughput and reduced latency.
Of course this is speculation on my part, but I think it’s an issue worth looking into.
And finally there is the session management layer. I won’t even try to get into that in this other than to say that in my opinion SCP needs a lot of work before it addresses the practical session management issues that are likely to be encountered as FAST is used in a wide variety of circumstances.
Dale Wilson
Principal Software Engineer
Object Computing, Inc.
Lead developer of the QuickFAST open source implementation of FAST.
Dale, thank you for your comments and suggestions.
In a recent post on another thread, Hanno said:
I’m working together with the core mdowg group to improve
documentation, presentation and training materials to improve the
learning curve for new-comers, as well as encouraging existing
implementers to be able to reap the full benefit of FAST.We welcome any comments and suggestions on what to include in the
work that we currently do
I believe you quoted me, not Hanno.
And finally there is the session management layer. I won’t even
try to get into that in this other than to say that in my opinion
SCP needs a lot of work before it addresses the practical session
management issues that are likely to be encountered as FAST is
used in a wide variety of circumstances.
When we were finishing SCP 1.1, we considered different alternative designs and scopes. A majority of the working group members wanted to implement a minimal set of features. Anything beyond what we all agreed was essential, was suggested to be added later after a more thorough analysis. We’ve had very few comments, questions and requests regarding the SCP so far.
The name SCP may be somewhat misleading. A FAST “session” is not a transport session nor anything like it, but rather a light-weight context for encoding and decoding FAST messages. Please refer to the introduction section of the SCP specification.
Maybe it is time to re-visit and discuss which features we would like to have.
I’d be interested in hearing what you think should be added to SCP.
Best,
Rolf
Finally someone is starting to pick up things. Obvious mistakes, and there is a lot more to it.
First civil criticism and technically sound it is, but I don’t think the list and people it is aimed at deserve it at all.
(btw, rolf, you have further fast-challenged comments in a response below to the concerns the same set of people keep ignoring and diverting from).
Continuing my suggestions for improving FAST…
Dynamic memory allocation can be a real performance killer. In C++ there are a number of techniques that can be used to avoid dynamic memory allocation or minimize its impact including keeping thread-specific memory pools and caching objects for re-use rather than deleteing/newing them, etc.
The one technique I would like to use, however, eliminates dynamic memory allocation altogether. That is preallocating space for the largest possible message. I have been able to use this with data feeds from particular sources to achieve some dramatic results. However, given the current definition of FAST templates there is no way to achieve these results in the general case, because there is no way to determine what “the largest possible message” is.
So my suggestion is add upper-bound information to field instructions that produce variable sized objects. In particular, the sequence, string, and byte vector instructions could have a bound=“n” attribute that would be a commitment by the data source/template author that field being transmitted would never exceed ‘n’ elements. This information is often available to the data source/encoder. Making it available in the template would give the decoder the information it needs to avoid dynamic allocation in the general case.
Of course this would be an optional attribute with the default being that the number of elements is unbounded (or at least indeterminate.)
Dale
Principal Software Engineer
Object Computing, Inc (www.ociweb.com)
Lead Developer of the QuickFAST Open Source C++ implementation of the FAST protocol. (www.quickfast.org)
Dale,
the extensibility mechanism in FAST 1.1 can be used to specify maximum lengths etc. Please refer to section 9.
/Rolf
Continuing my suggestions for improving FAST…
Dynamic memory allocation can be a real performance killer. In C++
there are a number of techniques that can be used to avoid dynamic
memory allocation or minimize its impact including keeping thread-
specific memory pools and caching objects for re-use rather than
deleteing/newing them, etc.The one technique I would like to use, however, eliminates dynamic
memory allocation altogether. That is preallocating space for the
largest possible message. I have been able to use this with data
feeds from particular sources to achieve some dramatic results.
However, given the current definition of FAST templates there is no
way to achieve these results in the general case, because there is
no way to determine what “the largest possible message” is.So my suggestion is add upper-bound information to field instructions
that produce variable sized objects. In particular, the sequence,
string, and byte vector instructions could have a bound=“n” attribute
that would be a commitment by the data source/template author that
field being transmitted would never exceed ‘n’ elements. This
information is often available to the data source/encoder. Making it
available in the template would give the decoder the information it
needs to avoid dynamic allocation in the general case.Of course this would be an optional attribute with the default being
that the number of elements is unbounded (or at least indeterminate.)Dale
Principal Software Engineer Object Computing, Inc (www.ociweb.com)
Lead Developer of the QuickFAST Open Source C++ implementation of
the FAST protocol. (www.quickfast.org)
the extensibility mechanism in FAST 1.1 can be used to specify maximum
lengths etc. Please refer to section 9.
Agreed, but because the length issue is not explicitly addressed in the specification, exchanges (template authors) may not think of it or may be reluctant to add custom extensions.
Even if they did, there’s nothing leading different exchanges to choose the same name for the extension attribute: (capacity, size, upper_bound, length, max_length, …) so I’m still stuck with writing custom adapters for different data feeds.
Of course all the market data sources could join together and choose a single name for the max-string-length and max-sequence-length attributes, but such an agreement seems to me to be exactly what the specification is there for in the first place.
The real question is, is this an important enough issue that it should be addressed in future versions of the specification. To me it seems like it is, but other opinions may vary. FAST works OK without this feature, but I think it would be better if the issue were addressed.
Dale
PS: As one example of this issue being addressed in another standard, CORBA allows the definition of bounded and unbounded strings in IDL:
attribute string name;
is unbounded
attribute string[10] serial_number;
has a maximum length of 10 characters.
(subbing square brackets for angle brackets to make the message archives happy)
Hi Dale,
This is a recurrent topic 
I posted about this earlier and I still think it should be a mandatory attribute in the templates.
Proposal for a MaxLength string attribute:
http://fixprotocol.org/discuss/read/702f62f0
maxLength attribute for strings and vectors:
http://fixprotocol.org/discuss/read/e5c03358
Marc
Continuing my suggestions for improving FAST…
Dynamic memory allocation can be a real performance killer. In C++ there
are a number of techniques that can be used to avoid dynamic memory
allocation or minimize its impact including keeping thread-specific
memory pools and caching objects for re-use rather than deleteing/newing
them, etc.The one technique I would like to use, however, eliminates dynamic
memory allocation altogether. That is preallocating space for the
largest possible message. I have been able to use this with data feeds
from particular sources to achieve some dramatic results. However, given
the current definition of FAST templates there is no way to achieve
these results in the general case, because there is no way to determine
what “the largest possible message” is.So my suggestion is add upper-bound information to field instructions
that produce variable sized objects. In particular, the sequence,
string, and byte vector instructions could have a bound=“n” attribute
that would be a commitment by the data source/template author that field
being transmitted would never exceed ‘n’ elements. This information is
often available to the data source/encoder. Making it available in the
template would give the decoder the information it needs to avoid
dynamic allocation in the general case.Of course this would be an optional attribute with the default being
that the number of elements is unbounded (or at least indeterminate.)Dale
Principal Software Engineer Object Computing, Inc (www.ociweb.com) Lead
Developer of the QuickFAST Open Source C++ implementation of the FAST
protocol. (www.quickfast.org)