Imported from previous forum
The Global Technical Committee and the Market Data Optimization Working Group are pleased to announce the availability of the FAST Specification Version 1.1 Release candidate. The document is publicly available at:
http://www.fixprotocol.org/documents/2811/FAST%20Specification%201.x.05.pdf
This release candidate has been recommended for adoption by the Market Data Optimization Working Group. The public review and comment period begins on August 17th, 2006 and concludes on September 15th, 2006.
Please post all review comments, questions and feedback by replying to this discussion thread. We will use this discussion thread to archive and respond to all feedback received from the public.
[ original email was from Jim Northey - jnorthey@jandj.com ]
We have repeatedly over the past six months had consistent feedback that the specification is not accessible enough and can’t be built from. While we have had consistent feedback to this effect, there also is precedence in other standards bodies for creating specifications that define the rules, but are not in and of themselves sufficient to implement a compliant solution. The proposal has been that we create a separate tutorial document towards that purpose, something very similar to the very accessible basic user’s guide.
However, I would like to state that it has often been the accessibility problem that has limited adoption.
With that in mind, I would like to steal a usenet response from about ten years ago that was posted by Bjarne Stourstrup, the originator of C++. Bjarne said, “I hate to break up this feast of conjecture with a little fact…”.
So, in this vein, I would like to propose a simple compromise. In addition the C++ specification (pre-ANSI, ISO), Stroustrup published an “Annotated C++ Reference Manual”. My thought is that we could create styles within Microsoft Word for annotation sections that would contain background information and examples that provide sufficient context and archetypical examples that would permit implementers to implement the specification.
These annotation styles could be created with hidden text. We could generate two PDFs, one without the annotation and one with the annotation.
Thoughts?
The EBF gammar needs to be review for accuracy and consistency
There were concerns regarding the support for an unsigned integer 64 datatype in 1.1 due to Java’s limitations in not supporting this datatype natively.
[ original email was from Sitaram Guruswamy - sitaram.guruswamy@transacttools.net ]
It looks like ByteVectorFieldInstruction should be associated with the preceding length field specifying the size of the raw byte data it contains - the corresponding byte vector field specification should be similar to the SequenceFieldInstruction in that the SequenceFieldInstruction is also always immediately preceded by a length field specifying the number of encoded elements it contains. For example:
This makes the FAST 1.1 specification consistent with itself – right now sequence and rawdata have similar requirements but are encoded and templated differently. Also, the length attribute will not be repeated when using FAST for FIX - currently ByteVector encoding has a size preamble which duplicates the preceding length field. In addition, this ‘composite’ field is represented by only one bit in the pmap rather than separate bits for the the two fields.
If this is done, the transfer encoding for ByteVectorFieldInstruction has to be changed to not contain the length.
The ByteVector FieldInstruction would follow the same length field rules as a Sequence wrt implicit and explicit declaration of the length field.
Thoughts?
[ original email was from Sitaram Guruswamy - sitaram.guruswamy@transacttools.net ]
More clarification since this came up during the last weekly phone call:
This proposal is only as implementation-specific as the 1.x.05 specification’s field instruction for ‘Sequence’. In other words, it is not implementation-specific. The proposal aims to take advantage of constructs in FIX that already define the length of the raw data field, just as ‘SequenceFieldInstruction’ aims to take advantage of countref fields for repeating groups. However, this approach is applicable for any usecase – even protocols that don’t have an explicit length field. To achieve this, the proposed ByteVector spec uses exactly the same approach that SequenceFieldInstruction takes.
To clarify the benefits, the easiest way is to see what such a field will look like on the stream.
Example 1: Protocol that defines length fields for rawdata fields (e.g., FIX)
Current Spec uses two presence map bits – 1 for length field, 1 for the rawdata field. Data on the wire looks like…|{length}|{length preamble}{raw bytes}|… length is duplicated.
Proposed spec uses one presence map bit – for the composite ByteVectorFieldInstruction. Data on the wire looks like …|{length}{raw bytes}|… no duplication of the length.
Example 2: Protocol that does NOT define a length for raw data fields
Current Spec uses one presence map bits –1 for the rawdata field. Data on the wire looks like… {length preamble}{raw bytes}|…
Proposed spec uses one presence map bit – for the composite ByteVectorFieldInstruction. Data on the wire looks like …|{length}{raw bytes}|…
As you can see, in example 2 there is no gain or loss. But in example 1 (FIX) the proposed spec offers higher compaction and avoids duplication of data.
This proposal is only as implementation-specific as the 1.x.05
specification’s field instruction for ‘Sequence’. In other words, it is
not implementation-specific.
I agree that the solution in it self is as generic as the sequence case. The comment made about implementation-specific was whether an implementation could reuse code for processing sequence and byte vector preambles and that this ability is not dependent on the presence of a length element on the byte vector.
To clarify the benefits, the easiest way is to see what such a field
will look like on the stream.Example 1: Protocol that defines length fields for rawdata fields
(e.g., FIX)Current Spec uses two presence map bits – 1 for length field, 1 for the
rawdata field. Data on the wire looks like…|{length}|{length
preamble}{raw bytes}|… length is duplicated.
The current FAST spec says nothing about how to map a FIX rawdata field to a byte vector. A FIX over FAST mapping should typically say that the length field of a rawdata field should be implied by the length of the byte vector (or most often Unicode string). This means that the decoder will have to generate the length field out of the intrinsic length of the byte vector/Unicode string. So it’s perfectly possible to just send |{length preamble}{raw bytes}| with the current spec and there is no difference in compaction in this regard.
So let’s back up a bit to your original motivation which was consistency. I agree that adding the feature to byte vectors would not be inconsistent with sequences. However whether not having the feature on byte vectors is inconsitent with sequences depends on if you see a byte vector as an atomic unit or not. I’d argue that treating it as two separate parts as in FIX is an artifact of the limitiations of that protocol that we can avoid duplicating in FAST. Actually the same holds for the length preamble of sequences, but the td:length element is there because we wanted to make it possible to specify field operators for the length preamble.
So there’s really only a single functional motivation for having a td:length element on byte vector (and Unicode string) and that is the possibility to specify a field operator for the length preamble. The question is if the need for such a feature is strong enough to motivate its inclusion at this point. Let’s hear what others think.
/David
[ original email was from Matt Simpson - msimpson@cme.com ]
I think we should defer this feature to a later version of FAST assuming we see a reasonable need for it in the future.
The proposed extension of Byte Vector would allow an external field to be substituted and used as a length parameter while using only a single pmap slot*. This is similar to the Sequence Field Instruction in that the length of the sequence can also be assigned to an explicit field. However, it is different in that the length of Byte Vector does not function as a stand-alone field which is the case for the length preamble in Sequence Field Instruction.
This would in essence be creating a composite field from two external fields (in this case, RawDataLength and RawData) and represented with a single pmap slot. While a potentially useful feature, it doesn’t match the behavior of the field types in the spec at this point and so I propose that it be deferred. I’d like to compliment Sitaram and thank him for the recommendation.
Of course, I’m open to further discussion on the topic.
Matt
*pmap slot required under certain conditions - see http://fixprotocol.org/discuss/read/40bbf326
This proposal is only as implementation-specific as the 1.x.05
specification’s field instruction for ‘Sequence’. In other words, it
is not implementation-specific.I agree that the solution in it self is as generic as the sequence case.
The comment made about implementation-specific was whether an
implementation could reuse code for processing sequence and byte vector
preambles and that this ability is not dependent on the presence of a
length element on the byte vector.To clarify the benefits, the easiest way is to see what such a field
will look like on the stream.Example 1: Protocol that defines length fields for rawdata fields
(e.g., FIX)Current Spec uses two presence map bits – 1 for length field, 1 for
the rawdata field. Data on the wire looks like…|{length}|{length
preamble}{raw bytes}|… length is duplicated.The current FAST spec says nothing about how to map a FIX rawdata field
to a byte vector. A FIX over FAST mapping should typically say that the
length field of a rawdata field should be implied by the length of the
byte vector (or most often Unicode string). This means that the decoder
will have to generate the length field out of the intrinsic length of
the byte vector/Unicode string. So it’s perfectly possible to just send
|{length preamble}{raw bytes}| with the current spec and there is no
difference in compaction in this regard.So let’s back up a bit to your original motivation which was
consistency. I agree that adding the feature to byte vectors would not
be inconsistent with sequences. However whether not having the feature
on byte vectors is inconsitent with sequences depends on if you see a
byte vector as an atomic unit or not. I’d argue that treating it as two
separate parts as in FIX is an artifact of the limitiations of that
protocol that we can avoid duplicating in FAST. Actually the same holds
for the length preamble of sequences, but the td:length element is there
because we wanted to make it possible to specify field operators for the
length preamble.So there’s really only a single functional motivation for having a
td:length element on byte vector (and Unicode string) and that is the
possibility to specify a field operator for the length preamble. The
question is if the need for such a feature is strong enough to motivate
its inclusion at this point. Let’s hear what others think./David