Imported from previous forum
An example of adding a repeating group would be a set of recommendations to any order. An example of adding var data would be a transaction code or one time key that must be used for further reference.
More importantly, an extension system that is not comprehensive can lead to nasty error cases. What happens if someone adds a new repeating group before existing var data? How is this handled if the spec does not say anything about it?
This proposal from the High Performance Working Group entails the use of a FIX Trading Community designed Simple Binary Encoding (SBE) to produce fast and compact encodings of FIX messages. SBE provides different characteristics than other binary encodings. It is optimized for low latency.
This new FIX binary encoding complements the existing only binary encoding developed in 2005 (FAST) which had a focus on reducing bandwidth utilization for market data. In addition, the encoding is also defined and controlled within the FIX Trading Community, in contrast to the binary encodings proposals to encode FIX with Google Protocol Buffers and ASN.1.
The second release candidate documents for SBE can be found here.
The proposed technical standard for Simple Binary Encoding of FIX is following the technical standards review process described in the document found here.
This second release candidate now enters a public comment period in which public review and feedback is encouraged. The public comment period will run for a period of 90 days beginning on Monday the 31st of March, 2014.
Please post feedback, comments, and questions as replies to this discussion thread.
Hello,
the posted SBE - RC2 document contains some errors:
1. Reference source not found:
p 20 clause 2.2.5 table entry for String
p 71 clause 5.3.1
2. symbolicName_t written w/o '_'
p 59 clause 4.4.5.3, two occurrences
p 65 clause 4.6, first line on p 65
p 66 clause 4.7, last but two line of first coloured block
Regards
Andreas Kolf
t
Andreas, thanks for pointing out document errors. I'll fix them in a future draft.
The broken link in section 2.2.5 should say "Variable-length string" and the one in section 5.3.1 should say "See section 3.1 above."
Hello,
Regarding section 2.2.2 (Versioning), I'd like to suggest adding two additional fields to the MessageHeader structure, designating the number of groups and number of variable length data fields in the message.
This would improve extensibility by allowing new groups and data fields to be added without breaking backwards compatibility (old decoders would skip the new groups/data fields it didn't understand based on their block length).
With the current spec it is possible to add new fields to the end of a group, but not a new group or new variable length data element.
There is some discussion of this here: https://github.com/real-logic/simple-binary-encoding/issues/109
Regards,
David Chapman
The SBE RC2 spec provides for fixed-length character fields and variable-length data. These correspond to traditional String and Data data types respectively.
Variable-length data may be used for encoded text (non-ASCII characters) or raw data, however. Both types can be qualified with “characterEncoding” attribute in a message schema to specify an encoding.
There is a gap in the spec in that there is no explicit way to declare non-textual data. It should be possible to declare raw data of both fixed and variable length.
Examples of fixed-length non-textual data:
- UUID is 16 octets, but is not a String. (The proposal for a new session protocol uses UUID for a unique identifer, so this is an important use case.)
- IP address could be declared either as an integer, or more conveniently, as a 4 octet non-text field. Like UUID, there is a common string representation, but it would be more efficient to send it as binary data.
In general, the SBE XML schema for message schemas is unnecessarily restrictive. Here are some suggestions for improvement, which are mainly supersets or less restrictive versions of the current format:
- The order of encoding types is controlled by a strict sequence: simple types, composites, enums and then sets. This strict ordering is not necessary. An author should be able to list encodings in any order deemed logical.
- Currently, a composite encoding type (equivalent of a C struct) can only contain simple types. It would be useful to allow containment of enumerated values and bitset elements as well. For example, a timestamp could be encoded as a composite of an integer for the number of ticks and an enumeration of time units of precision, such as milliseconds, microseconds, nanoseconds. Even nested composite encodings may be useful.
- Through the extension mechanism, it possible to add fields to a message or add a whole new message in a later version of a schema. However, there is no way to note that an old message or field should no longer be published. Add a way to mark an element as deprecated.
- Fields and encodings can be associated to a traditional FIX data type by specifying a “semanticType” attribute. However, there is no way to declare a new semantic (application layer) class. For example, to use UUID as a unique identifier (see posting above), it should be possible to declare a new semanticType for that purpose since it does not correspond to an existing FIX type. The declaration for a semanticType would tell its required properties, such as mantissa an exponent for a decimal price. Then an encoding that associates to the semantic class could be verified to contain required properties.
Even the existing FIX types should be removed from the XSD and extracted to a separate XML document as the base declaration of FIX types. (XSD just doesn’t work well for constants.)
The extension proposal by David Chapman would be very valuable. For full effect I would add the number of repeating groups and vardata fields to message header and the repeating groups header. This allows the repeating groups to be extended with repeating groups. This also makes the code much simpler for writing an OTF decoder as messages become a recursive structure.
With these header changes in place it would then be possible to allow vardata at the end of repeating groups. This is an often requested feature and something seriously lacking in SBE.
The full extension mechanism and ability to allow vardata within repeating groups could be implemented with negligible performance impact to those who do not use these features.
I would highly recommend these features are added for RC3 so that SBE can be generally applicable for finance applications because without them it is currently too restrictive.
I share Martin Thompson’s opinion on the additions for RC3 and believe the extension proposal by David Chapman would be valuable and open up the use cases for SBE dramatically. So, I very much support adding these changes.
To be specific, the number of repeating groups and vardata fields should be added to the message header and the repeating group headers. The results of these changes pave the way for allowing vardata in repeating groups as well as more freedom to add new repeating groups, etc.
I agree with Martin and Todd that SBE needs to be extended to allow extension. I am not convinced that adding the number of repeating groups and vardata fields to each message and group header is the optimal way to do it. I think some experimenting is needed to compare different design alternatives.
Even without changing the message header, it should be possible to append repeating groups and variable data at the end of a message, so long as a framing header provides overall message length. Then a decoder knows how many bytes to consume in a stream to reach the next message, even though the new elements cannot be parsed.
Don I think the message length alone is insufficient. Take the simple example of having one repeating group and one variable length data field in a message. Now try to add another repeating group by way of extension. This is not possible as the existing variable length data field is now in the wrong place and we have no way to tell existing compiled stubs of this change.
I’ve given this a fair amount of thought and believe the best solution for SBE to properly support extension, and variable data in repeating groups, is to add the count of repeating groups, and count of var data fields to both the message header and the repeating group header. This way for minimal performance and space hit we get much improved functionality.
Martin, agreed. I should have said you can add a repeating group OR variable length data field at the end of a message. You cannot insert an element in the middle without additional metadata. The working group discussed this proposal, and the technical impacts seems minimal. However, it would be helpful if someone could provide a use case as a business rationale. Can anyone cite an example of adding a whole repeating group or data element to an existing message?
We are tracking adding variable length data to repeating group entries as a separate issue, and the consensus is to allow it. The rationale is that there are existing FIX message with that layout. For example, Security Definition contains EncodedLegSecurityDesc in its InstrumentLeg group.
Hello, thank you for this RC2. Three remarks:
-
I’m not sure it is useful to specify a “semanticType” attribute in addition of the “type” attribute. I can see it leads to some confusion. For example, type=“intQty32” which mixes the name of a raw type with a semantic name. Or <field type=“ratio” name=“CurrencyRatio” id=“1382” semanticType=”float”/>.
-
blockLength in the message header does not make sense to me. If you really want to group fixed-length values into a block with some reserved space, why not doing something like repeating groups. For example:
<group blockLength="…">
<field name=“ClOrdID” …/>
<!-- other fixed-length values -->
</group>
<group repeating=“true”>
<field name=“PartyID” …/>
<!-- other values of the repeating group -->
</group>
It would be much more easier for an application to handle such definition and it would also allow to specify sub-groups and get rid of the “offset” attribute:
<group blockLength="…">
<field name=“ClOrdID” …/>
<group blockLength="…">
<field name=“OrderQty” …/>
</group>
<!-- other fixed-length values -->
</group>
- Maybe you need to add the bitmap type. For example, a int32 value to specify the presence or not of the next 32 field values. I think it could be useful for market data messages.
Regards,
Xavier.
Xavier, thanks for taking the time to look at RC2 and providing comments.
The semantic type was added to support an automatic conversion to the tag=value data type in the FIX repository. The sample type “intQty32” is just an example and not a primitive or semantic data type of SBE. Section 2.1.2 says “An encoding maps a FIX data type to either a simple, primitive data type, such as a 32 bit signed integer, or to a composite type. A composite type is composed of two or more simple primitive types.”. The names of composite types can be freely chosen. I agree the choice of “intQty32” can be confusing and we will look for a better name that does not contain predefined terms for data types, e.g. “myQuantity”.
The topic of repeating groups has not been a top priority for Version 1.0 as the recommendation is to avoid them (as well as variable length fields) in your message design in the first place if you want the highest performance. Direct access to fields has been the top priority. At the same time we wanted Version 1.0 to be complete in the sense that you can encode any existing FIX message type. If you look at high performance interfaces of exchanges out there then you will very rarely find repeating groups, mainly only for a list of partial fills of a single match event and for the legs of a multileg instrument. These can be put at the end of the message to give direct access to all root level fields. This is not possible if you mix and match block of fixed and variable length in your message and have to walk through it to parse it. However, we want to optimize repeating groups in a future version.
Bitmap types have a similar issue in terms of direct field access and parsing effort. FAST uses such presence maps and it adds to complexity. The “S” in SBE is there for a reason and bandwidth was not a primary concern. SBE was primarily intended for the transactional side of trading but it can also be used for market data. However, there is interest in an additional compact binary encoding specifically for market data. If you are interested then please contact the FPL Program Office to join the High Performance Working Group.
To add further explanation to Hanno Klein’s response to Xavier:
-
The “type” attribute of a field gives a reference to a wire format as a simple primitive or composite encoding. Wire encoding is at presentation layer (layer 6). On the other hand, “semanticType” maps a field to an application layer (layer 7) classification. For example, the presentation layer encoding of a field could be numeric, while its semanticType is “Price” to indicate that it represents a monetary value as opposed to a simple counter. An application can use this information to select an appropriate implementation. In the case of Price, it could use a class that knows about tick size, currency, etc.
-
By default, fields are packed without padding in SBE, so the length of a block is the sum of its field sizes. The “blockLength” field of message header and group dimension serves two purposes. First, specifying the optional attribute in a schema allows the root block of a message or a repeating group entry to be padded to effect byte alignment. Likewise, the optional “offset” attribute can be used to control alignment of fields within a block. Second, blockLength on the wire supports schema versioning and gradual migration of changes. A decoder consumes the number of octets specified for the block even if it wasn’t built with the latest version of the schema. See section 5 of the SBE spec for a detailed explanation.
-
To clarify, SBE does have bitmap as a data encoding, but it is not used for field presence. Although optional fields are allowed in SBE, null fields do not reduce message size as in other encodings. There are alternative approaches, but the philosophy is to target message templates to specialized use cases with just the fields required for that transaction as opposed to overloading a message design with many optional fields for many scenarios.
- In my opinion, “semanticType” should not be specified for fields but only for type entities.
<type name=“range06” primitiveType=”uint8” maxValue=”6”
presence=”optional” nullValue=”255” />
<field type=“range06” name=“MaxPriceLevels” id="1090"
semanticType=“int”/>
should actually be
<type name=“range06” primitiveType=”uint8” maxValue=”6”
presence=”optional” nullValue=”255” semanticType=“int” />
<field type=“range06” name=“MaxPriceLevels” id=“1090”/>
which would be consistent with
<composite name=“intQty32” semanticType=“Qty”>…</composite>
<field type=“intQty32” name=“OrderQty” id=“38” description=“Total number of shares” />
In that case, no problem to give a “semantic” name (type=“intQty32”) to the type of a field.
- Thank you to make clearer the goals of SBE. I understand better some design choices now. My purpose is to be able to decode/encode the body and the repeating groups in the same way. So we can decode/encode them recursively. This is the reason why I think blockLength should be moved to the beginning of each group (instead of the header for the body and instead of a global length for repeating groups). In this way, the format of a group would be
body = group
group = [block length][fixed-length value][[NumInGroup][group]][variable-length value]
- I have actually no requirements to support bitmaps. I just know it is commonly used to encode market data messages. Supporting them should not break the philosophy of the SBE since you already support variable values. However it could be added in a future version if really needed.
I still believe that the purpose of the semantic type has not been made clear enough. It is not an alternate representation of a type. It is the FIX data type defined for a FIX field in the FIX repository. User-defined fields have no semantic type as they are not part of the standard FIX repository.
Xavier has a point as we have also assigned semantic types to composite types in the SBE example in chapter 2.5.5. However, I cannot find “intQty32” in the FIX repository. It may be better to only use the attribute semanticType together with fields (in this case OrderQty(38)). The idea is that as a schema designer, I never have to enter the semanticType manually but can lookup the field name in the FIX repository. As I schema consumer, I do not have to do a lookup in the FIX repository when I want to create the correct data type for a tag=value syntax. The schema gives me the semantic type for convenience.