Imported from previous forum
This proposal from the High Performance Working Group entails the use of an FPL designed Simple Binary Encoding to produce fast and compact encodings of FIX messages. Simple Binary Encoding (SBE) provides different characteristics than other binary encodings. It is optimized for low latency. This new FPL binary encoding complements the existing only binary encoding developed in 2005 (FAST) with a focus on reducing bandwidth utilization for market data. In addition, the encoding is also defined and controlled within FPL only in contrast to the binary encodings proposals to encode FIX with Google Protocol Buffers and ASN.1. The gap analysis and the technical standard proposal for SBE can be found here.
The proposed technical standard for Simple Binary Encoding of FIX will following the technical standards review process described in the document found here.
Technical_Standards_Proposal_Process This release candidate now enters a public comment period in which public review and feedback is encouraged. The public comment period will run for a period of 90 days beginning on Monday the 3rd of June. Please post feedback, comments, and questions as replies to this discussion thread.
Table in section 2.5.2:
- Column headings are in the wrong order, should be decimal/decimal32/decimal64.
- Values inside the table need to be checked, e.g. nullValue for decimal32 should be -2^31 and not -2^32
Table in section 2.6.4:
Values should be distinguished between uint8 and uint16, e.g. maxValue for uint8 cannot be 65534.
Table in section 2.5.2:
- Column headings are in the wrong order, should be decimal/decimal32/decimal64.
- Values inside the table need to be checked, e.g. nullValue for decimal32 should be -2^31 and not -2^32
Table in section 2.6.4:
Values should be distinguished between uint8 and uint16, e.g. maxValue for uint8 cannot be 65534.
Agreed. I will fix those in the next draft.
All,
Most of the comments below are already on CDT in various discussion threads, as well as in various mail threads, but I thought it would be good to collect them in one place:
-
The section on repeating groups should be extended to also include support for groups with varying length. It should be possible to encode any FIX message (and any message for that matter) even if SBE will mostly be used to encode a small subset. There is no good reason for not supporting this. I don’t see how we can argue that we don’t even support common protocol construct even if we don’t optimize for them.
-
The schema language is verbose, incomplete and largely undocumented. Any modern protocol and especially a loosely defined meta-oriented protocol such as SBE should have a well-defined and specified schema language with an easy to use syntax for authoring. The current syntax is not fit for use by humans. It may be used as an intermediate representation for code generators and other tools. One (sub-optimal) alternative is to define a separate schema syntax for authoring.
-
Already defined features as well as the yet to be defined support for repeating groups with varying length fields should be thoroughly tested using validated test data sets containing use cases that are deemed to be relevant to live production.
-
The tests should both include evaluation of performance of different nesting mechanisms, as well as evaluation of the difficulty/ ease of implementation. This is essential to create trust in the proposed specification. There are a number of good protocols out there already and SBE must prove itself as an alternative that is worth implementing. Otherwise, it will mostly be implemented as one or more exchanges mandate its use.
-
At least two independent implementations are needed. It is unlikely that SBE will gain wider adoption without good open-source implementations for multiple language environments; C/C++, Java, Python, C# etc. There is also a risk that specification errors are not found until one or more groups do a serious implementation effort. This should ideally happen as early as possible.
Best,
Rolf
Hello i have a java implementation that is very close to the specs.
If you have test cases i can prove it stronger, i know that there is a gap i would like to fill it first, then, ask the others owners of the code, and i think is it possible.
Regards
Jaime.
All,
Most of the comments below are already on CDT in various discussion threads, as well as in various mail threads, but I thought it would be good to collect them in one place:
The section on repeating groups should be extended to also include support for groups with varying length. It should be possible to encode any FIX message (and any message for that matter) even if SBE will mostly be used to encode a small subset. There is no good reason for not supporting this. I don’t see how we can argue that we don’t even support common protocol construct even if we don’t optimize for them.
The schema language is verbose, incomplete and largely undocumented. Any modern protocol and especially a loosely defined meta-oriented protocol such as SBE should have a well-defined and specified schema language with an easy to use syntax for authoring. The current syntax is not fit for use by humans. It may be used as an intermediate representation for code generators and other tools. One (sub-optimal) alternative is to define a separate schema syntax for authoring.
Already defined features as well as the yet to be defined support for repeating groups with varying length fields should be thoroughly tested using validated test data sets containing use cases that are deemed to be relevant to live production.
The tests should both include evaluation of performance of different nesting mechanisms, as well as evaluation of the difficulty/ ease of implementation. This is essential to create trust in the proposed specification. There are a number of good protocols out there already and SBE must prove itself as an alternative that is worth implementing. Otherwise, it will mostly be implemented as one or more exchanges mandate its use.
At least two independent implementations are needed. It is unlikely that SBE will gain wider adoption without good open-source implementations for multiple language environments; C/C++, Java, Python, C# etc. There is also a risk that specification errors are not found until one or more groups do a serious implementation effort. This should ideally happen as early as possible.
Best,
Rolf
SBE supports enumerations of valid values that may be used in a field. Message schemas also support fields with a constant value. In some cases, that value might be defined in an enumeration. It should be possible to specify the constant value by reference to the enumeration rather than repeating the literal value.
Example: SecurityIDSource has an enumeration of valid values. A constant field of that domain should be able to specify its value by the symbol ‘ISIN number’ rather than the code value ‘4’.
Agree, but it should be the standard symbolic name of the enum as defined in the FIX Repository, i.e. ‘ISINNumber’ without any blanks.
SBE supports enumerations of valid values that may be used in a field. Message schemas also support fields with a constant value. In some cases, that value might be defined in an enumeration. It should be possible to specify the constant value by reference to the enumeration rather than repeating the literal value.
Example: SecurityIDSource has an enumeration of valid values. A constant field of that domain should be able to specify its value by the symbol ‘ISIN number’ rather than the code value ‘4’.
It is not always practical to update all message publishers and consumers simultaneously. Therefore, it would be desirable to be able to extend message schemas while keeping a level of back compatibility for old consumers.
- Schemas should be versioned
- It should be possible to add fields at the end of message or at the end of a repeating group entry without breaking compatibility.
Clarify the range of values for required integer fields. The table in section 2.4.2 gives the min and max values and null indicators for integer types. For example, for uint16 the min value is 0, max value is stated as 65534 and null value is 65535. However, null is only necessary for optional fields, not for required fields. Therefore, a required field of type uint16 should allow a max value of 65535. For clarity, the documentation could provide a table of optional field ranges (as-is) and a separate table of required field ranges without the row for nullValue.
What is the benefit of being able to have one more valid value in case of a required field? How often do applications need to convey the value 65535 and would save 2 bytes (not having to use uint32) if it is part of a required field? We should weigh any benefit against the additional complexity in making the maximum value conditional upon another attribute of the field.
Clarify the range of values for required integer fields. The table in section 2.4.2 gives the min and max values and null indicators for integer types. For example, for uint16 the min value is 0, max value is stated as 65534 and null value is 65535. However, null is only necessary for optional fields, not for required fields. Therefore, a required field of type uint16 should allow a max value of 65535. For clarity, the documentation could provide a table of optional field ranges (as-is) and a separate table of required field ranges without the row for nullValue.
What is the benefit of being able to have one more valid value in case of a required field? How often do applications need to convey the value 65535 and would save 2 bytes (not having to use uint32) if it is part of a required field? We should weigh any benefit against the additional complexity in making the maximum value conditional upon another attribute of the field.
Agreed that we should always weigh benefit against complexity, but I see this as a simplification for a decoder of a required field. It’s not about gaining one more integer value, but rather, to eliminate a range check in the absence of an explicit min or max for the field. I expect that most SBE templates will be designed to make the majority of fields required, so we could potentially eliminate many code branches.
On the other hand if we keep the null value as invalid even for a required field, then to be strict, the decoder needs to check for that value and raise an exception if it is received.
Our experience is that it is dangerous to have a single value of a given data type that is a null value in some context and a “real” value in another context. The user application has data structures with fields that can be initialized with null values. Until these values are put into the context of a message, it is undefined whether they have a valid value. If the application “forgets” to set a real value and sends it as a required field then the recipient would use the initial value, e.g. 65535, as “real” value which can have a serious impact if it happens to be the OrderQty field.
Range checks for required fields not having min/max only need to identify the presence of the null value. That process is identical for required and optional fields, the interpretation is different as it is allowed for optional and invalid for required fields.
We feel that it is safer to always have an explicit null value and raise an exception if it is provided for a required field. Otherwise you have values that sometimes represent null which in turn is sometimes permitted. The key question is probably what the penalty of a null value detection for required fields is. Maybe we need another schema level attribute to define whether null values apply to required fields or not. If they do not then no checks will be conducted and the user needs to be aware that null values will not lead to a rejection of messages if he uses them for required fields. Currently I see this as an optimization that requires testing to determines performance gains.
Agreed that we should always weigh benefit against complexity, but I see this as a simplification for a decoder of a required field. It’s not about gaining one more integer value, but rather, to eliminate a range check in the absence of an explicit min or max for the field. I expect that most SBE templates will be designed to make the majority of fields required, so we could potentially eliminate many code branches.
On the other hand if we keep the null value as invalid even for a required field, then to be strict, the decoder needs to check for that value and raise an exception if it is received.
Our experience is that it is dangerous to have a single value of a given data type that is a null value in some context and a “real” value in another context.
I agree with Hanno that it is dangerous to use a value from the ordinary value range of application type. It is an incomplete design and it will come back to bite us later.
The issue was discussed at length about a year ago at:
https://fixprotocol.centraldesktop.com/simplebinaryencodingsubgroup/discussion/19810900/
(the thread started July 23rd and ended 42 posts later on July 26th)
The proposed SBE NULL value mechanism is not generic enough and needs to be replaced or amended to cover all relevant use cases.
/Rolf
Some received feedback on SBE documentation:
Use of the term "template" is inconsistent. Sometimes the word refers to a particular message layout and other times to a whole message schema (a whole file of encoding and message declarations).
Readers have questioned the relationship of MsgType (traditionally tag 35) to a message template and they are used to seeing MsgType in a header or as a field. In a schema, there is an optional attribute of a "message" element called "fixMsgType" to document the FIX semantic message type. Clarify that template ID does not have one-to-one relationship with MsgType, and MsgType is not a field on the wire in SBE.
Hi,
I am new to SBE.
In section 2.5.3, it recommends decimal to be defined as a composite type that contains two subfields, mantissa and exponent.
Although it is obvious that how to generate decimal value from mantissa and exponent, the semantic meanings of both mantissa and exponent aren't defined in the specification. I wonder whether there is any other subfield that has a special meaning like mantissa and exponent.
Thanks in advance.
Thanks for pointing this out. It could be an inconsistency, i.e. the word "recommended" may be wrong in this place. 2.5.3 starts to say "Decimal encodings are composite types". I do not believe there are alternate , i.e. it is not only a recommendation on how to define decimal. The encoding specifications shown for decimal32 and decimal64 are at the same time an example as the number of decimal places is shown as 2 but could be different as long as it is a constant. We will seek to clarify the spec here.
Do you think we should explicitly define the semantics of the terms "mantissa" and "exponent" in the context of a decimal?
I think one of the US exchanges already interpreted that mantissa and exponent have specific semantic meanings even outside the scope of Decimal type. I interpret that that exchange basically assumes that any composite type with those 2 fields means mantissa x 10^exponent.
I don't see any harm to explicitly define the semantics of the terms "mantissa" and "exponent" in the context of a decimal.
For enumerations, the document does not state that valid values need to be unique or that their names need to be unique. At least the names should be unique within an enum. Does anyone think we should allow synonyms (different name with same value)?
It doesn't make sense to use the null value for integer types for a bitset field; presence="optional" is not logical for it. All bits can be off, of course. Also, we don't state that choices need to be unique within a bitset, but they should be.
On the subject of decimal encoding, we should define the members of the composite encoding, as suggested by H Chan. It seems that the term "mantissa" is discuouraged by IEEE and others in this context in favor of "significand" or "coefficient". We can define it as the part of the encoding that represents the significant digits and sign of a decimal number. Exponent represents scale as a power of 10.