Google Protocol Buffers mapping for FIX specification Release Candidate 1

,
1

This proposal entails the use of Google Protocol Buffers (GPB) technology to produce fast and compact encodings of FIX messages. The use of GPB is one of the three major approaches to the efficient encoding of FIX that have been developed and are being proposed by the High Performance Working Group—the others being the use of Abstract Syntax Notation 1 (ASN.1) and the use of a new, FIX-specific binary encoding called Simple Binary Encoding (SBE). The mapping to GPB defined in the proposed technical specification can be used for any FIX message (as defined in the FIX Unified Repository), and generates a set of GPB data structures. GPB data structures are defined as “messages” in a “.proto” template file. The template provides a machine-readable interface definition which is input to a language-specific code generator (protoc) to produce message encoders and decoders.

The gap analysis, the technical mapping proposal and a user guide can be found here:
https://www.fixtrading.org/standards/gpb

The proposed technical standard for Google Protocol Buffers mapping of FIX will following the technical standards review process described in the document found here:
https://www.fixtrading.org/fix-tech-std-lifecycle

This release candidate now enters a public comment period in which public review and feedback is encouraged. The public comment period will run for a period of 90 days beginning on Monday the 3rd of June.

Please post feedback, comments, and questions as replies to this discussion thread.

2

Joe Wood
4 June 2013 1:26pm

Are there any candidate schemas (.proto files) that can be reviewed alongside these documents?

Thanks

Joe Wood

3

Sara Rosen
4 June 2013 3:11pm

Are there any candidate schemas (.proto files) that can be reviewed alongside these documents?

Thanks

Joe Wood

See Appendix A of the “Encoding FIX Using Google Protocol Buffers Specification” for NewOrderSingle.proto, generated via an automated mapping from the FIX Unifiied Repository.

4

Joe Wood
5 June 2013 2:21pm

The proposed schemas in Appendix A of the specification break the Google Protocol Buffers style guidelines as described here:

https://developers.google.com/protocol-buffers/docs/style

For example: - the field names are not all lower-case separated by underscores, enumerated values are not all CAPS.

The result of this is that many code generator plugins fail to convert the identifiers to the correct conventions for target applications. This is currently breaking our tool chain.

Can the protobuf schemas be generated so that they adhere to the style standards and conventions?

5

Joe Wood
5 June 2013 2:39pm

The Protobuf schema does not have any associated mapping to the original FIX Tag. This makes FIX message and GBP message cross reference difficult.

Could additional meta-data be added to the schema to provide the cross reference FIX tag? This could be achieved using GBP ‘options’.

For example - a field extension would be defined as follows:

package fix;

extend google.protobuf.FieldOptions {
optional string tag = 56003;
}

And then used in the schema as follows:

message Instrument {
optional string symbol = 1 [(fix.tag)=“55”];
optional SecurityIDSource security_id_source = 3 [(fix.tag)=“22”];
}

6

Joe Wood
5 June 2013 4:01pm

The specification doesn’t appear to mention how the generated .proto files will be organized. Rather than using the FIX Repository convention of artifact type, could the data types be split into their respective categories?

For example, all messages associated with common category would go into a common.proto, market data messages and associated enums would go into market-data.proto etc…

This would allow the GPB schema to be used more modularly with explicit dependencies between the categories.

7

Joe Wood
5 June 2013 5:01pm

It appears that then schema in Appendix A (and examples througout the document) have all fields as ‘optional’ (i.e. there are no ‘required’ fields). Is this intentional or is the mapping from the repository still pending?

8

Joe Wood
5 June 2013 5:16pm

The annotations and comments from the FIX repository have not been generated into the .proto files as line comments.

Schemas with comments are useful because (as of GPB 2.5) the schema comments are now included in the data model. The comments are used to generate souce code level Java Doc, Doxygen, code completion comments etc… This would help to avoid field misinterpreation.

Could field, message and enum comments be included into the final generated .proto files?

9

Hanno Klein
5 June 2013 5:38pm

James,

this is intentional in the context of all binary encodings in high performance environments where we felt it may be wise to make required fields subject to bilateral agreement instead of mandating a small subset as the FIX Repository does for tag=value and FIXML today. I would expect any given implementation to provide a schema that defines a number of required fields.

The number of required fields in the FIX Standard has only been reduced over time because new use cases came up where the field cannot be provided. In high performance environments, one does not want to have to send these fields with dummy values. On the other hand, it will probably never make sense to send a NewOrderSingle without specifying an instrument. But take the OrderCancelRequest where OrderID(37) may or may not be sufficient from a technical point of view to find the order. Some may need the instrument because system order IDs are issued in parallel across multiple partitions while smaller systems may not. Some messages, e.g. ExecutionReport may be used as responses on the same session or as drop copies on separate sessions. The fields required in each context vary significantly. Responses do not need to echo request fields. Drop copies need to provide much more information to allow the recipient to process it.

Tag=value interfaces have nothing but the central FIX Repository, i.e. no meta-data. Additionally, parsing issues require the first field of every repeating group to be present. The new binary encodings do not have any of these limitations. Let’s discuss whether we need to define required fields or can leave it up to bilateral agreement. Thank you for raising this issue.

Regards,
Hanno.

It appears that then schema in Appendix A (and examples througout the document) have all fields as ‘optional’ (i.e. there are no ‘required’ fields). Is this intentional or is the mapping from the repository still pending?

10

Sara Rosen
5 June 2013 5:46pm

Please see chapter 4.1 of the “Encoding FIX Using Google Protocol Buffers User Guide”, where this recommendation is discussed.

Google recommends that all fields be marked as optional. This is especially important to support message evolution… Consequently, the GPB Encoding subgroup recommends that all FIX fields in the protobuf message definition be marked as optional. Semantically, fields can still be required, but enforcement of required message fields should be done on the application level, rather than by the GPB parser. Rules of engagement documents should be used to specify which fields are required in a given context. When such a “required” field is missing, the message should be rejected with an application or session level rejection message.

James,

this is intentional in the context of all binary encodings in high performance environments where we felt it may be wise to make required fields subject to bilateral agreement instead of mandating a small subset as the FIX Repository does for tag=value and FIXML today. I would expect any given implementation to provide a schema that defines a number of required fields.

The number of required fields in the FIX Standard has only been reduced over time because new use cases came up where the field cannot be provided. In high performance environments, one does not want to have to send these fields with dummy values. On the other hand, it will probably never make sense to send a NewOrderSingle without specifying an instrument. But take the OrderCancelRequest where OrderID(37) may or may not be sufficient from a technical point of view to find the order. Some may need the instrument because system order IDs are issued in parallel across multiple partitions while smaller systems may not. Some messages, e.g. ExecutionReport may be used as responses on the same session or as drop copies on separate sessions. The fields required in each context vary significantly. Responses do not need to echo request fields. Drop copies need to provide much more information to allow the recipient to process it.

Tag=value interfaces have nothing but the central FIX Repository, i.e. no meta-data. Additionally, parsing issues require the first field of every repeating group to be present. The new binary encodings do not have any of these limitations. Let’s discuss whether we need to define required fields or can leave it up to bilateral agreement. Thank you for raising this issue.

Regards,
Hanno.

It appears that then schema in Appendix A (and examples througout the document) have all fields as ‘optional’ (i.e. there are no ‘required’ fields). Is this intentional or is the mapping from the repository still pending?

11

Joe Wood
5 June 2013 6:29pm

I understand the intent around avoiding mandatory field, but I feel that making everything optional leaves too much for interpretation and significantly raises integration costs.

AFAIK having a field marked as required does not necessarily cause it to be carried on the wire (the generated field serialization and access code is the same). As part of this working group, I think it would be a good time to be more explicit about best behaviors for high performance message exchange. For example, tightening up requirements for issuing cancels and cancel/replaces will go a long way to help providers optimize their implementations in a high-performance environment. Having a specification that requires several different bilateral agreements does nothing but add complexity, which inevitably impacts performance.

Thanks

Joe

12

Joe Wood
5 June 2013 6:46pm

For field with ‘qty’ data types in FIX, the SDecimalE generated messages are being used in the .proto files.

Could the primitive floating point types be used here instead of message structures? For example, is a 64-bit mantissa really required where SDecimal64N0 is used over and above the regular IEEE double 53-bit mantissa?

Message structure fields would incur performance, message size and storage overhead. In the default C++ code generation, for example, any field with a structure type would incur an allocation.

Thanks

Joe Wood

13

Sara Rosen
6 June 2013 2:35pm

Encoding a FIX Qty as an IEEE float is an option, but is generally not recommended due to rounding issues. More likely a Qty is either an integer or a decimal value with a fixed number of decimal digits. When the number of decimal digits is pre-known, the recommended GPB mapping is an integer, where the precision is specified in metadata – either in a proto comment or in the rules of engagement document. A structure of type SDecimalE would be called for only when the number of decimal places must be specified on the wire, for example to provide flexibility to support either a large number of integral or decimal digits.

The GPB automated mapping described in the specification is based on a set of encoding attributes which annotate a FIX repository. Encoding attributes relevant to decimal datatypes are: minValue, maxValue, isFixedPoint, isBinaryFloat, and exponent. Other encoding attributes (epoch & timeUnit) are used to map timestamps to integer representations.

In the absence of empirical knowledge of how a particular FIX Qty field will be used, the default GPB mapping assumes a default encoding attribute of isFixedPoint=false, resulting in an SDecimalE0 structure. However, implementers are encouraged to use their domain knowledge to select the most efficient encoding which meets their needs. If the Qty field is indeed an integer or a decimal with a fixed number of decimal points, then isFixedPoint=true may be specified, which would generate either an sint32 or sint64 (depending on the value of the maxValue encoding attribute). To generate a true IEEE 574 float, the encoding attribute isBinaryFloat may be set.

In short, due to the variability of FIX usage, there can be no optimal one-size-fits-all binary mapping. Rather, a mechanism is provided for repository annotations which drive the automated mapping of a particular FIX field. The default settings are by necessity the most general purpose, and therefore are not necessarily the optimal representation. By adding domain knowledge to a FIX repository, optimized mappings can be automated.

Sara Rosen

For field with ‘qty’ data types in FIX, the SDecimalE generated messages are being used in the .proto files.

Could the primitive floating point types be used here instead of message structures? For example, is a 64-bit mantissa really required where SDecimal64N0 is used over and above the regular IEEE double 53-bit mantissa?

Message structure fields would incur performance, message size and storage overhead. In the default C++ code generation, for example, any field with a structure type would incur an allocation.

Thanks

Joe Wood

14

Joe Wood
6 June 2013 7:17pm

Thanks Sara

If a per field custom, non-variable fixed point model is preferable, then I would recommend using a field option (GPB’s extensible meta-data). A comment on the field is not ideal, for obvious reasons. Or better yet, use one of GPB’s fixed point primitive types.

If the exponent is variable then a primitive double would be preferred way to go. If it’s dependent on usage and must be manually specified then I would recommend an additional optional field at the same level with a default value. Although, I find it hard to understand the scenario where the exponent on a floating point number needs to be manually specified.

The use of nested sub-structures to model numeric quantities seems to go against the goals of a high-performance protocol.

Also, having different implementers annotate the schema for their own usage will hamper interoperability. Uncovering these serialization problems will be very difficult to debug. There must be a better solution to this, like through basic schema overrides that I described above.

Thanks

Joe

15

Chris Busbey
18 July 2013 5:00pm

The proposed schemas in Appendix A of the specification break the Google Protocol Buffers style guidelines as described here:

Style Guide | Protocol Buffers Documentation

For example: - the field names are not all lower-case separated by underscores, enumerated values are not all CAPS.

The result of this is that many code generator plugins fail to convert the identifiers to the correct conventions for target applications. This is currently breaking our tool chain.

Can the protobuf schemas be generated so that they adhere to the style standards and conventions?

16

Chris Busbey
18 July 2013 5:05pm

Is this an area that should be covered by the spec? It would be useful to have some extensible range of field values so that the base specs can by extended by 3rd parties.

17

Chris Busbey
18 July 2013 5:08pm

The example in 4.1.3 has a couple of typos in the StandardHeader message

  • mesaage vs message
  • duplicate tags for 5
18

Chris Busbey
18 July 2013 5:18pm

From 5.1.2.3

“”“
5.1.2.3 The enumerator values of the GPB enum type shall be generated from the position of each of the
child elements of the element starting from 0.
”""

This seems like an annoying translation detail. It depends on the structured orderings of the enums of the field element. So for example,

notReqXML=“0”>

would generate :

enum AdvSideEnum {
AdvSide_BUY = 0;
AdvSide_SELL = 1;
}

but

notReqXML=“0”>

would generate :

enum AdvSideEnum {
AdvSide_SELL = 0;
AdvSide_BUY = 1;
}

Perhaps a better convention would be to sort the enum children by value, and then assigning enum tag values based on that ordering.

In any case, the spec breaks with this convention in Example 1 of 5.1.4.4.

My understanding of 5.1.2.3 implies that PosMaintResult_OTHER should be 2, not 99.

19

Greg Malatestinic
18 July 2013 5:31pm

+1

The proposed schemas in Appendix A of the specification break the Google Protocol Buffers style guidelines as described here:

Style Guide | Protocol Buffers Documentation

For example: - the field names are not all lower-case separated by underscores, enumerated values are not all CAPS.

The result of this is that many code generator plugins fail to convert the identifiers to the correct conventions for target applications. This is currently breaking our tool chain.

Can the protobuf schemas be generated so that they adhere to the style standards and conventions?

20

Joe Wood
18 July 2013 5:40pm

Hi Chris

Unfortunately, if you sort the enums by value the sort order would not be maintained through versions (e.g. if ‘C’ was added it should shift the enum index values down and ripple through). This would break backward compatibility.

Generally speaking, the sort order would be based on when the field or values are added to the schema - e.g. FIX version, SP then EP (via the ‘added’ attribute in the FIX repo).

Thanks

Joe Wood

From 5.1.2.3

“”"
5.1.2.3 The enumerator values of the GPB enum type shall be generated from the position of each of the
child elements of the element starting from 0.
“”"

This seems like an annoying translation detail. It depends on the structured orderings of the enums of the field element. So for example,

notReqXML=“0”>

would generate :

enum AdvSideEnum {
AdvSide_BUY = 0;
AdvSide_SELL = 1;
}

but

notReqXML=“0”>

would generate :

enum AdvSideEnum {
AdvSide_SELL = 0;
AdvSide_BUY = 1;
}

Perhaps a better convention would be to sort the enum children by value, and then assigning enum tag values based on that ordering.

In any case, the spec breaks with this convention in Example 1 of 5.1.4.4.

My understanding of 5.1.2.3 implies that PosMaintResult_OTHER should be 2, not 99.