FAST protocol for market data adapters

, ,
1

Imported from previous forum

2

I am developing an automated trading system and one of the issues I have is about sourcing market data from several vendors such as Reuters RMDS etc. Instead of introducing a custom format I am considering using FAST protocol as backbone of market data for my system and then build market data adapters that can convert custom formats from different vendors into a format of FAST protocol.

To begin with I am looking for Reuters RMDS to FAST conversion. If anyone has experience in this area please advise me.

Best regards,
Pravin

3

Reuters RMDS is likely to have a different semantic than FIX. FAST is just the encoding piece, i.e. it should be an RMDS to FIX conversion whereby FIX is encoded in FAST and not in tag=value. Converting one encoding to another is usually not a problem compared to the conversion of semantics, i.e. how market data is expressed in terms of messages, fields and valid values. It is definitely a good idea to use FIX as your backbone but it is not trivial to plug in market data sources that do not already use FIX semantics like CME or Deutsche Börse Group do.
Regards,
Hanno.

I am developing an automated trading system and one of the issues I have is about sourcing market data from several vendors such as Reuters RMDS etc. Instead of introducing a custom format I am considering using FAST protocol as backbone of market data for my system and then build market data adapters that can convert custom formats from different vendors into a format of FAST protocol.

To begin with I am looking for Reuters RMDS to FAST conversion. If anyone has experience in this area please advise me.

Best regards,
Pravin

4

Hi Pravin,

RMDS has a different set of semantics compared to FIX.

As Hanno writes, it is definitely a good idea to use a standard way of expressing your market data flows. FIX is one alternative with its strengths and weaknesses. It will save you a lot of work in defining the meaning (semantics) of fields and messages , but FIX has some limitations when you want to express market data from non-FIX sources.

Also, the current set of FIX market data messages are far from optimal. We are working to improve this situation, but it will likely take some time before there is a new proposal.

Most market data vendors (including Reuters) use proprietary formats internally. This makes sense as long as you are not going to make your internal protocol public. One alternative is to base your protocol on FIX, but add whatever extensions that you need in order to conveniently implement your internal work flows.

One aspect is if you are planning to use third party software, in which case you may want to stick closer to someone’s “standard” protocol. RMDS is certainly one of those de-facto “standards”. It would be entirely possible to run an RMDS style app layer on top of FAST, although I’m not entirely convinced that FAST is the ideal encoding for internal communication, given that today you would prefer low codec overhead over compactness. The story is obviously different if you want to move info over networks with limited bandwidth. FAST is very effective in this usage scenario, although there are other alternatives in this space as well.

Yet another alternative could be to run RMDS over GPB (Google Protocol Buffers), but again GPB has its own set of strengths and weaknesses. There are many possible encodings available out there. I would suggest that you review and compare different alternatives. There is also some interesting work going on in the High Performance working group although this work is primarily oriented towards transaction (orders, trades) flows.

Hope this helps. Please don’t hesitate to post follow-up questions and comments.

Best,
Rolf

5

Thanks Rolf and Hanno. I understand that FAST is for encoding and both FIX or custom semantics over something like GPB have their limitations which I need to consider. I could probably borrow semantics from FIX and implement something over GBP or Thrift if that makes things faster.

Best regards,
Pravin

Hi Pravin,

RMDS has a different set of semantics compared to FIX.

As Hanno writes, it is definitely a good idea to use a standard way of expressing your market data flows. FIX is one alternative with its strengths and weaknesses. It will save you a lot of work in defining the meaning (semantics) of fields and messages , but FIX has some limitations when you want to express market data from non-FIX sources.

Also, the current set of FIX market data messages are far from optimal. We are working to improve this situation, but it will likely take some time before there is a new proposal.

Most market data vendors (including Reuters) use proprietary formats internally. This makes sense as long as you are not going to make your internal protocol public. One alternative is to base your protocol on FIX, but add whatever extensions that you need in order to conveniently implement your internal work flows.

One aspect is if you are planning to use third party software, in which case you may want to stick closer to someone’s “standard” protocol. RMDS is certainly one of those de-facto “standards”. It would be entirely possible to run an RMDS style app layer on top of FAST, although I’m not entirely convinced that FAST is the ideal encoding for internal communication, given that today you would prefer low codec overhead over compactness. The story is obviously different if you want to move info over networks with limited bandwidth. FAST is very effective in this usage scenario, although there are other alternatives in this space as well.

Yet another alternative could be to run RMDS over GPB (Google Protocol Buffers), but again GPB has its own set of strengths and weaknesses. There are many possible encodings available out there. I would suggest that you review and compare different alternatives. There is also some interesting work going on in the High Performance working group although this work is primarily oriented towards transaction (orders, trades) flows.

Hope this helps. Please don’t hesitate to post follow-up questions and comments.

Best,
Rolf

6

You may also want to take a look at ASN.1, which is one of the efficient encodings for FIX proposed by the High Performance Working Group, along with Simple Binary and Protocol Buffers:

http://www.fixprotocol.org/discuss/read/49a7096e.

Alessandro Triglia
OSS Nokalva

Thanks Rolf and Hanno. I understand that FAST is for encoding and both FIX or custom semantics over something like GPB have their limitations which I need to consider. I could probably borrow semantics from FIX and implement something over GBP or Thrift if that makes things faster.

7

One advantage of ASN.1 is that it has several different encoding rules that can be used depending on your preferences re compactness and encoding overhead.

You may also want to take a look at ASN.1, which is one of the efficient encodings for FIX proposed by the High Performance Working Group, along with Simple Binary and Protocol Buffers:

http://www.fixprotocol.org/discuss/read/49a7096e.

Alessandro Triglia
OSS Nokalva

Thanks Rolf and Hanno. I understand that FAST is for encoding and both FIX or custom semantics over something like GPB have their limitations which I need to consider. I could probably borrow semantics from FIX and implement something over GBP or Thrift if that makes things faster.

8

btw, which language environment (C++, Java, C#, …) are you primarily using?

Thanks Rolf and Hanno. I understand that FAST is for encoding and both FIX or custom semantics over something like GPB have their limitations which I need to consider. I could probably borrow semantics from FIX and implement something over GBP or Thrift if that makes things faster.

9

I am using C#. My potential clients are small and mid size players and prefer windows environment.

Best regards,
Pravin

btw, which language environment (C++, Java, C#, …) are you primarily using?

Thanks Rolf and Hanno. I understand that FAST is for encoding and both FIX or custom semantics over something like GPB have their limitations which I need to consider. I could probably borrow semantics from FIX and implement something over GBP or Thrift if that makes things faster.

10

There is no inherent limitation of encoding market data over google protocol buffers, as for any of the three binary encodings adopted by the FIX High Performance Working Group (SBE-Simple Binary Encoding, GPB, and ASN.1).
All provide binary serialization of FIX semantics, and are agnostic to their content.

The three encodings support various levels of the following features, which should be assessed to determine which best meets your needs:

  • speed
  • compression
  • ease of use
  • support for optional fields
  • support for backward compatibility
  • open source / proprietary
  • availability of encoder/decoder in your choice of language
  • maturity of solution

A full description of the three binary encodings specified by the FPL High Performance Working Group can be found at:
http://www.fixprotocol.org/encodings

Note that these are Release Candidate 1 specifications, still subject to change based on community feedback.

Protocol buffer implementations for C# can be found at:

I am using C#. My potential clients are small and mid size players and prefer windows environment.

Best regards,
Pravin

btw, which language environment (C++, Java, C#, …) are you primarily using?

Thanks Rolf and Hanno. I understand that FAST is for encoding and both FIX or custom semantics over something like GPB have their limitations which I need to consider. I could probably borrow semantics from FIX and implement something over GBP or Thrift if that makes things faster.

11

A good feature summary.

Speed, size - we need empirical results for some relevant data sets.
Ease of use - I would submit that this is a function of the codec software available.
Optional fields - The SBE is weaker than both GPB and ASN.1, and GPB has the best support for messages with many optional fields.
Backward compatibility - GPB has some features that are supposed to support phased migration. I would argue that both SBE and ASN could support backward compatibility using software with a two-schema model.
Open source - all of the HPWG protocols are open, but only GPB has a strong base of open implementations.
Language bindings - again, GPB has the strongest support.
Maturity of solution - not sure how this can/should be measured.

Best,
Rolf

There is no inherent limitation of encoding market data over google protocol buffers, as for any of the three binary encodings adopted by the FIX High Performance Working Group (SBE-Simple Binary Encoding, GPB, and ASN.1).
All provide binary serialization of FIX semantics, and are agnostic to their content.

The three encodings support various levels of the following features, which should be assessed to determine which best meets your needs:

  • speed
  • compression
  • ease of use
  • support for optional fields
  • support for backward compatibility
  • open source / proprietary
  • availability of encoder/decoder in your choice of language
  • maturity of solution
12

I would say that GPB and ASN.1 have a similar degree of support for backward compatibility. ASN.1 has a specific feature called “extensibility” that allows new fields to be added to a message or component, new enumerators to be added to an enumerated type, and so on. If the author of a schema uses this feature and abides by certain rules when creating new versions of the schema, an application built around “version N” of an ASN.1 schema will be able to decode a message that was encoded by an application implementing an earlier version of the schema or a later version of the schema. Of course, an older recipient may not understand the meaning of the new material found in the message, but it will still be able to fully decode the message. This feature is supported even when using the most compact binary encoding rules (PER Unaligned), and is one of the key features of ASN.1.

A good feature summary.

Speed, size - we need empirical results for some relevant data sets.
Ease of use - I would submit that this is a function of the codec software available.
Optional fields - The SBE is weaker than both GPB and ASN.1, and GPB has the best support for messages with many optional fields.
Backward compatibility - GPB has some features that are supposed to support phased migration. I would argue that both SBE and ASN could support backward compatibility using software with a two-schema model.
Open source - all of the HPWG protocols are open, but only GPB has a strong base of open implementations.
Language bindings - again, GPB has the strongest support.
Maturity of solution - not sure how this can/should be measured.

Best,
Rolf

There is no inherent limitation of encoding market data over google protocol buffers, as for any of the three binary encodings adopted by the FIX High Performance Working Group (SBE-Simple Binary Encoding, GPB, and ASN.1).
All provide binary serialization of FIX semantics, and are agnostic to their content.

The three encodings support various levels of the following features, which should be assessed to determine which best meets your needs:

  • speed
  • compression
  • ease of use
  • support for optional fields
  • support for backward compatibility
  • open source / proprietary
  • availability of encoder/decoder in your choice of language
  • maturity of solution
13

I believe the ASN support for backward compatibility is implemented through the codec libraries and schemas being exchanged between the communicating parties, whereas as the GPB wire encoding includes some meta-data that supports version compatibility. GPB supports partial backward compatibility without exchanging the new schema version.

I would say that GPB and ASN.1 have a similar degree of support for backward compatibility. ASN.1 has a specific feature called “extensibility” that allows new fields to be added to a message or component, new enumerators to be added to an enumerated type, and so on. If the author of a schema uses this feature and abides by certain rules when creating new versions of the schema, an application built around “version N” of an ASN.1 schema will be able to decode a message that was encoded by an application implementing an earlier version of the schema or a later version of the schema. Of course, an older recipient may not understand the meaning of the new material found in the message, but it will still be able to fully decode the message. This feature is supported even when using the most compact binary encoding rules (PER Unaligned), and is one of the key features of ASN.1.

14

In ASN.1, each endpoint can work with a different version of the schema. There is no need for an endpoint to obtain the newer (or the older) version which is used by the other endpoint. This is a characteristic of the ASN.1 language. Obviously this requires that the author of the schema followed certain rules when he created each new version of the schema. The “extensibility” feature of ASN.1 is based on the use of a special “extension marker” that indicates the point where new fields can be added to a SEQUENCE, new enumerators can be added to an ENUMERATED type, and so on. For example, if version 2 of a schema has some new fields after the extension marker of a SEQUENCE that is nested at any depth within the message, a message that utilizes the new fields will be encoded in such a way that an older recipient will be able to decode the message. Obviously the old recipient will not be able to decode the new fields, but the key point is that the rest of the message will be decoded normally. Unknown enumerators added after the extension marker in the ENUMERATED type definition can be skipped or decoded (as numbers) even though the recipient will not understand what they mean. The same applies, for example, to an INTEGER with a range that contains an extension marker. A later version of the schema can extend the range (by changing the part after the extension marker), and an old recipient will be able to decode new values within the extended range. The recipient does not need to know the “version number” of the schema nor to get the new schema. Obviously there are restrictions that must be followed. For example, an extension marker cannot be added if it was not already there since version 1, and it cannot be moved. The first extension must be at the extension marker, and each subsequent extension must follow the previous extensions. All of the above is inherent to ASN.1, it’s not a feature of a particular tool.

I believe the ASN support for backward compatibility is implemented through the codec libraries and schemas being exchanged between the communicating parties, whereas as the GPB wire encoding includes some meta-data that supports version compatibility. GPB supports partial backward compatibility without exchanging the new schema version.

I would say that GPB and ASN.1 have a similar degree of support for backward compatibility. ASN.1 has a specific feature called “extensibility” that allows new fields to be added to a message or component, new enumerators to be added to an enumerated type, and so on. If the author of a schema uses this feature and abides by certain rules when creating new versions of the schema, an application built around “version N” of an ASN.1 schema will be able to decode a message that was encoded by an application implementing an earlier version of the schema or a later version of the schema. Of course, an older recipient may not understand the meaning of the new material found in the message, but it will still be able to fully decode the message. This feature is supported even when using the most compact binary encoding rules (PER Unaligned), and is one of the key features of ASN.1.

15

yes, when re-reading my own post I realize that it was close to nonsense. apologies.

I was (implicitly) referring to the encoding chosen for the comparative tests performed within HPWG.

An alternative wording of my previous post could be:
ASN support for backward compatibility can be implemented through the codec libraries and schemas being exchanged between the communicating parties, whereas the GPB wire encoding always includes some meta-data that supports version compatibility.

The essence of this is:

  • GPB doesn’t define a meta-data less encoding, so there is an encoding and space overhead even though there is no need for backward compatibility.
  • ASN allows you to use an alternate model where the wire encoding is both more compact and efficient to encode and decode. At the same time it is entirely possible to support backward compatibility through codecs with multiple schema support.

The multi-schema approach could be used in SBE if a schema transfer encoding is defined.
This approach is also used in Blink and possibly in some other protocols as well.

To be clear; I think it is a weakness in GPB that there is no way to exclude meta-data. It limits the coding efficiency and therefore limits the applicability of GPB.

As Sara wrote in her post; “The three encodings support various levels of the following features, which should be assessed to determine which best meets your needs”

GPB is not the highest performing alternative because of the design trade-offs that were made by the GPB dev team.

There could be ways of relaxing the decoding assumptions made in GPB, but then the encoding would not be standard GPB. This has been discussed previously on CDT if I recall correctly.

/Rolf

In ASN.1, each endpoint can work with a different version of the schema. There is no need for an endpoint to obtain the newer (or the older) version which is used by the other endpoint. This is a characteristic of the ASN.1 language. Obviously this requires that the author of the schema followed certain rules when he created each new version of the schema.

16

Apologies for digressing but I found openMAMDA on the net. http://www.openmama.org/what-is-openmamda

They appear to be working on a vendor neutral market data API. Anyone has experience with this? Is it different to what FIX is trying to achieve?

Best regards,
Pravin

yes, when re-reading my own post I realize that it was close to nonsense. apologies.

I was (implicitly) referring to the encoding chosen for the comparative tests performed within HPWG.

An alternative wording of my previous post could be:
ASN support for backward compatibility can be implemented through the codec libraries and schemas being exchanged between the communicating parties, whereas the GPB wire encoding always includes some meta-data that supports version compatibility.

The essence of this is:

  • GPB doesn’t define a meta-data less encoding, so there is an encoding and space overhead even though there is no need for backward compatibility.
  • ASN allows you to use an alternate model where the wire encoding is both more compact and efficient to encode and decode. At the same time it is entirely possible to support backward compatibility through codecs with multiple schema support.

The multi-schema approach could be used in SBE if a schema transfer encoding is defined.
This approach is also used in Blink and possibly in some other protocols as well.

To be clear; I think it is a weakness in GPB that there is no way to exclude meta-data. It limits the coding efficiency and therefore limits the applicability of GPB.

As Sara wrote in her post; “The three encodings support various levels of the following features, which should be assessed to determine which best meets your needs”

GPB is not the highest performing alternative because of the design trade-offs that were made by the GPB dev team.

There could be ways of relaxing the decoding assumptions made in GPB, but then the encoding would not be standard GPB. This has been discussed previously on CDT if I recall correctly.

/Rolf

In ASN.1, each endpoint can work with a different version of the schema. There is no need for an endpoint to obtain the newer (or the older) version which is used by the other endpoint. This is a characteristic of the ASN.1 language. Obviously this requires that the author of the schema followed certain rules when he created each new version of the schema.

17

Hi Pravin,
OpenMAMA is tacking interoperability from a different angle, but could be used to leverage any standards that come out of FPL.

OpenMAMA is an open source framework which allows different middlewares, market data protocols and message encoding protocols to be supported though a common API.

Different message encodings are abstracted from application developers which allows them to switch protocols without modifying their applications. Multiple message formats can also be supported concurrently.

In this way, no matter what FPL message encoding protocol ‘wins’, if OpenMAMA supports supports that protocol then developers will not need to modify their applications.

FYI there’s already a project to support GPB in OpenMAMA, though I haven’t used it myself.
https://github.com/tgjonesuk/mama_protobuf

Regards,
Glenn

Apologies for digressing but I found openMAMDA on the net. http://www.openmama.org/what-is-openmamda

They appear to be working on a vendor neutral market data API. Anyone has experience with this? Is it different to what FIX is trying to achieve?

Best regards,
Pravin

18

Let me further clarify one point, which I am not sure was clear enough in what I wrote before: The extensibility feature of ASN.1 works with all the standard encoding rules, including BER, PER, and OER. There is no trade-off between encoding efficiency and extensibility, provided that the extension marker is used correctly across versions of the schema.

From a purely encoding-related perspective, BER is more tolerant of arbitrary changes to the schema–including some changes that violate the rules of ASN.1 extensibility across versions. In contrast, PER and OER require that those rules are followed strictly, otherwise the new version will not be backward-compatible.

yes, when re-reading my own post I realize that it was close to nonsense. apologies.

I was (implicitly) referring to the encoding chosen for the comparative tests performed within HPWG.

An alternative wording of my previous post could be:
ASN support for backward compatibility can be implemented through the codec libraries and schemas being exchanged between the communicating parties, whereas the GPB wire encoding always includes some meta-data that supports version compatibility.

The essence of this is:

  • GPB doesn’t define a meta-data less encoding, so there is an encoding and space overhead even though there is no need for backward compatibility.
  • ASN allows you to use an alternate model where the wire encoding is both more compact and efficient to encode and decode. At the same time it is entirely possible to support backward compatibility through codecs with multiple schema support.

The multi-schema approach could be used in SBE if a schema transfer encoding is defined.
This approach is also used in Blink and possibly in some other protocols as well.

To be clear; I think it is a weakness in GPB that there is no way to exclude meta-data. It limits the coding efficiency and therefore limits the applicability of GPB.

As Sara wrote in her post; “The three encodings support various levels of the following features, which should be assessed to determine which best meets your needs”

GPB is not the highest performing alternative because of the design trade-offs that were made by the GPB dev team.

There could be ways of relaxing the decoding assumptions made in GPB, but then the encoding would not be standard GPB. This has been discussed previously on CDT if I recall correctly.

/Rolf

In ASN.1, each endpoint can work with a different version of the schema. There is no need for an endpoint to obtain the newer (or the older) version which is used by the other endpoint. This is a characteristic of the ASN.1 language. Obviously this requires that the author of the schema followed certain rules when he created each new version of the schema.

19

Hm, how can extensibility be implemented using extension markers without adding some overhead?
I have understood the extension marker mechanism as follows:

  1. The schema will contain information about the presence of one or more extension markers.
  2. The encoder will either encode an extension or indicate using some kind of null marker that there is no extension present at a certain marker location (extension point). The null marker may just be absent for some encodings.
  3. The marker is embedded in the encoded wire data, or absent if the format allows detection of absent extension markers.
  4. The decoder detects the presence of an extension marker or reads and interprets the marker value to see if there is an extension present.

I realize that BER may have this embedded in its standard encoding, but as I have understood your description of OER (that was used in the tests), as well as PER, there are no markers in the stream unless you specify that in the schema.

Am I missing something here?

/Rolf

Let me further clarify one point, which I am not sure was clear enough in what I wrote before: The extensibility feature of ASN.1 works with all the standard encoding rules, including BER, PER, and OER. There is no trade-off between encoding efficiency and extensibility, provided that the extension marker is used correctly across versions of the schema.

From a purely encoding-related perspective, BER is more tolerant of arbitrary changes to the schema–including some changes that violate the rules of ASN.1 extensibility across versions. In contrast, PER and OER require that those rules are followed strictly, otherwise the new version will not be backward-compatible.

20

Your description is essentially correct.

Each encoding rule has its own way of dealing with extensibility.

In BER, the presence of extension markers in the schema is not reflected in the encodings.

In PER, an extensible type (i.e., a SEQUENCE, CHOICE, ENUMERATED, INTEGER, etc., type containing an extension marker) is encoded with an additional bit at the very beginning of the encoding of the value. That bit is called the “extension bit”. Any value that belongs to the original (unextended) type (as in version 1) is encoded with the extension bit set to zero. Any value that belongs to an extension (added in a subsequent version) is encoded with the extension bit set to one. When the extension bit is set to zero, the rest of the encoding of the value is identical to what it would be if there were no extension marker in the type definition. When the extension bit is set to one, the encoding is more verbose and such that it can be partially decoded by someone that does not know the type definition in the new version. Values added in subsequent versions of the schema are, in general, less efficient to encode and decode.

In OER, some extensible types are encoded as in PER (i.e., with an extension bit), and others don’t need an extension bit. For example, an integer with an unrestricted range is always encoded as a variable-length integer (length plus value octets) whether it is extensible or not. Enumerated types are encoded in the same way whether they are extensible or not.

You could say that the presence of an extension bit supporting extensibility adds some overhead (even in version 1) just because that bit needs to be read and tested. However, in most cases one doesn’t need to include many extension markers in a message definition. A few extension markers at critical locations will usually suffice, so the related overhead (in version 1) can be kept small.

Hm, how can extensibility be implemented using extension markers without adding some overhead?
I have understood the extension marker mechanism as follows:

  1. The schema will contain information about the presence of one or more extension markers.
  2. The encoder will either encode an extension or indicate using some kind of null marker that there is no extension present at a certain marker location (extension point). The null marker may just be absent for some encodings.
  3. The marker is embedded in the encoded wire data, or absent if the format allows detection of absent extension markers.
  4. The decoder detects the presence of an extension marker or reads and interprets the marker value to see if there is an extension present.

I realize that BER may have this embedded in its standard encoding, but as I have understood your description of OER (that was used in the tests), as well as PER, there are no markers in the stream unless you specify that in the schema.

Am I missing something here?

/Rolf

Let me further clarify one point, which I am not sure was clear enough in what I wrote before: The extensibility feature of ASN.1 works with all the standard encoding rules, including BER, PER, and OER. There is no trade-off between encoding efficiency and extensibility, provided that the extension marker is used correctly across versions of the schema.

From a purely encoding-related perspective, BER is more tolerant of arbitrary changes to the schema–including some changes that violate the rules of ASN.1 extensibility across versions. In contrast, PER and OER require that those rules are followed strictly, otherwise the new version will not be backward-compatible.