Imported from previous forum
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
I’m not altogether clear on the intended usage of application type by the FAST specification. Here’s what the specification tells us:
- the FAST stream is encoded from and decoded to an application type (section 2)
- application types have a particular data model (section 5)
- templates are associated with application types and there can be more than one template per application type (section 6)
- application type is changed by the td:typeRef element which can show up as the first element in td:template, td:group, td:sequence (section 6.1)
- field instruction names correspond to fields in the application type (section 6.2)
- the sequence field instruction corresponds to a sequence application type (section 6.2.5)
- the group field instruction does not suggest a group application type (section 6.2.6)
The specification does not clearly address the significance of the application type. I’ll address my questions specifically to FIX with the understanding that FAST is meant to support a much wider range of protocols. If a particular feature is not used for FIX, I’d appreciate if you could specify the use case the feature intends to address.
- td:typeRef located in td:template - one can infer that the usage of typeRef in the template element context can be used to specify a message type - information otherwise not carried in the template.
1a. What is the format of this message type? Is it the enumerated value or the descriptive name? While it is not the responsibility of the FAST specification to specify FIX usage, this needs to be specified somewhere to ensure interoperability between different implementations of FIX over FAST. I would think that one should be able to drop any set of templates defined in the normative xml syntax (for those implementations that chose to support it) into any implementation and expect a well-defined behavior. This can only be accomplished if the format of application type is specified.
1b. The specification stresses that one application type can be represented by more than one template. How does one communicate in the normative xml syntax that a particular template should be chosen for a particular message? If there is only one template per message type, this is straightforward. But what of the case where there are a number of templates for the same message type? One can concoct schemes using different application types, but what about the interoperability I mentioned above?
1c. Is it an error if a top-level (as opposed to a referenced) template does not specify a typeRef element?
1d. Are there other visions for application type usage in the td:template element?
-
td:typeRef located in td:group - what information is this meant to provide beyond the name of the group specified by the name attribute on td:group?
-
td:typeRef located in td:group
3a. what information is this meant to provide beyond the name of the sequence specified by the name attribute on td:sequence?
3b. Section 5 maintains that a sequence can contain multiple group types, resulting in a heterogeneous sequence at the application level. Can you supply a simple template that illustrates this?
Steven Grossman
NYSE TransactTools
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
correction - #3 should read:
- td:typeRef located in td:sequence
- td:typeRef located in td:template - one can infer that the
usage of typeRef in the template element context can be used to
specify a message type - information otherwise not carried in the
template.
Yes, the primary purpose of a td:typeRef appearing in a td:template is to name the type the template is a template for. This is useful since templates do not constitute types them selves. This method of associating an application type with a template is independent of the application protocol (for example FIX), but is not the only way to make this association. Some protocols, like FIX, send the application type in an ordinary
field, MsgType/35. So in the FIX case it would be possible (but not recommended) to do without the td:typeRef element provided that you
always explicitly specify the MsgType field in your template. The downside here would be that every piece of software wanting to associate a template with a type would have to know the specifics of the application protocol.
On such piece of software is the FAST encoder/decoder it self: It is possible to specify that an operator should share the same previous value among instances of a particular application type regardless of the template used. Another related use of this feature come in handy when static template references are used to express reusable blocks of fields:
... ...In this example, the copy operator of the MsgType uses an entry in a dictionary specific to the current application type. The effect is that MsgType will be transmitted once for each type of message. Subsequence messages will simply copy the previous value, which is different for each application type.
1a. What is the format of this message type? Is it the enumerated
value or the descriptive name? While it is not the responsibility of
the FAST specification to specify FIX usage, this needs to be
specified somewhere to ensure interoperability between different
implementations of FIX over FAST. I would think that one should be
able to drop any set of templates defined in the normative xml syntax
(for those implementations that chose to support it) into any
implementation and expect a well-defined behavior. This can only be
accomplished if the format of application type is specified.
The rules for naming and use of namespaces for application types must be specified for each application protocol, like FIX. FPL should produce a FIX-over-FAST specification containing this kind of information. Currently there is no such standard document.
1b. The specification stresses that one application type can be
represented by more than one template. How does one communicate in
the normative xml syntax that a particular template should be chosen
for a particular message? If there is only one template per message
type, this is straightforward. But what of the case where there are a
number of templates for the same message type? One can concoct
schemes using different application types, but what about the
interoperability I mentioned above?
The template selection rules cannot be expressed in the template syntax but is a feature of a particular encoder implementation. Interoperability is not a problem here as long as you have your typeRefs in order. Provided that a decoder does not make any assumptions about a particular template name or id (it shouldn’t), but always use the typeRef when communicating with the application it does not matter which template the encoder selects.
1c. Is it an error if a top-level (as opposed to a referenced)
template does not specify a typeRef element?
No. In some situations it may prove practical to use a single template for several application types, a.k.a a chameleon template, but in that case you have a more involved protocol where the actual application type represented is encoded in one of the fields and the decoder must know this.
1d. Are there other visions for application type usage in the
td:template element?
- td:typeRef located in td:group - what information is this
meant to provide beyond the name of the group specified by the name
attribute on td:group?
In FIX the only use would be together with the dictionary=“type” on operators. For other protocols, with dynamic type binding, it could be used to indicate the actual type used by a particular template.
- td:typeRef located in td:sequence
3a. what information is this meant to provide beyond the name of the
sequence specified by the name attribute on td:sequence?
See the group case
3b. Section 5 maintains that a sequence can contain multiple group
types, resulting in a heterogeneous sequence at the application
level. Can you supply a simple template that illustrates this?
A heterogeneous sequence in a single template requires a dynamic template reference. By using a typeRef in a sequence, all elements will have the same type, so the instance is not heterogeneous per se, but different templates for the same message and sequence could specify different element types.
/David
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
I appreciate your clear response. Allow me to take one step back.
You say that “template selection rules cannot be expressed in the template syntax but is a feature of a particular encoder implementation.”
I’m understanding that the FAST specification deliberately chose not to address encoder template selection (the process by which an encoder chooses a particular template to use for marshaling a particular application message to FAST bytes) to allow each implementation to do it its own way. Why? This prevents a set of templates written for one encoder from behaving the same way on another encoder (without additional implementation-specific configuration). More concretely, I (a template author) would want the ability to control template selection in an implementation-neutral way. Why doesn’t the FAST specification demand conformance here?
Steven
why do you think the FAST spec should demand conformance?
/Rolf
I appreciate your clear response. Allow me to take one step back.
You say that “template selection rules cannot be expressed in the
template syntax but is a feature of a particular encoder
implementation.”I’m understanding that the FAST specification deliberately chose not to
address encoder template selection (the process by which an encoder
chooses a particular template to use for marshaling a particular
application message to FAST bytes) to allow each implementation to do it
its own way. Why? This prevents a set of templates written for one
encoder from behaving the same way on another encoder (without
additional implementation-specific configuration). More concretely, I (a
template author) would want the ability to control template selection in
an implementation-neutral way. Why doesn’t the FAST specification demand
conformance here?Steven
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
When authoring a set of templates, one contemplates the most efficient way to serialize each and every application message. Some templates are meant to be used for multiple messages, while other templates are specifically designed for an individual message format. The template author needs a means to inform the encoder of his intentions - the mapping between application message and template.
Ideally, this mapping should be standardized by the specification. This would allow one to author a set of templates and their corresponding mappings to application messages once and to execute them on any FAST implementation.
Conformance permits me to replace one implementation with another with relative ease - I don’t have to redefine my mappings. I can bake off two different implementations using the same configuration (templates and mappings). I don’t have to learn implementation-specific semantics; I can rely on the FAST specification …
The argument for conformance is pretty common in the software space due to companies trying to avoid vendor-lockin. What’s the argument for not demanding conformance?
Steven
why do you think the FAST spec should demand conformance?
/Rolf
I appreciate your clear response. Allow me to take one step back.
You say that “template selection rules cannot be expressed in the
template syntax but is a feature of a particular encoder
implementation.”I’m understanding that the FAST specification deliberately chose not
to address encoder template selection (the process by which an encoder
chooses a particular template to use for marshaling a particular
application message to FAST bytes) to allow each implementation to do
it its own way. Why? This prevents a set of templates written for one
encoder from behaving the same way on another encoder (without
additional implementation-specific configuration). More concretely, I
(a template author) would want the ability to control template
selection in an implementation-neutral way. Why doesn’t the FAST
specification demand conformance here?Steven
Among arguments for not adding expressiveness for this type of conformance in the spec are the usual ones: let the spec do one thing and do it will. The golden rule is to be minimalistic and add features to the core specification that cannot be expressed at another level. For instance, the entire mapping of FIX onto FAST is kept outside the core specification to clearly separate protocol layers.
I think that template selection rules should be defined at another level. If it makes sense to create an industry wide basis for conformance regarding template selection, that definition could be an FPL specification.
Please note that the XML template format is extensible, which means that template selection rules could be incorporated together with the templates themselves in a single documnt.
In Chapter 9, the spec itself specifies the rules for extensibility so that different extensions don’t clash.
Anders Furuhed
When authoring a set of templates, one contemplates the most efficient
way to serialize each and every application message. Some templates are
meant to be used for multiple messages, while other templates are
specifically designed for an individual message format. The template
author needs a means to inform the encoder of his intentions - the
mapping between application message and template.Ideally, this mapping should be standardized by the specification. This
would allow one to author a set of templates and their corresponding
mappings to application messages once and to execute them on any FAST
implementation.Conformance permits me to replace one implementation with another with
relative ease - I don’t have to redefine my mappings. I can bake off two
different implementations using the same configuration (templates and
mappings). I don’t have to learn implementation-specific semantics; I
can rely on the FAST specification …The argument for conformance is pretty common in the software space due
to companies trying to avoid vendor-lockin. What’s the argument for not
demanding conformance?Steven
why do you think the FAST spec should demand conformance?
/Rolf
I appreciate your clear response. Allow me to take one step back.
You say that “template selection rules cannot be expressed in the
template syntax but is a feature of a particular encoder
implementation.”I’m understanding that the FAST specification deliberately chose
not to address encoder template selection (the process by which
an encoder chooses a particular template to use for marshaling a
particular application message to FAST bytes) to allow each
implementation to do it its own way. Why? This prevents a set of
templates written for one encoder from behaving the same way on
another encoder (without additional implementation-specific
configuration). More concretely, I (a template author) would want
the ability to control template selection in an implementation-
neutral way. Why doesn’t the FAST specification demand
conformance here?Steven
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
The argument for minimalism in a specification is certainly a good one. This is one of the principle arguments by the RELAX NG camp against the design of W3C XML Schema. However, I’m not sure that it’s relevant in our context.
The FAST Specification chose to define a data model for application messages suitable for FAST processing. That same data model is rich enough to control encoder template selection in a protocol independent manner. After all, template selection is tightly coupled with the application message described by a template. Since this is necessary work for every implementation, and the FAST specification has already provided the foundation, why not take this opportunity to define it once now, rather than leaving the work to protocol-specific sub-specifications?
I don’t believe that encoder template selection is an additional feature - it is part and parcel of template definition.
Steven
Among arguments for not adding expressiveness for this type of
conformance in the spec are the usual ones: let the spec do one thing
and do it will. The golden rule is to be minimalistic and add features
to the core specification that cannot be expressed at another level. For
instance, the entire mapping of FIX onto FAST is kept outside the core
specification to clearly separate protocol layers.I think that template selection rules should be defined at another
level. If it makes sense to create an industry wide basis for
conformance regarding template selection, that definition could be an
FPL specification.Please note that the XML template format is extensible, which means
that template selection rules could be incorporated together with the
templates themselves in a single documnt. In Chapter 9, the spec itself
specifies the rules for extensibility so that different extensions
don’t clash.Anders Furuhed
When authoring a set of templates, one contemplates the most efficient
way to serialize each and every application message. Some templates
are meant to be used for multiple messages, while other templates are
specifically designed for an individual message format. The template
author needs a means to inform the encoder of his intentions - the
mapping between application message and template.Ideally, this mapping should be standardized by the specification.
This would allow one to author a set of templates and their
corresponding mappings to application messages once and to execute
them on any FAST implementation.Conformance permits me to replace one implementation with another with
relative ease - I don’t have to redefine my mappings. I can bake off
two different implementations using the same configuration (templates
and mappings). I don’t have to learn implementation-specific
semantics; I can rely on the FAST specification …The argument for conformance is pretty common in the software space
due to companies trying to avoid vendor-lockin. What’s the argument
for not demanding conformance?Steven
why do you think the FAST spec should demand conformance?
/Rolf
I appreciate your clear response. Allow me to take one step back.
You say that “template selection rules cannot be expressed in the
template syntax but is a feature of a particular encoder
implementation.”I’m understanding that the FAST specification deliberately chose
not to address encoder template selection (the process by which an
encoder chooses a particular template to use for marshaling a
particular application message to FAST bytes) to allow each
implementation to do it its own way. Why? This prevents a set of
templates written for one encoder from behaving the same way on
another encoder (without additional implementation-specific
configuration). More concretely, I (a template author) would want
the ability to control template selection in an implementation-
neutral way. Why doesn’t the FAST specification demand conformance
here?Steven
I would like to understand better how you would like to express the selection rules.
Let’s say you have 3–8 different templates for MarketDataIncrementalRefresh, which is a realistic scenario if you would like to subset and constant code each type of event as much as possible.
So far, you would be able to select a template based on the value of two or three fields, but since there is no central template for MarketDataIncrementalRefresh, I guess that you would need a new specification entity that describes the dispatch/selection mechanism, rather than using the existing entities in the specification.
A very aggressive encoder could go even further and use more advanced selection criteria based on dynamically evaluating data patterns.
For counterparties that agree to use SCP, such an aggressive encoder could even create new short-lived templates on the fly to adapt to evolving data patterns.
But to get back to expressing more simple selection rules, would you be willing to propose an outline of how they should be expressed in the template spec?
Anders
The argument for minimalism in a specification is certainly a good one.
This is one of the principle arguments by the RELAX NG camp against the
design of W3C XML Schema. However, I’m not sure that it’s relevant in
our context.The FAST Specification chose to define a data model for application
messages suitable for FAST processing. That same data model is rich
enough to control encoder template selection in a protocol independent
manner. After all, template selection is tightly coupled with the
application message described by a template. Since this is necessary
work for every implementation, and the FAST specification has already
provided the foundation, why not take this opportunity to define it once
now, rather than leaving the work to protocol-specific sub-
specifications?I don’t believe that encoder template selection is an additional feature
- it is part and parcel of template definition.
Steven
Among arguments for not adding expressiveness for this type of
conformance in the spec are the usual ones: let the spec do one thing
and do it will. The golden rule is to be minimalistic and add features
to the core specification that cannot be expressed at another level.
For instance, the entire mapping of FIX onto FAST is kept outside the
core specification to clearly separate protocol layers.I think that template selection rules should be defined at another
level. If it makes sense to create an industry wide basis for
conformance regarding template selection, that definition could be an
FPL specification.Please note that the XML template format is extensible, which means
that template selection rules could be incorporated together with
the templates themselves in a single documnt. In Chapter 9, the spec
itself specifies the rules for extensibility so that different
extensions don’t clash.Anders Furuhed
When authoring a set of templates, one contemplates the most
efficient way to serialize each and every application message. Some
templates are meant to be used for multiple messages, while other
templates are specifically designed for an individual message
format. The template author needs a means to inform the encoder of
his intentions - the mapping between application message and
template.Ideally, this mapping should be standardized by the specification.
This would allow one to author a set of templates and their
corresponding mappings to application messages once and to execute
them on any FAST implementation.Conformance permits me to replace one implementation with another
with relative ease - I don’t have to redefine my mappings. I can
bake off two different implementations using the same configuration
(templates and mappings). I don’t have to learn implementation-
specific semantics; I can rely on the FAST specification …The argument for conformance is pretty common in the software space
due to companies trying to avoid vendor-lockin. What’s the argument
for not demanding conformance?Steven
why do you think the FAST spec should demand conformance?
/Rolf
I appreciate your clear response. Allow me to take one step
back.You say that “template selection rules cannot be expressed in
the template syntax but is a feature of a particular encoder
implementation.”I’m understanding that the FAST specification deliberately chose
not to address encoder template selection (the process by which
an encoder chooses a particular template to use for marshaling a
particular application message to FAST bytes) to allow each
implementation to do it its own way. Why? This prevents a set of
templates written for one encoder from behaving the same way on
another encoder (without additional implementation-specific
configuration). More concretely, I (a template author) would
want the ability to control template selection in an implementation-
neutral way. Why doesn’t the FAST specification demand
conformance here?Steven
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
You correctly point out that the expression of selection rules within template definitions is meant to benefit human-authored templates. The selection rules provide little to no value for programmatically hard-coded or runtime-generated templates. Perhaps this is reason enough to warrant a separate specification, as you suggest.
It’s my impression that human-authored templates will be the most common form of template declaration for the foreseeable future. In this vein, FPL should offer a standard semantic for expressing template selection.
I can take a stab at defining this semantic. We (NYSE TransactTools) perform a similar selection process for message routing and translation, independent of protocol. FAST templates are a bit different in that the fields that are relevant for selection are already enumerated in the definition, but the general concept is the same - selection is based on application message content.
Steven
I would like to understand better how you would like to express the
selection rules. Let’s say you have 3–8 different templates for
MarketDataIncrementalRefresh, which is a realistic scenario if you would
like to subset and constant code each type of event as much as possible.
So far, you would be able to select a template based on the value of two
or three fields, but since there is no central template for
MarketDataIncrementalRefresh, I guess that you would need a new
specification entity that describes the dispatch/selection mechanism,
rather than using the existing entities in the specification. A very
aggressive encoder could go even further and use more advanced selection
criteria based on dynamically evaluating data patterns. For
counterparties that agree to use SCP, such an aggressive encoder could
even create new short-lived templates on the fly to adapt to evolving
data patterns. But to get back to expressing more simple selection
rules, would you be willing to propose an outline of how they should be
expressed in the template spec?Anders
The argument for minimalism in a specification is certainly a good
one. This is one of the principle arguments by the RELAX NG camp
against the design of W3C XML Schema. However, I’m not sure that it’s
relevant in our context.The FAST Specification chose to define a data model for application
messages suitable for FAST processing. That same data model is rich
enough to control encoder template selection in a protocol independent
manner. After all, template selection is tightly coupled with the
application message described by a template. Since this is necessary
work for every implementation, and the FAST specification has already
provided the foundation, why not take this opportunity to define it
once now, rather than leaving the work to protocol-specific sub-
specifications?I don’t believe that encoder template selection is an
additional feature
- it is part and parcel of template definition.
Steven
Among arguments for not adding expressiveness for this type of
conformance in the spec are the usual ones: let the spec do one
thing and do it will. The golden rule is to be minimalistic and add
features to the core specification that cannot be expressed at
another level. For instance, the entire mapping of FIX onto FAST is
kept outside the core specification to clearly separate protocol
layers.I think that template selection rules should be defined at another
level. If it makes sense to create an industry wide basis for
conformance regarding template selection, that definition could be
an FPL specification.Please note that the XML template format is extensible, which means
that template selection rules could be incorporated together with
the templates themselves in a single documnt. In Chapter 9, the spec
itself specifies the rules for extensibility so that different
extensions don’t clash.Anders Furuhed
When authoring a set of templates, one contemplates the most
efficient way to serialize each and every application message.
Some templates are meant to be used for multiple messages, while
other templates are specifically designed for an individual
message format. The template author needs a means to inform the
encoder of his intentions - the mapping between application
message and template.Ideally, this mapping should be standardized by the specification.
This would allow one to author a set of templates and their
corresponding mappings to application messages once and to execute
them on any FAST implementation.Conformance permits me to replace one implementation with another
with relative ease - I don’t have to redefine my mappings. I can
bake off two different implementations using the same
configuration (templates and mappings). I don’t have to learn implementation-
specific semantics; I can rely on the FAST specification …The argument for conformance is pretty common in the software
space due to companies trying to avoid vendor-lockin. What’s the
argument for not demanding conformance?Steven
why do you think the FAST spec should demand conformance?
/Rolf
I appreciate your clear response. Allow me to take one step
back.You say that “template selection rules cannot be expressed in
the template syntax but is a feature of a particular encoder
implementation.”I’m understanding that the FAST specification deliberately
chose not to address encoder template selection (the process
by which an encoder chooses a particular template to use for
marshaling a particular application message to FAST bytes) to
allow each implementation to do it its own way. Why? This
prevents a set of templates written for one encoder from
behaving the same way on another encoder (without additional
implementation-specific configuration). More concretely, I (a
template author) would want the ability to control template
selection in an implementation- neutral way. Why doesn’t the
FAST specification demand conformance here?Steven
I agree that it may prove valuable to provide a standard way of expressing template selection and since it differs in nature from the definitions of the core FAST spec it is best put in its own spec. It differs in nature since it does not contain any information that must to be known by the receiving side in order to decode a FAST stream. However, as you point out, for human consumption, sharing of knowledge and for the implementation of encoders it would be a useful option.
I’d be happy to provide feedback if you come up with a proposal. As Anders suggested, one way of expressing the selection rules would be through an XML vocabulary that could be used as foreign elements and attributes to augment template definitions in the XML format.
/David
You correctly point out that the expression of selection rules within
template definitions is meant to benefit human-authored templates. The
selection rules provide little to no value for programmatically hard-
coded or runtime-generated templates. Perhaps this is reason enough to
warrant a separate specification, as you suggest.It’s my impression that human-authored templates will be the most common
form of template declaration for the foreseeable future. In this vein,
FPL should offer a standard semantic for expressing template selection.I can take a stab at defining this semantic. We (NYSE TransactTools)
perform a similar selection process for message routing and translation,
independent of protocol. FAST templates are a bit different in that the
fields that are relevant for selection are already enumerated in the
definition, but the general concept is the same - selection is based on
application message content.Steven