Imported from previous forum
[ original email was from Jim Northey - jnorthey@jandj.com ]
We need to finalize how templates are represented as part of FAST 1.1 (the documentation release that clarifies the current FAST 1.0 functionality) and includes the Session Control Protocol v1.0 that is being reviewed now.
There are a couple of proposals the XML proposal from Pantor that is documented in the TD specification. And the compact notation that initially appeared in Implicit Tagging and is represented in the FAST Basic User’s Guide.
So we have some choices: XML Only? Compact Notation Only? XML + Compact Notation as alternatives.
My view is I don’t have a problem with XML and a compact notation version. I think there are some reasonable arguments against having to have XML as part of the market data application. Probably equal arguments for it.
Either way MDOWG decides to go - my recommendation is that we uncouple the definition of templates from a specific implementation, be it XML or Compact Notation.
What do others think? There are arguments that Compact Notation does not have all the capabilities required. There are views that XML is a bit to verbose for high volume market data applications, even though they are only being used to define templates and are not being used to transmit data.
What do others think? There are arguments that Compact Notation does
not have all the capabilities required. There are views that XML is a
bit to verbose for high volume market data applications, even though
they are only being used to define templates and are not being used to
transmit data.
I think that there will be more comments on the utility of a Compact Notation if there is a proposal that covers the same ground as the proposed XML variant does. It will then be more obvious whether the CN scales beyond the original version. One concern is that a CN is not inherently extensible. If a new template feature is added, the Compact Notation syntax has to be amended.
It is not obvious to me how the space characteristics of an offline representation relates to the demands for space efficiency in a feed?
[ original email was from Jim Northey - jnorthey@jandj.com ]
Thank you Anders for the reply. I think that we have to decide if CN makes sense as an alternative for FAST 1.1 (FAST1.0+SCP1.0+Template Encoding) and see what the loss is.
As the self appointed document updater, I am working now on the specification for templates independent of representation that perhaps then those interested in CN can use to create the CN external representation specification using the Basic User’s Guide as a starting point.
I understand the merits of CN at this point in time, but not so much that I am willing to spend my own time developing that part of the specification.
Also, in regards to the XML - we have the TD - but we haven’t had feedback on the XML and the proposal contained within the TD. Right now for the updated documents I am trying to integrate as many TD concepts as possible that apply to what we thing is FAST 1.0.
[[Digression Every day I gleen new insights and value as I understand the TD model - at some point the rest of us will catch up to all the concepts and capabilities implied in the TD specification that David Rosenborg created. No matter if we assimilate the concepts into a different representation or not - I have to say we need to keep TD Specification 0.38 as one of the FAST documents and as a great source of conceptual wealth - it is a road map- one that requires persistence and several reads to gain the value. And this approach has precedent in keeping documents around for reference - this is the way the RFC (Request for Comment) process works that was the basis for the creation of the Internet. In fact, one of the BNF grammars I relied on and studied was ABNF - which is documented in an RFC. ]]
What do others think? There are arguments that Compact Notation does
not have all the capabilities required. There are views that XML is a
bit to verbose for high volume market data applications, even though
they are only being used to define templates and are not being used to
transmit data.I think that there will be more comments on the utility of a Compact
Notation if there is a proposal that covers the same ground as the
proposed XML variant does. It will then be more obvious whether the CN
scales beyond the original version. One concern is that a CN is not
inherently extensible. If a new template feature is added, the Compact
Notation syntax has to be amended.It is not obvious to me how the space characteristics of an offline
representation relates to the demands for space efficiency in a feed?
[ original email was from David Rosenborg - David.Rosenborg@pantor.com ]
The XML notation is not intended for high volume exchange of template definitions. This is clearly stated in the Introducion chapter in TD. For this purpose we proposed using FAST encoded messages. This was initially part of the SCP draft. And if it don’t find its way back in, I guess we’ll have to create yet another specification: FAST Template Exchange Protocol (TEP) or something. The merits of such a protocol is not just space efficiency. FAST encoded templates mean one parser less to include in the execution path of your application.
/David
What do others think? There are arguments that Compact Notation does
not have all the capabilities required. There are views that XML is a
bit to verbose for high volume market data applications, even though
they are only being used to define templates and are not being used to
transmit data.
[ original email was from Greg Orsini - go@cameronsystems.com ]
> Either way MDOWG decides to go - my recommendation is that we uncouple
the definition of templates from a specific implementation, be it XML or
Compact Notation.
I agree with this approach. Uncoupling the format of the definition (XML or compact) from the definition itself will quiet concerns about the complexity or simplicity of a particular format.
Greg
[ original email was from David Rosenborg - David.Rosenborg@pantor.com ]
Clearly the concrete syntax is not the same as the abstract notion of templates. However this doesn’t mean we must have it in separate documents. If we go down that road, split things in multiple documents just becaus it is possible from a theoretical point of view, I’m convinced the result will be unecessary complex. Actually, I think we should lean in the other direction: trying to minimize the number of documents when possible.
The fact that the specification describing the abstract concepts of templates also include a concrete syntax, a reference concrete syntax if you like, does not exclude any other syntaxes. The TD document is pretty specific on this point: there’s no requirement that a FAST processor must be able to read the XML syntax.
If you´re interested in examples from the world outside FIX/FAST you can look at how the RELAX NG (http://relaxng.org, the XML schema language used in TD) documents were laid out. First a core specification was developed which included a reference syntax in XML and the formal semantics. Then there was a tutorial which informally described the concepts of the core spec. Later, a compact notation specification was added. In fact, the existence of a reference XML syntax helped creating the compact notation since it is described as a syntactical transformation to the reference XML syntax.
The fact that the reference XML syntax was an intrinsic part of the core spec was never an issue. Nore would it need to be an issue in FAST.
/David
Either way MDOWG decides to go - my recommendation is that we uncouple
the definition of templates from a specific implementation, be it XML
or Compact Notation.I agree with this approach. Uncoupling the format of the definition (XML
or compact) from the definition itself will quiet concerns about the
complexity or simplicity of a particular format.Greg
My thinking is that there are 3 concepts here:
-
Establishing the definition of what “is” a template (what it can express)
-
Providing a way to express a “human interpeted” instance of a template (so that we can describe, understand, alter, etc)
-
Providing a way to express, and ideally optimize, a “computer parsed/interpreted” instance of a tempate
There should only be one #1.
There could be more than one #2 (i.e. Compact Notation and the XML) if it makes sense or there is precendence, although, it sounds like the target set of capabilities for #1 exceeds the Compact Notation semantic set. Presumably if there are two instance formats, a simple utility could be built to convert from one to the other.
While it is possible for requirement #3 to simply be met by #2, I don’t think that has to be the case especially for something performance intensive. I envisage a scenario where one might “compile” #2 into a #3 format in the same way that one compiles C code into a runtime module. While XML can ‘easily’ be parsed, that requires a parser and may not be the most efficient approach (akin to using PERL vs. C exe).
We need to finalize how templates are represented as part of FAST 1.1
(the documentation release that clarifies the current FAST 1.0
functionality) and includes the Session Control Protocol v1.0 that is
being reviewed now.There are a couple of proposals the XML proposal from Pantor that is
documented in the TD specification. And the compact notation that
initially appeared in Implicit Tagging and is represented in the FAST
Basic User’s Guide.So we have some choices: XML Only? Compact Notation Only? XML + Compact
Notation as alternatives.My view is I don’t have a problem with XML and a compact notation
version. I think there are some reasonable arguments against having to
have XML as part of the market data application. Probably equal
arguments for it.Either way MDOWG decides to go - my recommendation is that we uncouple
the definition of templates from a specific implementation, be it XML or
Compact Notation.What do others think? There are arguments that Compact Notation does
not have all the capabilities required. There are views that XML is a
bit to verbose for high volume market data applications, even though
they are only being used to define templates and are not being used to
transmit data.
I don’t think that the verbosity argument holds water. Even if the XML message would be sent in band, it is 1) sent only once, 2) sent at a time when no market data is being sent anyway (since it must first be processed before any can be accepted). I really don’t know what to say about an XML parser “not belonging” in a market data application. That doesn’t even make sense to me. Can someone clarify?
In my view, the Compact Notation is confusing and hard to understand in comparison to the XML format. It would also require a custom parser. I don’t really see it’s advantage since compactness doesn’t seem to help in any way. So all I see are the disadvantages of a compact format without any of the benefits since it is sent either out of band, or would be sent in band very infrequently.
So my preference would be to see a format based on a well supported standard such as XML or YAML.
I also do think it is important to have a standard, or at least recommended format. Of course anyone can implement their own format for whatever reason, but I think the ability to share these files is key. In any case, my view is we pick one format and run with it. I don’t see the value in two formats.
We need to finalize how templates are represented as part of FAST 1.1
(the documentation release that clarifies the current FAST 1.0
functionality) and includes the Session Control Protocol v1.0 that is
being reviewed now.There are a couple of proposals the XML proposal from Pantor that is
documented in the TD specification. And the compact notation that
initially appeared in Implicit Tagging and is represented in the FAST
Basic User’s Guide.So we have some choices: XML Only? Compact Notation Only? XML + Compact
Notation as alternatives.My view is I don’t have a problem with XML and a compact notation
version. I think there are some reasonable arguments against having to
have XML as part of the market data application. Probably equal
arguments for it.Either way MDOWG decides to go - my recommendation is that we uncouple
the definition of templates from a specific implementation, be it XML or
Compact Notation.What do others think? There are arguments that Compact Notation does
not have all the capabilities required. There are views that XML is a
bit to verbose for high volume market data applications, even though
they are only being used to define templates and are not being used to
transmit data.
I don’t know the distribution of that version of the document, but an earlier draft of SCP contained messages for template exchange, i.e. the templates were transmitted as FAST messages themselves.
This makes transfer efficient, which is nice if you would like to replace a template mid-day for instance or if you use extremely short-lived sessions. More importantly, your existing FAST processor can be reused so that no additional parser is needed.
I don’t think that the verbosity argument holds water. Even if the XML
message would be sent in band, it is 1) sent only once, 2) sent at a
time when no market data is being sent anyway (since it must first be
processed before any can be accepted). I really don’t know what to say
about an XML parser “not belonging” in a market data application. That
doesn’t even make sense to me. Can someone clarify?In my view, the Compact Notation is confusing and hard to understand in
comparison to the XML format. It would also require a custom parser. I
don’t really see it’s advantage since compactness doesn’t seem to help
in any way. So all I see are the disadvantages of a compact format
without any of the benefits since it is sent either out of band, or
would be sent in band very infrequently.So my preference would be to see a format based on a well supported
standard such as XML or YAML.I also do think it is important to have a standard, or at least
recommended format. Of course anyone can implement their own format for
whatever reason, but I think the ability to share these files is key. In
any case, my view is we pick one format and run with it. I don’t see the
value in two formats.We need to finalize how templates are represented as part of FAST 1.1
(the documentation release that clarifies the current FAST 1.0
functionality) and includes the Session Control Protocol v1.0 that is
being reviewed now.There are a couple of proposals the XML proposal from Pantor that is
documented in the TD specification. And the compact notation that
initially appeared in Implicit Tagging and is represented in the FAST
Basic User’s Guide.So we have some choices: XML Only? Compact Notation Only? XML +
Compact Notation as alternatives.My view is I don’t have a problem with XML and a compact notation
version. I think there are some reasonable arguments against having to
have XML as part of the market data application. Probably equal
arguments for it.Either way MDOWG decides to go - my recommendation is that we uncouple
the definition of templates from a specific implementation, be it XML
or Compact Notation.What do others think? There are arguments that Compact Notation does
not have all the capabilities required. There are views that XML is a
bit to verbose for high volume market data applications, even though
they are only being used to define templates and are not being used to
transmit data.
I had thought about this, but it wasn’t immediately clear to me how you could parse any FAST message without first having a template. How would the FAST message containing a template be parsed without first having a template?
I suppose we could standardize the template for transfering templates. Essentially reserve template id 1 for passing templates around, and all FAST engines would have this template preloaded. Is that what the document described?
I don’t know the distribution of that version of the document, but an
earlier draft of SCP contained messages for template exchange, i.e. the
templates were transmitted as FAST messages themselves. This makes
transfer efficient, which is nice if you would like to replace a
template mid-day for instance or if you use extremely short-lived
sessions. More importantly, your existing FAST processor can be reused
so that no additional parser is needed.
You’re absolutely correct, the SCP templates would be hard wired with reserved templates identifiers.
I had thought about this, but it wasn’t immediately clear to me how
you could parse any FAST message without first having a template. How
would the FAST message containing a template be parsed without first
having a template?I suppose we could standardize the template for transfering templates.
Essentially reserve template id 1 for passing templates around, and all
FAST engines would have this template preloaded. Is that what the
document described?I don’t know the distribution of that version of the document, but an
earlier draft of SCP contained messages for template exchange, i.e.
the templates were transmitted as FAST messages themselves. This makes
transfer efficient, which is nice if you would like to replace a
template mid-day for instance or if you use extremely short-lived
sessions. More importantly, your existing FAST processor can be reused
so that no additional parser is needed.
[ original email was from Matt Simpson - msimpson@cme.com ]
There are several good ideas circulating here. I agree that we need to separate the concept of template representation from the template wire format.
At the point template exchange is introduced as a part of SCP (future release) it makes sense that a special Template Exchange format will be used to convey the structure of a template. Neither XML nor compact notation will be sent across the wire. These are only to be used as the ‘human interpreted’ form of the data structure as coined by Scott Atwell. The SCP template format will be closest to a machine-level representation.
So the question of compact notation vs XML becomes one of what’s best for purposes of readability - how should I represent my message structure so that it is most intuitive to those reading it? For those accustomed to FIX tag/value the answer is probably compact notation since it emphasizes tag numbers in a format that evokes a FIX message. For those more comfortable with XML, then obviously TD-ML makes most sense.
My vote at this point is to keep both
You’re absolutely correct, the SCP templates would be hard wired with
reserved templates identifiers.I had thought about this, but it wasn’t immediately clear to me how
you could parse any FAST message without first having a template. How
would the FAST message containing a template be parsed without first
having a template?I suppose we could standardize the template for transfering templates.
Essentially reserve template id 1 for passing templates around, and
all FAST engines would have this template preloaded. Is that what the
document described?I don’t know the distribution of that version of the document, but
an earlier draft of SCP contained messages for template exchange,
i.e. the templates were transmitted as FAST messages themselves.
This makes transfer efficient, which is nice if you would like to
replace a template mid-day for instance or if you use extremely short-
lived sessions. More importantly, your existing FAST processor can
be reused so that no additional parser is needed.
Personally I prefer the XML format over CN for 2 reasons :
-
It is more precise with regards to the data types being used and also includes constructs to express concepts such as dictionary scope.
-
From a support perspective the XML format is more accessible to support engineers and non-coders supporting/configuring a FAST based application.
Darren.
[ original email was from Clive Browning - clive@rapidaddition.com ]
I concur: we are currently using the XML format rather than CN.
We also feel that the XML format is more extensible if we decide that we want to add our own custom attributes for example to define the layout of the intput or output streams.
Personall I also feel it is much more readable.
Clive
Rapid Addition Ltd
Personally I prefer the XML format over CN for 2 reasons :
It is more precise with regards to the data types being used and also
includes constructs to express concepts such as dictionary scope.From a support perspective the XML format is more accessible to
support engineers and non-coders supporting/configuring a FAST based
application.Darren.