Imported from previous forum
OPRA is not a FAST compliant market data feed - it is FAST-like but is not compliant with the FAST Specification. What are other vendors doing to address the OPRA Market data feed - which should not be confused with being a FAST market data feed?
We are a bit concerned to alter or break our standard implementations to support OPRA. Seems like an alternative product or code branch might be preferred to corrupting the main code line of a FAST compliant product.
Jim,
This has been a long standing issue with OPRA that seems to have fallen on deaf ears. It all started when OPRA simply used the proof on concept code that SpryWare and Pantor wrote to demonstrate the concept of FAST and released it as a production feed. They never consulted the group to review their implementation, in fact they distributed decoder source coded that still contained the original proof of concept cvs version control headers.
The proof of concept code was not FAST compliant, because FAST was not even formalized at that point. I do believe they consulted Jordan and Jordan before releasing their version 2 of OPRA FAST, “OPRA FAST for Symbology”, about correcting the non compliant portions of the code. For some reason they chose not to bother.
So you are left with a feed that cannot be decoded with a generic FAST decoder. The changes are minor, simply adding a few more templates rather than having messages templates dynamically changed based on message content.
/Daniel
OPRA is not a FAST compliant market data feed - it is FAST-like but is
not compliant with the FAST Specification. What are other vendors doing
to address the OPRA Market data feed - which should not be confused with
being a FAST market data feed? We are a bit concerned to alter or break
our standard implementations to support OPRA. Seems like an alternative
product or code branch might be preferred to corrupting the main code
line of a FAST compliant product.
It was mind over matter.
(They don’t mind because we obviously don’t matter to them)
Jim, This has been a long standing issue with OPRA that seems to have
fallen on deaf ears. It all started when OPRA simply used the proof on
concept code that SpryWare and Pantor wrote to demonstrate the concept
of FAST and released it as a production feed. They never consulted the
group to review their implementation, in fact they distributed decoder
source coded that still contained the original proof of concept cvs
version control headers.The proof of concept code was not FAST compliant, because FAST was not
even formalized at that point. I do believe they consulted Jordan and
Jordan before releasing their version 2 of OPRA FAST, “OPRA FAST for
Symbology”, about correcting the non compliant portions of the code. For
some reason they chose not to bother.So you are left with a feed that cannot be decoded with a generic FAST
decoder. The changes are minor, simply adding a few more templates
rather than having messages templates dynamically changed based on
message content./Daniel
OPRA is not a FAST compliant market data feed - it is FAST-like but is
not compliant with the FAST Specification. What are other vendors
doing to address the OPRA Market data feed - which should not be
confused with being a FAST market data feed? We are a bit concerned to
alter or break our standard implementations to support OPRA. Seems
like an alternative product or code branch might be preferred to
corrupting the main code line of a FAST compliant product.
[ original email was from Rich Shriver - shriver@jandj.com ]
FPL and members of MDOWG provided feedback to SIAC that the OPRA FAST implementation was non-compliant with the FAST specifications. The feedback has yet to be incorporated or addressed. What has not occured is pressure from OPRA direct feed customers to make the technology compliant with the standards. As a standards organization, we have also not effectively lobbied OPRA and their technology service provider to comply with the standards as developed. I am not saying we have not lobbied, I am simply stating we have yet to be successful.
Rich Shriver
Jim, This has been a long standing issue with OPRA that seems to have
fallen on deaf ears. It all started when OPRA simply used the proof on
concept code that SpryWare and Pantor wrote to demonstrate the concept
of FAST and released it as a production feed. They never consulted the
group to review their implementation, in fact they distributed decoder
source coded that still contained the original proof of concept cvs
version control headers.The proof of concept code was not FAST compliant, because FAST was not
even formalized at that point. I do believe they consulted Jordan and
Jordan before releasing their version 2 of OPRA FAST, “OPRA FAST for
Symbology”, about correcting the non compliant portions of the code. For
some reason they chose not to bother.So you are left with a feed that cannot be decoded with a generic FAST
decoder. The changes are minor, simply adding a few more templates
rather than having messages templates dynamically changed based on
message content./Daniel
OPRA is not a FAST compliant market data feed - it is FAST-like but is
not compliant with the FAST Specification. What are other vendors
doing to address the OPRA Market data feed - which should not be
confused with being a FAST market data feed? We are a bit concerned to
alter or break our standard implementations to support OPRA. Seems
like an alternative product or code branch might be preferred to
corrupting the main code line of a FAST compliant product.
Jim, This has been a long standing issue with OPRA that seems to have
fallen on deaf ears. It all started when OPRA simply used the proof on
concept code that SpryWare and Pantor wrote to demonstrate the concept
of FAST and released it as a production feed. They never consulted the
group to review their implementation, in fact they distributed decoder
source coded that still contained the original proof of concept cvs
version control headers.The proof of concept code was not FAST compliant, because FAST was not
even formalized at that point. I do believe they consulted Jordan and
Jordan before releasing their version 2 of OPRA FAST, “OPRA FAST for
Symbology”, about correcting the non compliant portions of the code. For
some reason they chose not to bother.So you are left with a feed that cannot be decoded with a generic FAST
decoder. The changes are minor, simply adding a few more templates
rather than having messages templates dynamically changed based on
message content./Daniel
OPRA is not a FAST compliant market data feed - it is FAST-like but is
not compliant with the FAST Specification. What are other vendors
doing to address the OPRA Market data feed - which should not be
confused with being a FAST market data feed? We are a bit concerned to
alter or break our standard implementations to support OPRA. Seems
like an alternative product or code branch might be preferred to
corrupting the main code line of a FAST compliant product.
Hello,
Could you explain please where OPRA is not compliant?
Is it about the Template ID?
It would be fine to have some facts.
Thank you,
Oleg
Please check with your colleagues. Both Dan, I, and J&J have responded to questions and provided advice on how to make the OPRA FAST-alike feed compliant with the spec.
What’s your role at NYSE/SIAC?
Best,
Rolf
Hello,
Could you explain please where OPRA is not compliant? Is it about the
Template ID? It would be fine to have some facts.Thank you, Oleg
Oleg,
As far as I can recall, there are several points where the feed is not FAST compliant.
-
It includes a proprietary header and I believe a footer on each message. This would not be the end of the world if the payload was actually compliant.
-
The biggest issue is the Quote template. The quote template definition is not constant, it is defined after you process the BBO appendage fid, and based on the value, you can then determine the number of fields that follow. This could be solved by having a template ID for each possible BBO appendage.
This one was my own creation in the early FAST proof of concept days, that is how I coded it to show haw FAST could work on Opra data. It made sense at the time, but that was before FAST was formalized.
Look at the supplied fast_decode.c from Opra, and search for:
//decode bbo indicator
msg->bboIndicator = fast->decode_u32(fast, BBO_INDICATOR);
switch(msg->bboIndicator)
{
This explains it all (if you read ‘C’ code). The value of BBO_INDICATOR determines the number and type of fields that follow, this is not FAST compliant.
I do believe Rolf and some other members of the group are performing a more complete survey of the Opra implementation.
/Daniel
Oleg, As far as I can recall, there are several points where the feed is
not FAST compliant.
It includes a proprietary header and I believe a footer on each
message. This would not be the end of the world if the payload was
actually compliant.The biggest issue is the Quote template. The quote template
definition is not constant, it is defined after you process the BBO
appendage fid, and based on the value, you can then determine the
number of fields that follow. This could be solved by having a
template ID for each possible BBO appendage.This one was my own creation in the early FAST proof of concept days,
that is how I coded it to show haw FAST could work on Opra data. It made
sense at the time, but that was before FAST was formalized.Look at the supplied fast_decode.c from Opra, and search for:
//decode bbo indicator msg->bboIndicator = fast->decode_u32(fast,
BBO_INDICATOR);switch(msg->bboIndicator) {This explains it all (if you read ‘C’ code). The value of BBO_INDICATOR
determines the number and type of fields that follow, this is not FAST
compliant.I do believe Rolf and some other members of the group are performing a
more complete survey of the Opra implementation./Daniel
For the point # 1.
As far as I understood the specification of FAST1.1, proprietary header and trailer is completely independent of the compliance. This is application specific and does not play any role for the FAST (like UDP packet header has nothing to do with FAST). More over the greatest advantage of the packet header is for the network group who wants to check the sequence of the messages without decoding the FAST messages. This header give flexibility and information about the sequence number of the 1st message and the number of messages/packet, size of the message etc. This information will be vary valuable for the tools like Sniffer where without integrating FAST protocol to tools network engineer can analyze the gaps in the packets. Actually FAST working group should consider this idea for the future.
For the point # 2.
OPRA consist of generalize template for all the messages. FAST engine can (must) run without any business knowledge of the message and strictly based on the information received from the wire. So if the FAST 1.1 compliant decoder is placed with these messages, it should work. Later application needs to build the state engine to constitute the message based on the information received from the FAST engine(decoder/template).
Sample decoder is given as an example and to reconstitute the original message to ASCII. State machine is being used to decode the original message.
Now going back to the original comments about the compliance. OPRA uses only two different type of operators.
- Copy : There is no change from FAST 1.0 to FAT 1.1 spec. So this operator for 1.0 is comply with 1.1
- increment : There is no change from FAST 1.0 to FAT 1.1 spec. So this operator for 1.0 is comply with 1.1
So if the reference for adding more template IDs for bbo indicator then it is incorrect as per the spec as it will make FAST engine from generalize engine to OPRA specific.
I am still looking in the section 10.0 of FAST 1.1 where there are some ambiguity which needs to be resolve. For example, the xml statement doesn’t mandate to have the templateIdentifier as the 1st bit of the presence map.
message ::= segment
segment ::= PresenceMap TemplateIdentifier? (field | segment)*
and the other statement where TID is optional as the 1st bit.:
“A segment has a header consisting of a Presence Map followed by an optional Template Identifier. The segment has a template identifier either if it is a message segment, or if the segment appears as the result of a dynamic template reference instruction. A template identifier is encoded as if a copy operator was specified. The operator uses the global dictionary and has an internal key common to all template identifier fields. This means that a segment with a template identifier does not always contain the template identifier physically. However, the first bit in the presence map is allocated by its copy operator. “
Please let me know if I am misinterpreting the spec.
-Shwetang
Shwetang -
SIAC’s new-found interest in FAST compliance is very welcome.
Just to make sure that I’m using the right OPRA spec:
Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf
the current version of the OPRA spec?
/Rolf
Rolf,
Yes this is the correct URL for symbology. Please refer to page 8 for generalize template for all messages.
Individual templates are given for those vendors who were using individual state machine for separate messages.
Thanks,
Shwetang
Shwetang -
SIAC’s new-found interest in FAST compliance is very welcome.
Just to make sure that I’m using the right OPRA spec:
Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf the current
version of the OPRA spec?/Rolf
Thx, I’ll base my comments on that doc then.
I’m not sure I understand your comment about individual templates?
Best,
Rolf
Rolf, Yes this is the correct URL for symbology. Please refer
to page 8 for generalize template for all messages.Individual templates are given for those vendors who were using
individual state machine for separate messages.Thanks, Shwetang
Shwetang,
please elaborate on the “Individual templates” that you refer to below. I wasn’t aware of any vendor specific templates?
Best,
Rolf
Rolf, Yes this is the correct URL for symbology. Please refer to page 8
for generalize template for all messages.Individual templates are given for those vendors who were using
individual state machine for separate messages.Thanks, Shwetang
Shwetang -
SIAC’s new-found interest in FAST compliance is very welcome.
Just to make sure that I’m using the right OPRA spec:
Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf the current
version of the OPRA spec?/Rolf
Rolf,
Individual templates are give to help at the application layer for stat machine not at the FAST decoding.
OPRA uses only one generic template. Point 11 On page 16 has explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf document.
Thanks,
Shwetang
Shwetang,
please elaborate on the “Individual templates” that you refer to below.
I wasn’t aware of any vendor specific templates?Best, Rolf
Rolf, Yes this is the correct URL for symbology. Please refer to page
8 for generalize template for all messages.Individual templates are given for those vendors who were using
individual state machine for separate messages.Thanks, Shwetang
Shwetang -
SIAC’s new-found interest in FAST compliance is very welcome.
Just to make sure that I’m using the right OPRA spec:
Is http://www.opradata.com/specs/fast_for_oprav2_00.pdf the current
version of the OPRA spec?/Rolf
OK, so those templates are not FAST templates but some other kind of templates specific to you impl?
/Rolf
Rolf, Individual templates are give to help at the application
layer for stat machine not at the FAST decoding.OPRA uses only one generic template. Point 11 On page 16 has
explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf
document.Thanks, Shwetang
Rolf,
yes, Specifically the state machine given in our decoder.
Thanks,
Shwetang
OK, so those templates are not FAST templates but some other kind of
templates specific to you impl?/Rolf
Rolf, Individual templates are give to help at the application layer
for stat machine not at the FAST decoding.OPRA uses only one generic template. Point 11 On page 16 has
explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf
document.Thanks, Shwetang
Hm, I still don’t get what you are referring to. I may be dense, but in terms of “individual templates”, who gets what and why?
/Rolf
Rolf, yes, Specifically the state machine given in our decoder.
Thanks, Shwetang
OK, so those templates are not FAST templates but some other kind of
templates specific to you impl?/Rolf
Rolf, Individual templates are give to help at the application layer
for stat machine not at the FAST decoding.OPRA uses only one generic template. Point 11 On page 16 has
explanation in the http://opradata.com/specs/fast_for_oprav2_00.pdf
document.Thanks, Shwetang
OK. Ignore these individual templates for your evaluation. Just refer to general template on page 8.
Thanks,
Shwetang
Hm, I still don’t get what you are referring to. I may be dense, but in
terms of “individual templates”, who gets what and why?/Rolf
Rolf, yes, Specifically the state machine given in our decoder.
Thanks, Shwetang
OK, so those templates are not FAST templates but some other kind of
templates specific to you impl?/Rolf
Rolf, Individual templates are give to help at the application
layer for stat machine not at the FAST decoding.OPRA uses only one generic template. Point 11 On page 16 has
explanation in the
http://opradata.com/specs/fast_for_oprav2_00.pdf document.Thanks, Shwetang
please?
OK. Ignore these individual templates for your evaluation. Just refer to
general template on page 8.Thanks, Shwetang
Hm, I still don’t get what you are referring to. I may be dense, but
in terms of “individual templates”, who gets what and why?/Rolf
Rolf, yes, Specifically the state machine given in our decoder.
Thanks, Shwetang
OK, so those templates are not FAST templates but some other kind
of templates specific to you impl?/Rolf
Rolf, Individual templates are give to help at the application
layer for stat machine not at the FAST decoding.OPRA uses only one generic template. Point 11 On page 16 has
explanation in the
http://opradata.com/specs/fast_for_oprav2_00.pdf document.Thanks, Shwetang
I’ve had a look at the current spec (FAST for OPRA version 02.00 dated January 23, 2008) and I have the following observations:
The Template specification is not compliant with FAST 1.1 for the following reasons:
-
Fields are not in pmap sequence
Some pmap slots are unused for each of the specified messages. -
The presence of some fields depend on the value of another field.
The presence of best bid/ask fields depend on the BBO indicator field -
A proprietary framing protocol is used.
To add insult to injury, the FAST standard specifies an alternative that provides the same function.
In my view, the use of one “generic” template creates much of the compatibility problem. Careful use of a few templates would have made it possible to efficiently use FAST-api 1.0 in a FAST 1.1 compliant manner.
I found a way to specify a single FAST template that can be used by a compliant FAST 1.1 decoder to process the individual messages after removing the proprietary framing. The resulting output however contains superfluous incorrect content and additional processing is required to remove the incorrect data generated by the decoder.
In summary: While not impossible to use together with a proprietary de-framer, the generic OPRA template prevents recipients from efficiently using a standard FAST 1.1 implementation and requires a custom built decoder in order to get decent performance.
/Rolf
(on a side-note: The FAST for OPRA specification needs to be reviewed and improved. It is incomplete, ambiguous and contains a number of errors. It doesn’t differentiate clearly between FAST 1.0 and 1.1. It mixes snippets of text from the FAST 1.0 and 1.1 specs. The text is partly presented out of context. Some information is not relevant to the current implementation of FAST for OPRA (eg. deltas are not used, but the spec contains a partly erroneous description of delta coding))
OK. Ignore these individual templates for your evaluation.
Just refer to general template on page 8.Thanks, Shwetang