Imported from previous forum
[ original email was from Jim Northey - jnorthey@jandj.com ]
During my initial review of the FAST Template Definition (TD) Speciifcation - the specification of type conversion is one point that I really felt was embellishment and should not be part of the FAST Protocol and should be omitted. This is better left to implementors.
There is a difference between an API and a library layer of convenience functions and presentation services. I believe we should stay below these levels and omit this practice.
Also, FAST is a message protocol, and while it does have datatypes, it does not have the demands for type conversion/coercion implicit or otherwise as that of a programming language.
I would also point out that type safety and coercion / conversion rules are a rife area for philosophical debate within computer science, and as such I would like to exclude this layer and these arguments as being of diminished value for the application of our very limited time in specification creation and maintenance.
With that said, I also think I am one of the less capable folks in this discussion – so I would like some feedback and justification if others feel strongly that we should expand the scope of the FAST specification to include type conversion / coercion.
The section on type conversions relates to a previous discussion
on converting back and forth between ASCII formatted numbers in
FIX.
For example;
“100000” has multiple possible representations in FAST.
- “100000” - a string
- 100000 - an (un)signed integer
- 1E5 - a decimal with exponent=5 and mantissa=1
Alternative 1 would yield a 6 byte wire representation,
2 => 3 bytes and 3 => 2 bytes.
The number can be unabiguously converted back and forth.
Let’s take another example;
“001000” again has multiple representations fot the value, but
the added problem of leading zeroes is difficult to solve in a
straight forward way in FAST.
The obvious alternative is to use a string representation in
FAST but this comes at the extra cost of using 6 bytes instead
of 2 bytes for a binary representation (integer or decimal).
“001000” => 1000 (or 1E3) => “1000” would break format identity
but would preserve the original numeric value.
I believe type conversions are relevant to discuss in the context
of values versus representations in unecoded streams.
We may want to break this section out of the TD spec, but it is
important that we describe this problem and at least provide some
guidance as how to encoding and decoding should be done.
The cases I’ve seen with e.g. “0048” instead of “48” have been
optimizations for body length where an implementation reserves
space in the output buffer and then patches the buffer when the
rest of the message is completed (and the length is known) but
before calculating the message checksum.
There is another related case where the display format is encoded
into the value of a FIX field.
For example:
“121.50” has the same value as “121.5” but “121.50” also conveys
how the value could (should) be displayed.
I believe this to be a mixup of concerns. The number of displayed
decimals should be conveyed separately (and transferred whenever
the number changes), not as an implicit part of the value.
There is however a way to represent (with less efficiency) the
value in a way that could be applied to retain the extra decimal
place (even though the last zero is insignificant).
“191.50” => 19150E-2 => “191.50”
(instead of “191.50” => 1915E-1 => “191.5”)
(1915E-1 => 3 bytes, while 19150E-2 => 4 bytes)
Anyway, I believe we need to discuss this further.
During my initial review of the FAST Template Definition (TD)
Speciifcation - the specification of type conversion is one point
that I really felt was embellishment and should not be part of the
FAST Protocol and should be omitted.
This is better left to implementors.
There is a difference between an API and a library layer of
convenience functions and presentation services. I believe we
should stay below these levels and omit this practice.Also, FAST is a message protocol, and while it does have datatypes,
it does not have the demands for type conversion/coercion implicit
or otherwise as that of a programming language.I would also point out that type safety and coercion / conversion
rules are a rife area for philosophical debate within computer
science, and as such I would like to exclude this layer and these
arguments as being of diminished value for the application of our
very limited time in specification creation and maintenance.With that said, I also think I am one of the less capable folks in
this discussion – so I would like some feedback and justification
if others feel strongly that we should expand the scope of the FAST
specification to include type conversion / coercion.
Well put Rolf. I believe the problem lies in the historical practice of conveying multiple pieces of information within a single field. The most common example is a last price field that conveys both the last sale price point and the display precision of that price.
In the FAST Proof of Concept, we were very careful to decode an exact copy of the original input file. Often, this meant encoding some fields in a less efficient manor than if we were only required to convey the original information.
When FAST enabling a feed, it is also a good time to look at these “nuances” and correct them.
/Daniel
The section on type conversions relates to a previous discussion on
converting back and forth between ASCII formatted numbers in FIX.For example; “100000” has multiple possible representations in FAST.
- “100000” - a string
- 100000 - an (un)signed integer
- 1E5 - a decimal with exponent=5 and mantissa=1
Alternative 1 would yield a 6 byte wire representation, 2 => 3 bytes and
3 => 2 bytes.The number can be unabiguously converted back and forth.
Let’s take another example;
“001000” again has multiple representations fot the value, but the
added problem of leading zeroes is difficult to solve in a straight
forward way in FAST.The obvious alternative is to use a string representation in FAST but
this comes at the extra cost of using 6 bytes instead of 2 bytes for a
binary representation (integer or decimal).“001000” => 1000 (or 1E3) => “1000” would break format identity but
would preserve the original numeric value.I believe type conversions are relevant to discuss in the context of
values versus representations in unecoded streams.We may want to break this section out of the TD spec, but it is
important that we describe this problem and at least provide some
guidance as how to encoding and decoding should be done.The cases I’ve seen with e.g. “0048” instead of “48” have been
optimizations for body length where an implementation reserves space in
the output buffer and then patches the buffer when the rest of the
message is completed (and the length is known) but before calculating
the message checksum.There is another related case where the display format is encoded into
the value of a FIX field.For example:
“121.50” has the same value as “121.5” but “121.50” also conveys how the
value could (should) be displayed.I believe this to be a mixup of concerns. The number of displayed
decimals should be conveyed separately (and transferred whenever the
number changes), not as an implicit part of the value.There is however a way to represent (with less efficiency) the value in
a way that could be applied to retain the extra decimal place (even
though the last zero is insignificant).“191.50” => 19150E-2 => “191.50” (instead of “191.50” => 1915E-1
=> “191.5”)(1915E-1 => 3 bytes, while 19150E-2 => 4 bytes)
Anyway, I believe we need to discuss this further.
During my initial review of the FAST Template Definition (TD)
Speciifcation - the specification of type conversion is one point that
I really felt was embellishment and should not be part of the FAST
Protocol and should be omitted.This is better left to implementors.
There is a difference between an API and a library layer of
convenience functions and presentation services. I believe we should
stay below these levels and omit this practice.Also, FAST is a message protocol, and while it does have datatypes, it
does not have the demands for type conversion/coercion implicit or
otherwise as that of a programming language.I would also point out that type safety and coercion / conversion
rules are a rife area for philosophical debate within computer
science, and as such I would like to exclude this layer and these
arguments as being of diminished value for the application of our very
limited time in specification creation and maintenance.With that said, I also think I am one of the less capable folks in
this discussion – so I would like some feedback and justification if
others feel strongly that we should expand the scope of the FAST
specification to include type conversion / coercion.
[ original email was from Jim Northey - jnorthey@jandj.com ]
Sounds to me like we need to add Type Conversion to the list of enhancements and have people start to build experimental techniques now.
Rolf and Dan - thanks for educating me.
Well put Rolf. I believe the problem lies in the historical practice of
conveying multiple pieces of information within a single field. The most
common example is a last price field that conveys both the last sale
price point and the display precision of that price.In the FAST Proof of Concept, we were very careful to decode an exact
copy of the original input file. Often, this meant encoding some fields
in a less efficient manor than if we were only required to convey the
original information.When FAST enabling a feed, it is also a good time to look at these
“nuances” and correct them./Daniel
The section on type conversions relates to a previous discussion on
converting back and forth between ASCII formatted numbers in FIX.For example; “100000” has multiple possible representations in FAST.
- “100000” - a string
- 100000 - an (un)signed integer
- 1E5 - a decimal with exponent=5 and mantissa=1
Alternative 1 would yield a 6 byte wire representation, 2 => 3 bytes
and 3 => 2 bytes.The number can be unabiguously converted back and forth.
Let’s take another example;
“001000” again has multiple representations fot the value, but the
added problem of leading zeroes is difficult to solve in a straight
forward way in FAST.The obvious alternative is to use a string representation in FAST but
this comes at the extra cost of using 6 bytes instead of 2 bytes for a
binary representation (integer or decimal).“001000” => 1000 (or 1E3) => “1000” would break format identity but
would preserve the original numeric value.I believe type conversions are relevant to discuss in the context of
values versus representations in unecoded streams.We may want to break this section out of the TD spec, but it is
important that we describe this problem and at least provide some
guidance as how to encoding and decoding should be done.The cases I’ve seen with e.g. “0048” instead of “48” have been
optimizations for body length where an implementation reserves space
in the output buffer and then patches the buffer when the rest of the
message is completed (and the length is known) but before calculating
the message checksum.There is another related case where the display format is encoded into
the value of a FIX field.For example:
“121.50” has the same value as “121.5” but “121.50” also conveys how
the value could (should) be displayed.I believe this to be a mixup of concerns. The number of displayed
decimals should be conveyed separately (and transferred whenever the
number changes), not as an implicit part of the value.There is however a way to represent (with less efficiency) the value
in a way that could be applied to retain the extra decimal place (even
though the last zero is insignificant).“191.50” => 19150E-2 => “191.50” (instead of “191.50” => 1915E-1
=> “191.5”)(1915E-1 => 3 bytes, while 19150E-2 => 4 bytes)
Anyway, I believe we need to discuss this further.
During my initial review of the FAST Template Definition (TD)
Speciifcation - the specification of type conversion is one point
that I really felt was embellishment and should not be part of the
FAST Protocol and should be omitted.This is better left to implementors.
There is a difference between an API and a library layer of
convenience functions and presentation services. I believe we should
stay below these levels and omit this practice.Also, FAST is a message protocol, and while it does have datatypes,
it does not have the demands for type conversion/coercion implicit
or otherwise as that of a programming language.I would also point out that type safety and coercion / conversion
rules are a rife area for philosophical debate within computer
science, and as such I would like to exclude this layer and these
arguments as being of diminished value for the application of our
very limited time in specification creation and maintenance.With that said, I also think I am one of the less capable folks in
this discussion – so I would like some feedback and justification if
others feel strongly that we should expand the scope of the FAST
specification to include type conversion / coercion.