Imported from previous forum
Following a conversation on calculated fields, the discussion moved to discussing time representation:
Hanno:
Another example for the concatenation is again the UTCTimestamp datatype in FIX. If it can be split into date+time on the wire, then date can be coded very efficiently as a constant if it does not change during the session, requiring only time to be sent on the wire. The receiving end can put the two fields back together to create a valid FIX datatype again.
JimN:
I like the idea of splitting time up and using a smaller integer value for the time component. This has been a common practice in market data for quite some time.
However, hardware has changed forcing assumptions and existing practices to be revisited, it might be more efficient to use a delta encoding and a single integer value based upon epoch when taking into account calculation and conversion time. I am not sure what the correct answer is without testing.
Clive:
The delta would probably be marginally better as only 1 pmap entry would be necessary.
Hanno:
I agree, you could even combine the two to have a smaller integer value for msec since midnight instead of epoch. We do not need to define a single method, rather a toolbox of options to be chosen from. Greg made the case for ISE where members want to pick up the binary version whereas CME has stuck to the string representation and omitted the punctuation marks. The latter will benefit more from a concatenation approach than the former but there is no need to rule one or the other out. As you said, this is up to testing to determine the most efficient way.
The language to describe such conversions is what is needed for any of this. It is only needed in the context of standard messages where datatypes, field orders and field content are typically defined from a non-technical perspective. I believe this is a good thing. However, it does not preclude optimization on a technical level which is what FAST is intended for.
[ original email was from Greg Orsini - greg.orsini@orcsoftware.com ]
One clarification on the CME timestamp.
CME converts FIX time to an integer by removing punctuation. As a result, delta encoding can make timestamp quite efficient.
Greg.
Following a conversation on calculated fields, the discussion moved to
discussing time representation: Hanno: Another example for the
concatenation is again the UTCTimestamp datatype in FIX. If it can be
split into date+time on the wire, then date can be coded very
efficiently as a constant if it does not change during the session,
requiring only time to be sent on the wire. The receiving end can put
the two fields back together to create a valid FIX datatype again.JimN: I like the idea of splitting time up and using a smaller integer
value for the time component. This has been a common practice in market
data for quite some time. However, hardware has changed forcing
assumptions and existing practices to be revisited, it might be more
efficient to use a delta encoding and a single integer value based upon
epoch when taking into account calculation and conversion time. I am not
sure what the correct answer is without testing.Clive:The delta would probably be marginally better as only 1 pmap entry
would be necessary.Hanno: I agree, you could even combine the two to have a smaller
integer value for msec since midnight instead of epoch. We do not need
to define a single method, rather a toolbox of options to be chosen
from. Greg made the case for ISE where members want to pick up the
binary version whereas CME has stuck to the string representation and
omitted the punctuation marks. The latter will benefit more from a
concatenation approach than the former but there is no need to rule one
or the other out. As you said, this is up to testing to determine the
most efficient way.The language to describe such conversions is what is needed for any of
this. It is only needed in the context of standard messages where
datatypes, field orders and field content are typically defined from a
non-technical perspective. I believe this is a good thing. However, it
does not preclude optimization on a technical level which is what FAST
is intended for.
I propose we use the the FAST Scaled Decimal type to represent the FIX UTC Timestamp, with the mantissa value representing days since the UNIX epoch, and the exponent portion representing time since midnight (midnight epoch). By varying the precision of the scaling factor, we can represent a time granularity from seconds (exponent value = 0) to microseconds (exponent value = 11) and beyond. This is an efficient method as UTC Time Only and UTC Date Only can be efficiently represented when a full timestamp is not needed.
Decoding/Encoding rules:
UTC Date Only will have a exponent value of 0, indicating there is no time portion encoded
UTC Timestamp with seconds resolution will have an exponent value of 5 to cover the range of 0-86,400 seconds in a day
UTC Timestamp with milliseconds resolution will have an exponent value of 8 to cover the range of 0-86,400,000 milliseconds in a day
UTC Timestamp with microseconds resolution will have an exponent value of 11 to cover the range of 0-86,400,000,000 microseconds in a day
UTC Timestamp with nanosecond resolution will have an exponent value of 14 to cover the range of 0-86,400,000,000,000 nanoseconds in a day
UTC Time only will follow the rules above based on resolution required, but will carry no days. So a value decoded >=0 < 1.0 is considered a day. Negative numbers could be used to define timestamps before the UNIX epoch.
Here are some examples:
UTC Timestamp with seconds resolution: 2008-Mar-28 13:26:59
Days Unix epoch: 13,966
Seconds since Midnight epoch: 48419 (86,400 seconds in a day)
FAST Scaled Decimal Representation: 13,866.48419 (mantissa 13966648419, exponent 5)
FAST Bytes on Wire: 6
UTC Timestamp with seconds resolution: 13:26:59
Days Unix epoch: 0
Seconds since Midnight epoch: 48419
FAST Scaled Decimal Representation: 0.48419 (mantissa 48419, exponent 5)
FAST Bytes on Wire: 4
UTC Timestamp with seconds resolution: 2008-Mar-28
Days Unix epoch: 13,966
Seconds since Midnight epoch: 0 (86,400 seconds in a day)
FAST Scaled Decimal Representation: 13966 (mantissa 13966, exponent 0)
FAST Bytes on Wire: 4
UTC Timestamp with milliseconds resolution: 2008-Mar-28 13:26:59.720
Days Unix epoch: 13,966
Milliseconds since Midnight epoch: 48419720
FAST Scaled Decimal Representation: 13,866.48419720 (mantissa 13966648419720, exponent 8)
FAST Bytes on Wire: 7
UTC Timestamp with milliseconds resolution: 13:26:59.720
Days Unix epoch: 0
Seconds since Midnight epoch: 48419720
FAST Scaled Decimal Representation: 0.48419720 (mantissa 48419720, exponent 8)
FAST Bytes on Wire: 5
Things to consider:
- Alternate epochs ? (Epoch - Wikipedia)
- Encoding efficiencies when using default exponent values will reduce the above stated FAST bytes on wire.
- Delta and Copy encoding will add significant wire size reduction based on specific implementations.
- Disclaimer, check my math above !
Daniel May
daniel@spryware.com
One clarification on the CME timestamp.
CME converts FIX time to an integer by removing punctuation. As a
result, delta encoding can make timestamp quite efficient.Greg.
Following a conversation on calculated fields, the discussion moved to
discussing time representation: Hanno: Another example for the
concatenation is again the UTCTimestamp datatype in FIX. If it can be
split into date+time on the wire, then date can be coded very
efficiently as a constant if it does not change during the session,
requiring only time to be sent on the wire. The receiving end can put
the two fields back together to create a valid FIX datatype again.JimN: I like the idea of splitting time up and using a smaller integer
value for the time component. This has been a common practice in
market data for quite some time. However, hardware has changed forcing
assumptions and existing practices to be revisited, it might be more
efficient to use a delta encoding and a single integer value based
upon epoch when taking into account calculation and conversion time. I
am not sure what the correct answer is without testing.Clive:The delta would probably be marginally better as only 1 pmap
entry would be necessary.Hanno: I agree, you could even combine the two to have a smaller
integer value for msec since midnight instead of epoch. We do not need
to define a single method, rather a toolbox of options to be chosen
from. Greg made the case for ISE where members want to pick up the
binary version whereas CME has stuck to the string representation and
omitted the punctuation marks. The latter will benefit more from a
concatenation approach than the former but there is no need to rule
one or the other out. As you said, this is up to testing to determine
the most efficient way.The language to describe such conversions is what is needed for any of
this. It is only needed in the context of standard messages where
datatypes, field orders and field content are typically defined from a
non-technical perspective. I believe this is a good thing. However, it
does not preclude optimization on a technical level which is what FAST
is intended for.
I think we should benchmark this from an encoding/decoding efficiency as well as compaction standpoint and compare vs an integer based approach.
While the use of scaled decimals is cool and allows the precision to be determined every time a timestamp field is passed, it should be possible to get better performance by making the precision a property at the template level. It may also be a bit easier on the receiver if it can make a decision on how to handle the field ahead of time rather than having to be prepared to select the native type for the field every time it is handled.
There is some computational and bit overhead using a scaled decimal over an integer based approach but I haven’t yet figured out if/when it will give better compaction?
Kind Regards,
Anders
I propose we use the the FAST Scaled Decimal type to represent the FIX
UTC Timestamp, with the mantissa value representing days since the UNIX
epoch, and the exponent portion representing time since midnight
(midnight epoch). By varying the precision of the scaling factor, we can
represent a time granularity from seconds (exponent value = 0) to
microseconds (exponent value = 11) and beyond. This is an efficient
method as UTC Time Only and UTC Date Only can be efficiently represented
when a full timestamp is not needed.Decoding/Encoding rules: UTC Date Only will have a exponent value of 0,
indicating there is no time portion encoded UTC Timestamp with seconds
resolution will have an exponent value of 5 to cover the range of 0-
86,400 seconds in a day UTC Timestamp with milliseconds resolution will
have an exponent value of 8 to cover the range of 0-86,400,000
milliseconds in a day UTC Timestamp with microseconds resolution will
have an exponent value of 11 to cover the range of 0-86,400,000,000
microseconds in a day UTC Timestamp with nanosecond resolution will have
an exponent value of 14 to cover the range of 0-86,400,000,000,000
nanoseconds in a dayUTC Time only will follow the rules above based on resolution required,
but will carry no days. So a value decoded >=0 < 1.0 is considered a
day. Negative numbers could be used to define timestamps before the
UNIX epoch.Here are some examples: UTC Timestamp with seconds resolution: 2008-Mar-
28 13:26:59 Days Unix epoch: 13,966 Seconds since Midnight epoch: 48419
(86,400 seconds in a day) FAST Scaled Decimal Representation:
13,866.48419 (mantissa 13966648419, exponent 5) FAST Bytes on Wire: 6UTC Timestamp with seconds resolution: 13:26:59 Days Unix epoch: 0
Seconds since Midnight epoch: 48419 FAST Scaled Decimal Representation:
0.48419 (mantissa 48419, exponent 5) FAST Bytes on Wire: 4UTC Timestamp with seconds resolution: 2008-Mar-28 Days Unix epoch:
13,966 Seconds since Midnight epoch: 0 (86,400 seconds in a day) FAST
Scaled Decimal Representation: 13966 (mantissa 13966, exponent 0) FAST
Bytes on Wire: 4UTC Timestamp with milliseconds resolution: 2008-Mar-28 13:26:59.720
Days Unix epoch: 13,966 Milliseconds since Midnight epoch: 48419720 FAST
Scaled Decimal Representation: 13,866.48419720 (mantissa 13966648419720,
exponent 8) FAST Bytes on Wire: 7UTC Timestamp with milliseconds resolution: 13:26:59.720 Days Unix
epoch: 0 Seconds since Midnight epoch: 48419720 FAST Scaled Decimal
Representation: 0.48419720 (mantissa 48419720, exponent 8) FAST Bytes
on Wire: 5Things to consider:
- Alternate epochs ?
(Epoch - Wikipedia)- Encoding efficiencies when using default exponent values will reduce
the above stated FAST bytes on wire.- Delta and Copy encoding will add significant wire size reduction
based on specific implementations.- Disclaimer, check my math above !
Daniel May daniel@spryware.com
One clarification on the CME timestamp.
CME converts FIX time to an integer by removing punctuation. As a
result, delta encoding can make timestamp quite efficient.Greg.
Following a conversation on calculated fields, the discussion moved
to discussing time representation: Hanno: Another example for the
concatenation is again the UTCTimestamp datatype in FIX. If it can
be split into date+time on the wire, then date can be coded very
efficiently as a constant if it does not change during the session,
requiring only time to be sent on the wire. The receiving end can
put the two fields back together to create a valid FIX datatype
again.JimN: I like the idea of splitting time up and using a smaller
integer value for the time component. This has been a common
practice in market data for quite some time. However, hardware has
changed forcing assumptions and existing practices to be revisited,
it might be more efficient to use a delta encoding and a single
integer value based upon epoch when taking into account calculation
and conversion time. I am not sure what the correct answer is
without testing.Clive:The delta would probably be marginally better as only 1 pmap
entry would be necessary.Hanno: I agree, you could even combine the two to have a smaller
integer value for msec since midnight instead of epoch. We do not
need to define a single method, rather a toolbox of options to be
chosen from. Greg made the case for ISE where members want to pick
up the binary version whereas CME has stuck to the string
representation and omitted the punctuation marks. The latter will
benefit more from a concatenation approach than the former but there
is no need to rule one or the other out. As you said, this is up to
testing to determine the most efficient way.The language to describe such conversions is what is needed for any
of this. It is only needed in the context of standard messages where
datatypes, field orders and field content are typically defined from
a non-technical perspective. I believe this is a good thing.
However, it does not preclude optimization on a technical level
which is what FAST is intended for.
Based on feedback from Greg Maynard from the ISE and Jim Northy, there may be a need for two approaches. The “simple” approach would be a FAST unsigned using milliseconds since the UNIX epoch. If the value is <= 86,400,000, then it is assumed to be a UTC Time only. UTC Date only would simply encode the days since epoch converted to milliseconds. My original scaled decimal approach can always be used by those looking for higher precision or better wire compaction for certain situations.
I would also have to do the testing on the scaled decimal in terms of efficiency, but it seems that if you can “default” the mantissa and have a value that is easily copy or delta encoded to a very small size. The approach provides the smallest number representation for both date and time, allowing the user to use only the precision needed.
Daniel
I think we should benchmark this from an encoding/decoding efficiency as
well as compaction standpoint and compare vs an integer based approach.
While the use of scaled decimals is cool and allows the precision to be
determined every time a timestamp field is passed, it should be possible
to get better performance by making the precision a property at the
template level. It may also be a bit easier on the receiver if it can
make a decision on how to handle the field ahead of time rather than
having to be prepared to select the native type for the field every time
it is handled. There is some computational and bit overhead using a
scaled decimal over an integer based approach but I haven’t yet figured
out if/when it will give better compaction?Kind Regards, Anders
I propose we use the the FAST Scaled Decimal type to represent the FIX
UTC Timestamp, with the mantissa value representing days since the
UNIX epoch, and the exponent portion representing time since midnight
(midnight epoch). By varying the precision of the scaling factor, we
can represent a time granularity from seconds (exponent value = 0) to
microseconds (exponent value = 11) and beyond. This is an efficient
method as UTC Time Only and UTC Date Only can be efficiently
represented when a full timestamp is not needed.Decoding/Encoding rules: UTC Date Only will have a exponent value of
0, indicating there is no time portion encoded UTC Timestamp with
seconds resolution will have an exponent value of 5 to cover the range
of 0- 86,400 seconds in a day UTC Timestamp with milliseconds
resolution will have an exponent value of 8 to cover the range of 0-
86,400,000 milliseconds in a day UTC Timestamp with microseconds
resolution will have an exponent value of 11 to cover the range of 0-
86,400,000,000 microseconds in a day UTC Timestamp with nanosecond
resolution will have an exponent value of 14 to cover the range of 0-
86,400,000,000,000 nanoseconds in a dayUTC Time only will follow the rules above based on resolution
required, but will carry no days. So a value decoded >=0 < 1.0 is
considered a day. Negative numbers could be used to define timestamps
before the UNIX epoch.Here are some examples: UTC Timestamp with seconds resolution: 2008-Mar-
28 13:26:59 Days Unix epoch: 13,966 Seconds since Midnight epoch:
48419 (86,400 seconds in a day) FAST Scaled Decimal Representation:
13,866.48419 (mantissa 13966648419, exponent 5) FAST Bytes on Wire: 6UTC Timestamp with seconds resolution: 13:26:59 Days Unix epoch:
0 Seconds since Midnight epoch: 48419 FAST Scaled Decimal
Representation: 0.48419 (mantissa 48419, exponent 5) FAST Bytes
on Wire: 4UTC Timestamp with seconds resolution: 2008-Mar-28 Days Unix epoch:
13,966 Seconds since Midnight epoch: 0 (86,400 seconds in a day) FAST
Scaled Decimal Representation: 13966 (mantissa 13966, exponent 0) FAST
Bytes on Wire: 4UTC Timestamp with milliseconds resolution: 2008-Mar-28 13:26:59.720
Days Unix epoch: 13,966 Milliseconds since Midnight epoch: 48419720
FAST Scaled Decimal Representation: 13,866.48419720 (mantissa
13966648419720, exponent 8) FAST Bytes on Wire: 7UTC Timestamp with milliseconds resolution: 13:26:59.720 Days Unix
epoch: 0 Seconds since Midnight epoch: 48419720 FAST Scaled Decimal
Representation: 0.48419720 (mantissa 48419720, exponent 8) FAST Bytes
on Wire: 5Things to consider:
- Alternate epochs ?
(Epoch - Wikipedia)- Encoding efficiencies when using default exponent values will
reduce the above stated FAST bytes on wire.- Delta and Copy encoding will add significant wire size reduction
based on specific implementations.- Disclaimer, check my math above !
Daniel May daniel@spryware.com
One clarification on the CME timestamp.
CME converts FIX time to an integer by removing punctuation. As a
result, delta encoding can make timestamp quite efficient.Greg.
Following a conversation on calculated fields, the discussion
moved to discussing time representation: Hanno: Another example
for the concatenation is again the UTCTimestamp datatype in FIX.
If it can be split into date+time on the wire, then date can be
coded very efficiently as a constant if it does not change during
the session, requiring only time to be sent on the wire. The
receiving end can put the two fields back together to create a
valid FIX datatype again.JimN: I like the idea of splitting time up and using a smaller
integer value for the time component. This has been a common
practice in market data for quite some time. However, hardware has
changed forcing assumptions and existing practices to be
revisited, it might be more efficient to use a delta encoding and
a single integer value based upon epoch when taking into account
calculation and conversion time. I am not sure what the correct
answer is without testing.Clive:The delta would probably be marginally better as only 1 pmap
entry would be necessary.Hanno: I agree, you could even combine the two to have a smaller
integer value for msec since midnight instead of epoch. We do not
need to define a single method, rather a toolbox of options to be
chosen from. Greg made the case for ISE where members want to pick
up the binary version whereas CME has stuck to the string
representation and omitted the punctuation marks. The latter will
benefit more from a concatenation approach than the former but
there is no need to rule one or the other out. As you said, this
is up to testing to determine the most efficient way.The language to describe such conversions is what is needed for
any of this. It is only needed in the context of standard messages
where datatypes, field orders and field content are typically
defined from a non-technical perspective. I believe this is a good
thing. However, it does not preclude optimization on a technical
level which is what FAST is intended for.