Imported from previous forum
Hi,
Here is our feedback on the fast 1.2 extension proposal.
Our comments are obviously biased by our ns latency concern.
New type definition syntax: OK
Enumeration: OK
Set: OK
Bit Group: OK
PMAP field placement: OK
For the Bit group and the PMAP field placement, special care should be taken to properly define how they interact with the PMAP and the operators. Default and copy are probably the only safe operators usable with Bit groups, enum and set anyway.
The PMAP fields should be restricted to a few bits only. For an uInt7 there is nothing to gain by putting it into the PMAP and for uInt6 you only gain 1 bit. This will be very good though for bits or very small enum/sets though.
Timestamp: OK.
Map is really not OK!
1- It breaks the FAST conceptual model where both the sender and receiver share a pre-defined set of templates.
With this, you will also need a long term context and a large memory. There are already a context used for the operators like copy/inc/etc. but it’s a short term one generally limited in scope to one multicast frame for instance and also its memory use is known and small.
2 - As it’s a dynamic construct, it can’t be optimized at compile time and so will be slow.
3- It will not work reliably. FAST is used a lot in multicast and unicast market data diffusion. So what do you do when you loose a packet?
How do you know if you are in sync with the sender?
When you start to process a feed, how do you get the current map? A giant snapshot?
As consecutive IDs are not mandatory, you can’t use an array and so you will need to use an hash table (or trees, skip-lists, whatever) and this will use a lot of memory and will be SLOW. So when I read “only a number of thousand values” I get really scared for the latency. 
So IMO this should not be added to FAST as it’s really not fast at all. If really it is added to FAST, let’s at least force it to use consecutive IDs.
Note that a pre-shared map is just a big enum and that is OK of course.
Best regards,
Marc
Marc,
thank you for your feedback.
I have commented below on your comments about the map operator.
At least one major feed, the ISE depth feed, includes a mapping of instrument identifiers. I think it is worth while to try to determine if this mapping should occur above the FAST layer or we should try to include a mapping feature into FAST.
That said, I’m currently leaning towards sending the map operator proposal back to the drawing board. I believe we need to spend more time analyzing different alternatives for a mapping feature and to discuss more with the implementors.
Your
Best,
Rolf
Hi,
Here is our feedback on the fast 1.2 extension proposal.
Our comments are obviously biased by our ns latency concern.New type definition syntax: OK
Enumeration: OK
Set: OK
Bit Group: OK
PMAP field placement: OKFor the Bit group and the PMAP field placement, special care
should be taken to properly define how they interact with the
PMAP and the operators. Default and copy are probably the only
safe operators usable with Bit groups, enum and set anyway.The PMAP fields should be restricted to a few bits only.
For an uInt7 there is nothing to gain by putting it into the
PMAP and for uInt6 you only gain 1 bit. This will be very good
though for bits or very small enum/sets though.Timestamp: OK.
Map is really not OK!
1 - It breaks the FAST conceptual model where both the sender
and receiver share a pre-defined set of templates. With this,
you will also need a long term context and a large memory.
There are already a context used for the operators like
copy/inc/etc. but it’s a short term one generally limited in
scope to one multicast frame for instance and also its memory
use is known and small.
I don’t see how the map operator breaks the conceptual model.
The sender and the receiver will still share the same templates.
The map operator may however make use of a large “context memory” to store previously defined mappings. This differs from previous FAST constructs and may present a challenge in resource constrained environments.
The basic use of the map is obviously within the same context as all other operators. In the case of a multicast feed the lifetime of a mapping must either be one frame or mappings after a lost frame will be impossible to interpret correctly. It is entirely possible to recover from this situation, much as current feeds offer ways of recovering from lost frames when reconstructing a full orderbook from incremental updates. Both the ISE and the CME provide recovery mechanism that are designed to work above the FAST level.
Furthermore, the ISE has a construct that is similar to the map construct, but the ISE construct is defined above the FAST level. The ISE template defines a “full refresh” message that contains a mapping between a complete but verbose instrument identifier and a smaller numeric identifier. The defined “incremental refresh” message contains the numeric identifier only. If a frame is lost you’ll need to wait for a complete “full refresh” cycle in order to be sure that you have complete and correct information available.
In the case of the ISE it seems to be implied that mappings don’t change and identifiers don’t get reused during a day and this provides an opportunity to make simplifying assumptions about the validity of instrument id mappings.
2 - As it’s a dynamic construct, it can’t be optimized at
compile time and so will be slow.
See above. I don’t see that it is more dynamic than other operators, but it requires more memory and a longer access code path to look up values in the mapping context (possibly some sort of hash table)
3- It will not work reliably. FAST is used a lot in multicast
and unicast market data diffusion. So what do you do when you
loose a packet? How do you know if you are in sync with the
sender? When you start to process a feed, how do you get the
current map? A giant snapshot? As consecutive IDs are not
mandatory, you can’t use an array and so you will need to use
an hash table (or trees, skip-lists, whatever) and this will
use a lot of memory and will be SLOW. So when I read “only a
number of thousand values” I get really scared for the latency.
For example if you want short instrument identifiers instead of a more verbose symbol, you have to do the translation somewhere. Question is if we’re ready to commit to placing this feature in the FAST layer or if it better to leave it to upper layer processing.
I think this is analogous to maintaining a complete orderbook with a feed that provides incremental refresh messages without a guaranteed strict delivery order (as in the case with multicast feeds).
(On a side-note: If you look at the execution path in the next processing step it may be better (performance wise) to have numeric instrument identifiers rather than more verbose string identifiers. So it could well be that you would want a FAST decoder to provide you with the numeric ids rather than the string variety. In the case that the feed doesn’t present numeric ids you may even want the decoder to produce them …)
So IMO this should not be added to FAST as it’s really not fast
at all.
If really it is added to FAST, let’s at least force it to use
consecutive IDs.Note that a pre-shared map is just a big enum and that is OK of
course.Best regards,
Marc
Rolf,
I agree with you that very often the exchange use a mapping between the contracts names and integer contracts IDs but they do it out of band on a separate multicast channel or messages with specific templates and with an application level protocol. At least this is what EUREX EBS does. ISE uses the snapshot messages to give the IDs. These mappings are valid for the day and an important point is that the market data messages use these integers IDs but do not convert them to the long ascii names. So the whole chain can work with integers IDs and never have to go back to the text names.
The proposed map element would make the FAST decoder replace the short integer IDs by the textual names and so the applications down the chain will have to rehash it again slowing down the order book fetch for instance.
BTW this is also what you said in your reply below so I guess we agree on this. 
Best Regards,
Marc
Marc,
thank you for your feedback.
I have commented below on your comments about the map operator.
At least one major feed, the ISE depth feed, includes a mapping of
instrument identifiers. I think it is worth while to try to determine if
this mapping should occur above the FAST layer or we should try to
include a mapping feature into FAST.That said, I’m currently leaning towards sending the map operator
proposal back to the drawing board. I believe we need to spend more time
analyzing different alternatives for a mapping feature and to discuss
more with the implementors.Your
Best, Rolf
Hi,
Here is our feedback on the fast 1.2 extension proposal. Our comments
are obviously biased by our ns latency concern.New type definition syntax: OK Enumeration: OK Set: OK Bit Group: OK
PMAP field placement: OKFor the Bit group and the PMAP field placement, special care should be
taken to properly define how they interact with the PMAP and the
operators. Default and copy are probably the only safe operators
usable with Bit groups, enum and set anyway.The PMAP fields should be restricted to a few bits only. For an uInt7
there is nothing to gain by putting it into the PMAP and for uInt6 you
only gain 1 bit. This will be very good though for bits or very small
enum/sets though.Timestamp: OK.
Map is really not OK!
1 - It breaks the FAST conceptual model where both the sender and
receiver share a pre-defined set of templates. With this, you will
also need a long term context and a large memory. There are already a
context used for the operators like copy/inc/etc. but it’s a short
term one generally limited in scope to one multicast frame for
instance and also its memory use is known and small.I don’t see how the map operator breaks the conceptual model. The sender
and the receiver will still share the same templates.The map operator may however make use of a large “context memory” to
store previously defined mappings. This differs from previous FAST
constructs and may present a challenge in resource constrained
environments.The basic use of the map is obviously within the same context as all
other operators. In the case of a multicast feed the lifetime of a
mapping must either be one frame or mappings after a lost frame will be
impossible to interpret correctly. It is entirely possible to recover
from this situation, much as current feeds offer ways of recovering from
lost frames when reconstructing a full orderbook from incremental
updates. Both the ISE and the CME provide recovery mechanism that are
designed to work above the FAST level.Furthermore, the ISE has a construct that is similar to the map
construct, but the ISE construct is defined above the FAST level. The
ISE template defines a “full refresh” message that contains a mapping
between a complete but verbose instrument identifier and a smaller
numeric identifier. The defined “incremental refresh” message contains
the numeric identifier only. If a frame is lost you’ll need to wait for
a complete “full refresh” cycle in order to be sure that you have
complete and correct information available.In the case of the ISE it seems to be implied that mappings don’t change
and identifiers don’t get reused during a day and this provides an
opportunity to make simplifying assumptions about the validity of
instrument id mappings.2 - As it’s a dynamic construct, it can’t be optimized at compile time
and so will be slow.See above. I don’t see that it is more dynamic than other operators, but
it requires more memory and a longer access code path to look up values
in the mapping context (possibly some sort of hash table)3- It will not work reliably. FAST is used a lot in multicast and
unicast market data diffusion. So what do you do when you loose a
packet? How do you know if you are in sync with the sender? When you
start to process a feed, how do you get the current map? A giant
snapshot? As consecutive IDs are not mandatory, you can’t use an array
and so you will need to use an hash table (or trees, skip-lists,
whatever) and this will use a lot of memory and will be SLOW. So when
I read “only a number of thousand values” I get really scared for the
latency.For example if you want short instrument identifiers instead of a more
verbose symbol, you have to do the translation somewhere. Question is if
we’re ready to commit to placing this feature in the FAST layer or if it
better to leave it to upper layer processing.I think this is analogous to maintaining a complete orderbook with a
feed that provides incremental refresh messages without a guaranteed
strict delivery order (as in the case with multicast feeds).(On a side-note: If you look at the execution path in the next
processing step it may be better (performance wise) to have numeric
instrument identifiers rather than more verbose string identifiers. So
it could well be that you would want a FAST decoder to provide you with
the numeric ids rather than the string variety. In the case that the
feed doesn’t present numeric ids you may even want the decoder to
produce them …)So IMO this should not be added to FAST as it’s really not fast at
all. If really it is added to FAST, let’s at least force it to use
consecutive IDs.Note that a pre-shared map is just a big enum and that is OK of
course.Best regards,
Marc
Hello gentlemen,
I thought I’d chime in and support Marc’s comments here, and also add that the application level protocol is not only a good idea, but rather mandatory.
Here’s a couple of examples that I think justify this attitude:
- problems in the instruments’ definitions and integer id mapping can invalidate the good-for-the-day mappings and require a refresh of the information; in practice this happens a couple of times a year at most;
- exchange backend failure/failover can trigger a refresh of this referential information;
- instrument lifecycle is a bit more complex, as instruments can be introduced and removed during the day; a good example are strategies (combos).
In short, the point is that the issue of mapping is probably best left off to the specific application.
All the best, kind regards,
Radu
Rolf,
I agree with you that very often the exchange use a mapping between the
contracts names and integer contracts IDs but they do it out of band on
a separate multicast channel or messages with specific templates and
with an application level protocol. At least this is what EUREX EBS
does. ISE uses the snapshot messages to give the IDs. These mappings are
valid for the day and an important point is that the market data
messages use these integers IDs but do not convert them to the long
ascii names. So the whole chain can work with integers IDs and never
have to go back to the text names. The proposed map element would make
the FAST decoder replace the short integer IDs by the textual names and
so the applications down the chain will have to rehash it again slowing
down the order book fetch for instance.BTW this is also what you said in your reply below so I guess we agree
on this.Best Regards,
Marc
Marc,
thank you for your feedback.
I have commented below on your comments about the map operator.
At least one major feed, the ISE depth feed, includes a mapping of
instrument identifiers. I think it is worth while to try to determine
if this mapping should occur above the FAST layer or we should try to
include a mapping feature into FAST.That said, I’m currently leaning towards sending the map operator
proposal back to the drawing board. I believe we need to spend more
time analyzing different alternatives for a mapping feature and to
discuss more with the implementors.Your
Best, Rolf
Hi,
Here is our feedback on the fast 1.2 extension proposal. Our
comments are obviously biased by our ns latency concern.New type definition syntax: OK Enumeration: OK Set: OK Bit Group: OK
PMAP field placement: OKFor the Bit group and the PMAP field placement, special care should
be taken to properly define how they interact with the PMAP and the
operators. Default and copy are probably the only safe operators
usable with Bit groups, enum and set anyway.The PMAP fields should be restricted to a few bits only. For an
uInt7 there is nothing to gain by putting it into the PMAP and for
uInt6 you only gain 1 bit. This will be very good though for bits or
very small enum/sets though.Timestamp: OK.
Map is really not OK!
1 - It breaks the FAST conceptual model where both the sender and
receiver share a pre-defined set of templates. With this, you will
also need a long term context and a large memory. There are already
a context used for the operators like copy/inc/etc. but it’s a short
term one generally limited in scope to one multicast frame for
instance and also its memory use is known and small.I don’t see how the map operator breaks the conceptual model. The
sender and the receiver will still share the same templates.The map operator may however make use of a large “context memory” to
store previously defined mappings. This differs from previous FAST
constructs and may present a challenge in resource constrained
environments.The basic use of the map is obviously within the same context as all
other operators. In the case of a multicast feed the lifetime of a
mapping must either be one frame or mappings after a lost frame will
be impossible to interpret correctly. It is entirely possible to
recover from this situation, much as current feeds offer ways of
recovering from lost frames when reconstructing a full orderbook from
incremental updates. Both the ISE and the CME provide recovery
mechanism that are designed to work above the FAST level.Furthermore, the ISE has a construct that is similar to the map
construct, but the ISE construct is defined above the FAST level. The
ISE template defines a “full refresh” message that contains a mapping
between a complete but verbose instrument identifier and a smaller
numeric identifier. The defined “incremental refresh” message contains
the numeric identifier only. If a frame is lost you’ll need to wait
for a complete “full refresh” cycle in order to be sure that you have
complete and correct information available.In the case of the ISE it seems to be implied that mappings don’t
change and identifiers don’t get reused during a day and this provides
an opportunity to make simplifying assumptions about the validity of
instrument id mappings.2 - As it’s a dynamic construct, it can’t be optimized at compile
time and so will be slow.See above. I don’t see that it is more dynamic than other operators,
but it requires more memory and a longer access code path to look up
values in the mapping context (possibly some sort of hash table)3- It will not work reliably. FAST is used a lot in multicast and
unicast market data diffusion. So what do you do when you loose a
packet? How do you know if you are in sync with the sender? When you
start to process a feed, how do you get the current map? A giant
snapshot? As consecutive IDs are not mandatory, you can’t use an
array and so you will need to use an hash table (or trees, skip-
lists, whatever) and this will use a lot of memory and will be SLOW.
So when I read “only a number of thousand values” I get really
scared for the latency.For example if you want short instrument identifiers instead of a more
verbose symbol, you have to do the translation somewhere. Question is
if we’re ready to commit to placing this feature in the FAST layer or
if it better to leave it to upper layer processing.I think this is analogous to maintaining a complete orderbook with a
feed that provides incremental refresh messages without a guaranteed
strict delivery order (as in the case with multicast feeds).(On a side-note: If you look at the execution path in the next
processing step it may be better (performance wise) to have numeric
instrument identifiers rather than more verbose string identifiers. So
it could well be that you would want a FAST decoder to provide you
with the numeric ids rather than the string variety. In the case that
the feed doesn’t present numeric ids you may even want the decoder to
produce them …)So IMO this should not be added to FAST as it’s really not fast at
all. If really it is added to FAST, let’s at least force it to use
consecutive IDs.Note that a pre-shared map is just a big enum and that is OK of
course.Best regards,
Marc
