Imported from previous forum
Hi everyone,
what is the best way to broadcast template structure to FAST clients?
There are couple of reasons why it is important to broadcast template structure periodically:
-
Template structure might change while current clients are still receiving data with the old structure
-
Late-joiners do not know the structure of data until they receive template structure.
I thinking of using Fast SCP 1.1 but I also would like to avoid sending the same template information over the network too often.
thanks,
Dimitry
I thinking of using Fast SCP 1.1 but I also would like to avoid sending
the same template information over the network too often.
Hi Dimitri
What do you call too often?
I would think that templates are fairly static objects.
Think about what would occur if templates were broadcast on the fly?
There would be syncronization problems hard to solve, such as a distributed consensus, or atomic broadcast (total order).
For FAST assumes the current set of templates is common knowledge that is, everybody knows the set of templates, and everybody knows that everybody knows the set of templates, and so forth.
I doubt that SCP is handling this problems, hence, there must be a priori rules for changing current set of templates.
You should not worry about update frequencies.
The change rate of templates is very much application dependent. Some applications will have templates as static as the application protocol they are encoding. Otheres will change once a month or every other week. Yet others will change templates midstream to gain higher performance (better compression and/or lower latency) or to minimize manual administration and deployment overhead.
Since FAST is fairly new and SCP even newer, it is likely that early implementations will lean to the static side. However, as the technology matures and more implementations become available, the opportunities to use the dynamic features of SCP increases.
FAST actually says nothing about templates being common knowledge. The only thing it says is that a template must be known to the decoder before it decodes the first instance of that template. Templates may very well be short lived creatures, even generated on the fly, and only used in a single session between two peers.
In addition to defining the actual messages to encode templates, SCP also defines template interchange for stream oriented scenarios with reliable delivery (think TCP). Though, you’re correct in that it doesn’t define a template exchange model for unreliable, datagram oriented transports (think UDP). The reason is that multicast applications present a number of implementation strategies where none is superior in general. Different types of applications may have different types of optimal template interchange schemes.
There are challenges in multicast template distribution but they are not harder to solve than the similar challenges you have for the application data you intend to distribute.
And in the long run, I’d rather have the software solve the template synchronization problem for me than relying on manual intervention, e.g. perodically check for new templates on some web site and make sure I run on the latest and greatest.
/David
I thinking of using Fast SCP 1.1 but I also would like to avoid
sending the same template information over the network too often.Hi Dimitri
What do you call too often?
I would think that templates are fairly static objects. Think about what
would occur if templates were broadcast on the fly? There would be
syncronization problems hard to solve, such as a distributed consensus,
or atomic broadcast (total order).For FAST assumes the current set of templates is common knowledge that
is, everybody knows the set of templates, and everybody knows that
everybody knows the set of templates, and so forth.I doubt that SCP is handling this problems, hence, there must be a
priori rules for changing current set of templates.You should not worry about update frequencies.
Cheers for this David – that was very useful.
I agree with the fact that as the technology matures, FAST actors will move towards more dynamic templates “diffusion”.
I like even more the fact that templates might even be generated on the fly, for the lifetime of a session!!
Yet when this is the case, there is going to be situations where a message encoded in T_(i+1) is received by an decoding agent before receiving template T_i.
It might be nice that SCP provides with similar guaranties, i.e., without specifying a template dissemination protocol, yet provides its users with some sort of consistency guaranties.
And I agree that SCP and encoding/decoding are orthogonal matters beyond the fact that encoding and decoding are based on a template ID – whatever that template id means.
That’s something I’m interested in and have some experience from past jobs 
The change rate of templates is very much application dependent. Some
applications will have templates as static as the application protocol
they are encoding. Otheres will change once a month or every other week.
Yet others will change templates midstream to gain higher performance
(better compression and/or lower latency) or to minimize manual
administration and deployment overhead.Since FAST is fairly new and SCP even newer, it is likely that early
implementations will lean to the static side. However, as the technology
matures and more implementations become available, the opportunities to
use the dynamic features of SCP increases.FAST actually says nothing about templates being common knowledge. The
only thing it says is that a template must be known to the decoder
before it decodes the first instance of that template. Templates may
very well be short lived creatures, even generated on the fly, and only
used in a single session between two peers.In addition to defining the actual messages to encode templates, SCP
also defines template interchange for stream oriented scenarios with
reliable delivery (think TCP). Though, you’re correct in that it doesn’t
define a template exchange model for unreliable, datagram oriented
transports (think UDP). The reason is that multicast applications
present a number of implementation strategies where none is superior in
general. Different types of applications may have different types of
optimal template interchange schemes.There are challenges in multicast template distribution but they are not
harder to solve than the similar challenges you have for the application
data you intend to distribute.And in the long run, I’d rather have the software solve the template
synchronization problem for me than relying on manual intervention, e.g.
perodically check for new templates on some web site and make sure I run
on the latest and greatest./David
I thinking of using Fast SCP 1.1 but I also would like to avoid
sending the same template information over the network too often.Hi Dimitri
What do you call too often?
I would think that templates are fairly static objects. Think about
what would occur if templates were broadcast on the fly? There would
be syncronization problems hard to solve, such as a distributed
consensus, or atomic broadcast (total order).For FAST assumes the current set of templates is common knowledge
that is, everybody knows the set of templates, and everybody knows
that everybody knows the set of templates, and so forth.I doubt that SCP is handling this problems, hence, there must be a
priori rules for changing current set of templates.You should not worry about update frequencies.
SCP supports versioning through the use of template names.
You can view the template id as a (potentially short-lived) descriptor and the template name as the persistent id. This is described to some extent in the SCP spec.
By not reusing template ids in the short time span, your receivers can then detect any “new” templates (ie. templates that they have not seen a definition for). The normal case would obviously be that receivers see the definitions before the first message referencing the new template id.
Template Ids need to be reused fairly soon or you may run out of one byte ids (1-119). If you accept two byte ids (128-15999) then you can create a fair amount new template without running out of available dynamic template values.
The remaining problem is then to make sure that a receiver recognizes and acts on template id rollover. Implementors need to make sure that senders receivers agree on the current meaning of a template id (ie. the current association between a template name and a template id). It is still an instance of the reliable messaging problem, but with drastically lower latency requirements.
I personally don’t think it should be mandated in the spec but rather documented as a practice. Let’s continue this discussion. I’m sure other organizations will follow in Morgan Stanley’s footsteps and our analysis now will hopefully help others in future implementations.
Thx,
Rolf
Cheers for this David – that was very useful.
I agree with the fact that as the technology matures, FAST actors
will move towards more dynamic templates “diffusion”.I like even more the fact that templates might even be generated on
the fly, for the lifetime of a session!!Yet when this is the case, there is going to be situations where a
message encoded in T_(i+1) is received by an decoding agent before
receiving template T_i.It might be nice that SCP provides with similar guaranties, i.e.,
without specifying a template dissemination protocol, yet provides
its users with some sort of consistency guaranties.And I agree that SCP and encoding/decoding are orthogonal matters
beyond the fact that encoding and decoding are based on a template
ID – > whatever that template id means.That’s something I’m interested in and have some experience from
past jobs
The remaining problem is then to make sure that a receiver recognizes
and acts on template id rollover. Implementors need to make sure that
senders receivers agree on the current meaning of a template id (ie. the
current association between a template name and a template id). It is
still an instance of the reliable messaging problem, but with
drastically lower latency requirements.
That’s precisely the hard part:
IMHO this is more than an instance of the Reliable messaging problem – because it involves some ordering considerations.
Given a sender and a possibly large number of receivers: no receiver should deliver a message prior to delivering its related template.
e.g.,
… [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1] …
Not sure that’s an instance of the distributed consensus, but rather instance of the virtual synchrony problem. That is, something (SCP) should guarantee that template changes within a group of receiver, are observed in the same order by all receivers. Furthermore, template changes in SCP must be totally ordered with respect to all regular FAST messages.
BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3 third item one should read “…defined in 6.1” and not “defined in 5.1”
I couldn’t find mention of this in the SCP spec.
I personally don’t think it should be mandated in the spec but rather
documented as a practice. Let’s continue this discussion. I’m sure other
organizations will follow in Morgan Stanley’s footsteps and our analysis
now will hopefully help others in future implementations.Thx, Rolf
Cheers for this David – that was very useful.
I agree with the fact that as the technology matures, FAST actors will
move towards more dynamic templates “diffusion”.I like even more the fact that templates might even be generated on
the fly, for the lifetime of a session!!Yet when this is the case, there is going to be situations where a
message encoded in T_(i+1) is received by an decoding agent before
receiving template T_i.It might be nice that SCP provides with similar guaranties, i.e.,
without specifying a template dissemination protocol, yet provides its
users with some sort of consistency guaranties.And I agree that SCP and encoding/decoding are orthogonal matters
beyond the fact that encoding and decoding are based on a template ID
– > whatever that template id means.That’s something I’m interested in and have some experience from
past jobs
I was maybe a bit terse.
By using immutable template ids (in the short term) you can relax the requirements on synchrony between template definitions and messages using template ids. As a receiver, you still need a new template definition before you can decode a message with the new template id, but the existing definitions don’t change (within a time window).
This allows for a lower overhead template distribution solution, which is what Dimitry was asking about, I think.
An example;
Let’s assume you have a template named “foo/1” where 1 is the version of the template. Further let’s assume the Template Name “foo/1” is associated with the Id 7. Now, if you want to modify this template (extend or change), you can create “foo/2” which differs in some aspect from “foo/1” and, depending on other Ids in use your first unused Id is 27, so you associate the name “foo/2” with the id 27.
Now receivers may see different representations of a message type. Template id 7 will still refer to the old layout and id 27 will be unknown until receivers see / retrieve the definition for id 27. The important difference is that you can relax strict ordering as id 7 is immutable for some period of time before it may be reused. Not seeing the new definition of id 27 can be detected.
You still need a level of synchronization between templates and messages in order to consume messages with newly created templates, but you don’t need a strict ordering; new definitions can be sent out earlier as there is no (short-term) change to the meaning of an existing template.
There needs to be additional synchronization if several sources are allowed to allocate template ids, but that is a separate issue.
/Rolf
The remaining problem is then to make sure that a receiver
recognizes and acts on template id rollover. Implementors
need to make sure that senders receivers agree on the current
meaning of a template id (ie. the current association between
a template name and a template id). It is still an instance of
the reliable messaging problem, but with drastically lower
latency requirements.That’s precisely the hard part: IMHO this is more than an instance of
the Reliable messaging problem – because it involves some ordering
considerations. Given a sender and a possibly large number of
receivers: no receiver should deliver a message prior to delivering its
related template.
e.g., … [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1] …Not sure that’s an instance of the distributed consensus, but rather
instance of the virtual synchrony problem. That is, something (SCP)
should guarantee that template changes within a group of receiver, are
observed in the same order by all receivers. Furthermore, template
changes in SCP must be totally ordered with respect to all regular
FAST messages.BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3 third
item one should read “…defined in 6.1” and not “defined in 5.1”I couldn’t find mention of this in the SCP spec.
Rolf,
I have a related question. Is there a way to send a new message without having sent a new template definition, i.e. if I add a field to my message, is there a way for the receiver to ignore that new field until he receives a new template definition from me, perhaps by appending it to the end of my message and having a placeholder in my initial template definition? Or is it a strict requirement to always send a new template definition prior to sending a message for it.
Thanks,
Hanno.
I was maybe a bit terse.
By using immutable template ids (in the short term) you can relax the
requirements on synchrony between template definitions and messages
using template ids. As a receiver, you still need a new template
definition before you can decode a message with the new template id, but
the existing definitions don’t change (within a time window).This allows for a lower overhead template distribution solution, which
is what Dimitry was asking about, I think.An example; Let’s assume you have a template named “foo/1” where 1 is
the version of the template. Further let’s assume the Template Name
“foo/1” is associated with the Id 7. Now, if you want to modify this
template (extend or change), you can create “foo/2” which differs in
some aspect from “foo/1” and, depending on other Ids in use your first
unused Id is 27, so you associate the name “foo/2” with the id 27.Now receivers may see different representations of a message type.
Template id 7 will still refer to the old layout and id 27 will be
unknown until receivers see / retrieve the definition for id 27. The
important difference is that you can relax strict ordering as id 7 is
immutable for some period of time before it may be reused. Not seeing
the new definition of id 27 can be detected.You still need a level of synchronization between templates and messages
in order to consume messages with newly created templates, but you don’t
need a strict ordering; new definitions can be sent out earlier as there
is no (short-term) change to the meaning of an existing template.There needs to be additional synchronization if several sources are
allowed to allocate template ids, but that is a separate issue./Rolf
Hanno, this is an interesting problem.
The short answer is no. There is no way for the receiver to detect that there is another field in the message. As you mention, you could put a placeholder into the template, maybe an optional field at the end of the template. This would maybe work in some instances, but you would be restricted in choice of operator etc.
We did discuss this some when we were updating the spec from 1.0 to 1.1, but nothing came out of that discussion if I recall correctly.
It’s worth discussing again, though. Anyone has ideas on this?
/Rolf
Rolf,
I have a related question. Is there a way to send a new message
without having sent a new template definition, i.e. if I add a
field to my message, is there a way for the receiver to ignore
that new field until he receives a new template definition from
me, perhaps by appending it to the end of my message and having
a placeholder in my initial template definition? Or is it a
strict requirement to always send a new template definition
prior to sending a message for it.Thanks, Hanno.
[ original email was from Darshan Khedekar - darshan.khedekar.ext@deutsche-boerse.com ]
Hi,
IMO dissemination of templates to the receivers over a unrealiable channel such as multicast is not a very good idea. Because even if they detect that the new message they receive is not as per the template they are working with the receivers stay blocked.
A reliable stream with some form of acknowledgment should be used to ensure that all the parties actually receive the templates.
The parties are free to use the channel whether FTP or connection stream to get the most recent copy of the template.
The sender should ensure that the messages using changed templates structure are sent only when most of the receivers actually are aware of the change. (version number can be used to identify which template is being used for the sent message)
IMO if the sender starts disseminating the new message without anyone aware about it it breaks the motive of electronic communication.
Regards,
Darshan
Hanno, this is an interesting problem.
The short answer is no. There is no way for the receiver to detect that
there is another field in the message. As you mention, you could put a
placeholder into the template, maybe an optional field at the end of the
template. This would maybe work in some instances, but you would be
restricted in choice of operator etc.We did discuss this some when we were updating the spec from 1.0 to 1.1,
but nothing came out of that discussion if I recall correctly.It’s worth discussing again, though. Anyone has ideas on this?
/Rolf
Rolf,
I have a related question. Is there a way to send a new message
without having sent a new template definition, i.e. if I add a field
to my message, is there a way for the receiver to ignore that new
field until he receives a new template definition from me, perhaps by
appending it to the end of my message and having a placeholder in my
initial template definition? Or is it a strict requirement to always
send a new template definition prior to sending a message for it.Thanks, Hanno.
[ original email was from Greg Orsini - greg.orsini@orcsoftware.com ]
Hanno,
There is no way to know how long the message is without the correct template, so a new message with an unknown or ‘extended’ template can’t be fully decoded (a result of the decision to not include a message length in FAST or SCP).
Perhaps an optional length for messages requiring a new template would maintain the integrity of the transport layer without affecting the message contents.
Then you could create a revised template (instead of 7 it would 7.1?) that could have additional fields that would be ignored until you get a hold of the new template.
Greg.
Hanno, this is an interesting problem.
The short answer is no. There is no way for the receiver to detect that
there is another field in the message. As you mention, you could put a
placeholder into the template, maybe an optional field at the end of the
template. This would maybe work in some instances, but you would be
restricted in choice of operator etc.We did discuss this some when we were updating the spec from 1.0 to 1.1,
but nothing came out of that discussion if I recall correctly.It’s worth discussing again, though. Anyone has ideas on this?
/Rolf
Rolf,
I have a related question. Is there a way to send a new message
without having sent a new template definition, i.e. if I add a field
to my message, is there a way for the receiver to ignore that new
field until he receives a new template definition from me, perhaps by
appending it to the end of my message and having a placeholder in my
initial template definition? Or is it a strict requirement to always
send a new template definition prior to sending a message for it.Thanks, Hanno.
May be we could create a special relation between templates – like “Russian nesting dolls”, or template inheritance.
If template T3 ← T2 ← T1
then if message M3 has been T3-encoded, it then can be decoded by T2 and or T1.
What you get when you decode M3 following T2 or T1 is not specified.
That would solve Hanno’s problem. That would also allow the notion of template specialization etc.
What do you think???
Hanno,
There is no way to know how long the message is without the correct
template, so a new message with an unknown or ‘extended’ template can’t
be fully decoded (a result of the decision to not include a message
length in FAST or SCP).Perhaps an optional length for messages requiring a new template would
maintain the integrity of the transport layer without affecting the
message contents.Then you could create a revised template (instead of 7 it would 7.1?)
that could have additional fields that would be ignored until you get a
hold of the new template.Greg.
Hanno, this is an interesting problem.
The short answer is no. There is no way for the receiver to detect
that there is another field in the message. As you mention, you could
put a placeholder into the template, maybe an optional field at the
end of the template. This would maybe work in some instances, but you
would be restricted in choice of operator etc.We did discuss this some when we were updating the spec from 1.0 to
1.1, but nothing came out of that discussion if I recall correctly.It’s worth discussing again, though. Anyone has ideas on this?
/Rolf
Rolf,
I have a related question. Is there a way to send a new message
without having sent a new template definition, i.e. if I add a
field to my message, is there a way for the receiver to ignore that
new field until he receives a new template definition from me,
perhaps by appending it to the end of my message and having a
placeholder in my initial template definition? Or is it a strict
requirement to always send a new template definition prior to
sending a message for it.Thanks, Hanno.
Not sure I understand what you are proposing.
Could you please elaborate?
/Rolf
May be we could create a special relation between templates –
like “Russian nesting dolls”, or template inheritance.If template T3 ← T2 ← T1 then if message M3 has been T3-encoded,
it then can be decoded by T2 and or T1.What you get when you decode M3 following T2 or T1 is not specified.
That would solve Hanno’s problem. That would also allow the notion of
template specialization etc.What do you think???
Sure.
Establish by spec a relationship (inheritance) between templates noted ←
Say for example that T1 ← T2
then if message M has been encoded in T2, then it can be decoded both with T2 and T1.
As an analogy C++ inheritance, and pointer to base.
does it make sense?
Not sure I understand what you are proposing. Could you please
elaborate?/Rolf
May be we could create a special relation between templates – like
“Russian nesting dolls”, or template inheritance.If template T3 ← T2 ← T1 then if message M3 has been T3-encoded,
it then can be decoded by T2 and or T1.What you get when you decode M3 following T2 or T1 is not specified.
That would solve Hanno’s problem. That would also allow the notion of
template specialization etc.What do you think???
Hm, I’m still not sure how this should be used.
If you have both T1 and T2 then you can use the right template to decode. If you don’t have T2 then you’re in trouble as T1 doesn’t contain the info need? Am I missing something?
/Rolf
Sure.
Establish by spec a relationship (inheritance) between templates noted
←Say for example that T1 ← T2 then if message M has been encoded in T2,
then it can be decoded both with T2 and T1.As an analogy C++ inheritance, and pointer to base.
does it make sense?
Not sure I understand what you are proposing. Could you please
elaborate?/Rolf
May be we could create a special relation between templates – like
“Russian nesting dolls”, or template inheritance.If template T3 ← T2 ← T1 then if message M3 has been T3-encoded,
it then can be decoded by T2 and or T1.What you get when you decode M3 following T2 or T1 is not specified.
That would solve Hanno’s problem. That would also allow the notion
of template specialization etc.What do you think???
No.
Given T1 ← T2 and M encoded with T2
If I have both T1 & T2, I can chose either.
If I don’t need the extra data added per T2, I can chose to use T1.
By definition of T1 ← T2 then a message encoded with T2 must be decodable by T1.
If you decode M with T1, then you will “slice” your message somewhat, but that might be something you want, or still better that nothing when you don’t have T2.
This would allow senders to “evolve” their templates, giving the choice to receiver to use or not this “evolution”
Hm, I’m still not sure how this should be used. If you have both T1 and
T2 then you can use the right template to decode. If you don’t have T2
then you’re in trouble as T1 doesn’t contain the info need? Am I missing
something?/Rolf
Sure.
Establish by spec a relationship (inheritance) between templates
noted ←Say for example that T1 ← T2 then if message M has been encoded in
T2, then it can be decoded both with T2 and T1.As an analogy C++ inheritance, and pointer to base.
does it make sense?
Not sure I understand what you are proposing. Could you please
elaborate?/Rolf
May be we could create a special relation between templates –
like “Russian nesting dolls”, or template inheritance.If template T3 ← T2 ← T1 then if message M3 has been T3-
encoded, it then can be decoded by T2 and or T1.What you get when you decode M3 following T2 or T1 is not
specified.That would solve Hanno’s problem. That would also allow the notion
of template specialization etc.What do you think???
← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to carry the length so that slicing can be operated properly
No.
Given T1 ← T2 and M encoded with T2
If I have both T1 & T2, I can chose either.
If I don’t need the extra data added per T2, I can chose to use T1.
By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.If you decode M with T1, then you will “slice” your message somewhat,
but that might be something you want, or still better that nothing when
you don’t have T2.This would allow senders to “evolve” their templates, giving the choice
to receiver to use or not this “evolution”Hm, I’m still not sure how this should be used. If you have both T1
and T2 then you can use the right template to decode. If you don’t
have T2 then you’re in trouble as T1 doesn’t contain the info need? Am
I missing something?/Rolf
Sure.
Establish by spec a relationship (inheritance) between templates
noted ←Say for example that T1 ← T2 then if message M has been encoded in
T2, then it can be decoded both with T2 and T1.As an analogy C++ inheritance, and pointer to base.
does it make sense?
Not sure I understand what you are proposing. Could you please
elaborate?/Rolf
May be we could create a special relation between templates –
like “Russian nesting dolls”, or template inheritance.If template T3 ← T2 ← T1 then if message M3 has been T3-
encoded, it then can be decoded by T2 and or T1.What you get when you decode M3 following T2 or T1 is not
specified.That would solve Hanno’s problem. That would also allow the
notion of template specialization etc.What do you think???
our posts crossed. So you’re suggesting a message length to handle added fields. So what about field operator changes, added fields to repeating groups, etc?
← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to
carry the length so that slicing can be operated properlyNo.
Given T1 ← T2 and M encoded with T2
If I have both T1 & T2, I can chose either.
If I don’t need the extra data added per T2, I can chose to use T1.
By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.If you decode M with T1, then you will “slice” your message somewhat,
but that might be something you want, or still better that nothing
when you don’t have T2.This would allow senders to “evolve” their templates, giving the
choice to receiver to use or not this “evolution”Hm, I’m still not sure how this should be used. If you have both T1
and T2 then you can use the right template to decode. If you don’t
have T2 then you’re in trouble as T1 doesn’t contain the info need?
Am I missing something?/Rolf
Field operator changes: not possible, for that wouldn’t be a IS_A relation.
Added fields to repeating groups: same as above.
T2 would be T1 + something so T2 is a T1.
Although
our posts crossed. So you’re suggesting a message length to handle
added fields. So what about field operator changes, added fields to
repeating groups, etc?← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to carry
the length so that slicing can be operated properlyNo.
Given T1 ← T2 and M encoded with T2
If I have both T1 & T2, I can chose either.
If I don’t need the extra data added per T2, I can chose to use T1.
By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.If you decode M with T1, then you will “slice” your message somewhat,
but that might be something you want, or still better that nothing
when you don’t have T2.This would allow senders to “evolve” their templates, giving the
choice to receiver to use or not this “evolution”Hm, I’m still not sure how this should be used. If you have both T1
and T2 then you can use the right template to decode. If you don’t
have T2 then you’re in trouble as T1 doesn’t contain the info need?
Am I missing something?/Rolf