Fast SCP

[hanno.klein]

First of all thanks to everybody for chipping in and commenting on my problem.

The most interesting case (which we have had in the past) is the addition of a field that is linked to functionality which can optionally be used by the member. It would be great if a member would not be forced to update the template and simply ignore the new field(s). This is the only way he can avoid retesting of his application. If the member needs the new functionality, he has to test this anyway and then a new template is not an issue.

For more complex changes such as field deletions, operator changes, data type changes, I do not see an actual benefit in terms of being able to skip re-testing.

Regards,
Hanno.

our posts crossed. So you’re suggesting a message length to handle
added fields. So what about field operator changes, added fields to
repeating groups, etc?

← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to carry
the length so that slicing can be operated properly

No.

Given T1 ← T2 and M encoded with T2

If I have both T1 & T2, I can chose either.

If I don’t need the extra data added per T2, I can chose to use T1.

By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.

If you decode M with T1, then you will “slice” your message somewhat,
but that might be something you want, or still better that nothing
when you don’t have T2.

This would allow senders to “evolve” their templates, giving the
choice to receiver to use or not this “evolution”

Hm, I’m still not sure how this should be used. If you have both T1
and T2 then you can use the right template to decode. If you don’t
have T2 then you’re in trouble as T1 doesn’t contain the info need?
Am I missing something?

/Rolf

[jsulmon_x]

Hence the “is a” / inheritance relation I was suggesting. T2 must be a super-set of T1

First of all thanks to everybody for chipping in and commenting on
my problem.

The most interesting case (which we have had in the past) is the
addition of a field that is linked to functionality which can optionally
be used by the member. It would be great if a member would not be forced
to update the template and simply ignore the new field(s). This is the
only way he can avoid retesting of his application. If the member needs
the new functionality, he has to test this anyway and then a new
template is not an issue.

For more complex changes such as field deletions, operator changes, data
type changes, I do not see an actual benefit in terms of being able to
skip re-testing.

Regards, Hanno.

our posts crossed. So you’re suggesting a message length to handle
added fields. So what about field operator changes, added fields to
repeating groups, etc?

← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to carry
the length so that slicing can be operated properly

No.

Given T1 ← T2 and M encoded with T2

If I have both T1 & T2, I can chose either.

If I don’t need the extra data added per T2, I can chose to use T1.

By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.

If you decode M with T1, then you will “slice” your message
somewhat, but that might be something you want, or still better
that nothing when you don’t have T2.

This would allow senders to “evolve” their templates, giving the
choice to receiver to use or not this “evolution”

Hm, I’m still not sure how this should be used. If you have both
T1 and T2 then you can use the right template to decode. If you
don’t have T2 then you’re in trouble as T1 doesn’t contain the
info need? Am I missing something?

/Rolf

[rolfandersson58_x]

FAST 1.1 section 10 (transfer Encoding) specifies that a FAST stream consists of a sequence of messages or a stream of blocks. A block is a sequence of one or more messages. One of the reasons for including blocks into the std was to enable skipping of a sequence of messages.

The block construct could be used to make it possible to add fields at the end of a message. there is still a question of how operators may be used for added fields. The current rules for pmap content must be reviewed and possibly updated.

/Rolf

First of all thanks to everybody for chipping in and commenting on
my problem.

The most interesting case (which we have had in the past) is the
addition of a field that is linked to functionality which can optionally
be used by the member. It would be great if a member would not be forced
to update the template and simply ignore the new field(s). This is the
only way he can avoid retesting of his application. If the member needs
the new functionality, he has to test this anyway and then a new
template is not an issue.

For more complex changes such as field deletions, operator changes, data
type changes, I do not see an actual benefit in terms of being able to
skip re-testing.

Regards, Hanno.

our posts crossed. So you’re suggesting a message length to handle
added fields. So what about field operator changes, added fields to
repeating groups, etc?

← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to carry
the length so that slicing can be operated properly

No.

Given T1 ← T2 and M encoded with T2

If I have both T1 & T2, I can chose either.

If I don’t need the extra data added per T2, I can chose to use T1.

By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.

If you decode M with T1, then you will “slice” your message
somewhat, but that might be something you want, or still better
that nothing when you don’t have T2.

This would allow senders to “evolve” their templates, giving the
choice to receiver to use or not this “evolution”

Hm, I’m still not sure how this should be used. If you have both
T1 and T2 then you can use the right template to decode. If you
don’t have T2 then you’re in trouble as T1 doesn’t contain the
info need? Am I missing something?

/Rolf

[rolfandersson58_x]

oops, should be “sequence of blocks” not “stream of blocks”

FAST 1.1 section 10 (transfer Encoding) specifies that a FAST stream
consists of a sequence of messages or a stream of blocks. A block is a
sequence of one or more messages. One of the reasons for including
blocks into the std was to enable skipping of a sequence of messages.

The block construct could be used to make it possible to add fields at
the end of a message. there is still a question of how operators may be
used for added fields. The current rules for pmap content must be
reviewed and possibly updated.

/Rolf

First of all thanks to everybody for chipping in and commenting on my
problem.

The most interesting case (which we have had in the past) is the
addition of a field that is linked to functionality which can
optionally be used by the member. It would be great if a member would
not be forced to update the template and simply ignore the new
field(s). This is the only way he can avoid retesting of his
application. If the member needs the new functionality, he has to test
this anyway and then a new template is not an issue.

For more complex changes such as field deletions, operator changes,
data type changes, I do not see an actual benefit in terms of being
able to skip re-testing.

Regards, Hanno.

our posts crossed. So you’re suggesting a message length to handle
added fields. So what about field operator changes, added fields
to repeating groups, etc?

← would be a is_a relationship.
i/o implement this one would need , as suggested by Hanno, to
carry the length so that slicing can be operated properly

No.

Given T1 ← T2 and M encoded with T2

If I have both T1 & T2, I can chose either.

If I don’t need the extra data added per T2, I can chose to use
T1.

By definition of T1 ← T2 then a message encoded with T2 must
be decodable by T1.

If you decode M with T1, then you will “slice” your message
somewhat, but that might be something you want, or still better
that nothing when you don’t have T2.

This would allow senders to “evolve” their templates, giving the
choice to receiver to use or not this “evolution”

Hm, I’m still not sure how this should be used. If you have both
T1 and T2 then you can use the right template to decode. If you
don’t have T2 then you’re in trouble as T1 doesn’t contain the
info need? Am I missing something?

/Rolf

[rolfandersson58_x]

OK, I believe this would require a different wire coding.
AFAICT T1 in the current wire coding doesn’t contain enough information to decode a msg encoded with T2 if T2 contains any changed ops or added fields.

What are your thoughts on that? Could you provide an example?

/Rolf

No.

Given T1 ← T2 and M encoded with T2

If I have both T1 & T2, I can chose either.

If I don’t need the extra data added per T2, I can chose to use T1.

By definition of T1 ← T2 then a message encoded with T2 must be
decodable by T1.

If you decode M with T1, then you will “slice” your message somewhat,
but that might be something you want, or still better that nothing when
you don’t have T2.

This would allow senders to “evolve” their templates, giving the choice
to receiver to use or not this “evolution”

Hm, I’m still not sure how this should be used. If you have both T1
and T2 then you can use the right template to decode. If you don’t
have T2 then you’re in trouble as T1 doesn’t contain the info need? Am
I missing something?

/Rolf

Sure.

Establish by spec a relationship (inheritance) between templates
noted ←

Say for example that T1 ← T2 then if message M has been encoded in
T2, then it can be decoded both with T2 and T1.

As an analogy C++ inheritance, and pointer to base.

does it make sense?

Not sure I understand what you are proposing. Could you please
elaborate?

/Rolf

May be we could create a special relation between templates –
like “Russian nesting dolls”, or template inheritance.

If template T3 ← T2 ← T1 then if message M3 has been T3-
encoded, it then can be decoded by T2 and or T1.

What you get when you decode M3 following T2 or T1 is not
specified.

That would solve Hanno’s problem. That would also allow the
notion of template specialization etc.

What do you think???

[davidrosenborg30_x]

I don’t think we should label such a device inheritance and I think it should be sliced in another way if we in the end decide that the added complexity is worth the effort.

First, let’s summarize a few scenarios:

1. Fully Dynamic - The most generic way of solving template evolution is to use non mutating TID and SCP Level 3 (i.e, dynamic template exchange). The base for this is already defined, but additional work is required for multicast applications.

2. Non Mutating TID and Accessible Extension - You want to add new data without modifying or sending out a new template. You also want all receivers to be able to discover and use the new data. To solve this you’d have to tack on an explicit extension field. It could look something like

The benefits of this soultion is that it is already available. The downside is that it is not particular flexible, it is intrusive, and it requires explicit processing at the application level.

3. Mutating TID and Opaque Extension - You have different versions of the same template (same identifier). The versions are compatible. Receivers only knowing of V1 will simply ignore content available to those receivers that use V2.

Scenario 3 is what I beleive you tried to solve with your inheritance proposal. This is an outline of how I’d do it:

Add a new template construct named ‘extension’:

The semantics of this would be that messages of all these three templates would be fully interchangable (with a few constraints, see below). That is, the templates are both backward and forward compatible. V1 - V3 decoders can decode any message encoded with V1 - V3.

The fine print:

The wire representation would need a new predefined template identifier in the reserved range. Let’s use 121 for now. The definition of 121 would look something like this.

The predefined template is assumed to be known in advanced by any decoder supporting the feature.

When an encoder encodes an section, it wraps it in an Extension message:

PMAP TID=121 LEN [… encoded extension section …]

If fields in the extension section must utlilize bits in a presence map, the encoded extension section starts with PMAP.

Encoding backward compatibility: Let’s say a V1 decoder receives an message encoded with V2. The decoder first reads the V1 fields (all it knows). The next thing in the stream is the extension secition, which starts with a PMAP and the tid 121. It recognizes this as an Extension message and just skips it.

Encoding forward compatibility: Let’s say a V2 decoder receives an message encoded with V1. The decoder first read the non-extension part. It then looks for the next message in the stream. Since this message is not the 121 - Extension message, it knows the extension section is not present. The additional constraint here is that the fields in the extension section must be optional in the corresponding application type.

For this to work in general, there are few constraints that must be met:

• elements may only appear as the last components of a top level template (after resolving any static template refereces).

• Any field operator in an extension section must either specify no dictionary or must specify the special dictionary “extension”. An extension dictionary is local to a template and a specific extension section. This way we avoid that operators in extension sections pollute dictionaries seen by ordinary fields (otherwise it would be dangerous to skip 121 messages).

• The 121 Extension messsage must not be used in any other situation than as a result of encoding a template with an section. If this constraint is not met, the forward compatibility feature won’t work.

• Dynamic template references are not allowed in an extension section. There are two reasons: They’d pollute the previous value of the implicit copy operator for TIDs, and the actual contained templates could possible pollute arbitrary dictionaries. The dictionary constraint described above should be possible to enforce by static examination of a template.

This solution might look complex at first but wouldn’t be particulary hard to implement. The simplest level of support would be for a decoder to recognize the 121 tid and know how to skip that message.

/David

May be we could create a special relation between templates – like
“Russian nesting dolls”, or template inheritance.

If template T3 ← T2 ← T1 then if message M3 has been T3-encoded, it
then can be decoded by T2 and or T1.

What you get when you decode M3 following T2 or T1 is not specified.

That would solve Hanno’s problem. That would also allow the notion of
template specialization etc.

What do you think???

Hanno,

There is no way to know how long the message is without the correct
template, so a new message with an unknown or ‘extended’ template
can’t be fully decoded (a result of the decision to not include a
message length in FAST or SCP).

Perhaps an optional length for messages requiring a new template would
maintain the integrity of the transport layer without affecting the
message contents.

Then you could create a revised template (instead of 7 it would 7.1?)
that could have additional fields that would be ignored until you get
a hold of the new template.

Greg.

Hanno, this is an interesting problem.

The short answer is no. There is no way for the receiver to detect
that there is another field in the message. As you mention, you
could put a placeholder into the template, maybe an optional field
at the end of the template. This would maybe work in some instances,
but you would be restricted in choice of operator etc.

We did discuss this some when we were updating the spec from 1.0 to
1.1, but nothing came out of that discussion if I recall correctly.

It’s worth discussing again, though. Anyone has ideas on this?

/Rolf

Rolf,

I have a related question. Is there a way to send a new message
without having sent a new template definition, i.e. if I add a
field to my message, is there a way for the receiver to ignore
that new field until he receives a new template definition from
me, perhaps by appending it to the end of my message and having a
placeholder in my initial template definition? Or is it a strict
requirement to always send a new template definition prior to
sending a message for it.

Thanks, Hanno.

[jacobnorthey_x]

I think the easiest approach is to create a FAST best practices guide that includes instructions for achieving support for multiple message template versions at the same time. Summarizing what has been said before, this would require:

1. Length prefixed blocks
2. Fields cannot be added to sequences
3. Fields can only be added to a group if the group is the last field instruction in a template and the added fields do not affect the groups use of a presence map (e.g. a group that does not need a presence map can only have fields added that don’t require entries in the presence map).
4. New fields cannot use operators that would affect shared dictionary values.

Jake

I don’t think we should label such a device inheritance and I think it
should be sliced in another way if we in the end decide that the added
complexity is worth the effort.

First, let’s summarize a few scenarios:

1. Fully Dynamic - The most generic way of solving template evolution is
   to use non mutating TID and SCP Level 3 (i.e, dynamic template
   exchange). The base for this is already defined, but additional work is
   required for multicast applications.

2. Non Mutating TID and Accessible Extension - You want to add new data
   without modifying or sending out a new template. You also want all
   receivers to be able to discover and use the new data. To solve this
   you’d have to tack on an explicit extension field. It could look
   something like

The benefits of this soultion is that it is already available. The
downside is that it is not particular flexible, it is intrusive, and it
requires explicit processing at the application level.

3. Mutating TID and Opaque Extension - You have different versions of
   the same template (same identifier). The versions are compatible.
   Receivers only knowing of V1 will simply ignore content available to
   those receivers that use V2.

Scenario 3 is what I beleive you tried to solve with your inheritance
proposal. This is an outline of how I’d do it:

Add a new template construct named ‘extension’:

The semantics of this would be that messages of all these three
templates would be fully interchangable (with a few constraints, see
below). That is, the templates are both backward and forward compatible.
V1 - V3 decoders can decode any message encoded with V1 - V3.

The fine print:

The wire representation would need a new predefined template identifier
in the reserved range. Let’s use 121 for now. The definition of 121
would look something like this.

The predefined template is assumed to be known in advanced by any
decoder supporting the feature.

When an encoder encodes an section, it wraps it in an
Extension message:

PMAP TID=121 LEN [… encoded extension section …]

If fields in the extension section must utlilize bits in a presence map,
the encoded extension section starts with PMAP.

Encoding backward compatibility: Let’s say a V1 decoder receives an
message encoded with V2. The decoder first reads the V1 fields (all it
knows). The next thing in the stream is the extension secition, which
starts with a PMAP and the tid 121. It recognizes this as an Extension
message and just skips it.

Encoding forward compatibility: Let’s say a V2 decoder receives an
message encoded with V1. The decoder first read the non-extension
part. It then looks for the next message in the stream. Since this
message is not the 121 - Extension message, it knows the extension
section is not present. The additional constraint here is that the
fields in the extension section must be optional in the corresponding
application type.

For this to work in general, there are few constraints that must be met:

• elements may only appear as the last components of a top
  level template (after resolving any static template refereces).

• Any field operator in an extension section must either specify no
  dictionary or must specify the special dictionary “extension”. An
  extension dictionary is local to a template and a specific extension
  section. This way we avoid that operators in extension sections
  pollute dictionaries seen by ordinary fields (otherwise it would be
  dangerous to skip 121 messages).

• The 121 Extension messsage must not be used in any other situation
  than as a result of encoding a template with an section.
  If this constraint is not met, the forward compatibility feature
  won’t work.

• Dynamic template references are not allowed in an extension section.
  There are two reasons: They’d pollute the previous value of the
  implicit copy operator for TIDs, and the actual contained templates
  could possible pollute arbitrary dictionaries. The dictionary
  constraint described above should be possible to enforce by static
  examination of a template.

This solution might look complex at first but wouldn’t be particulary
hard to implement. The simplest level of support would be for a decoder
to recognize the 121 tid and know how to skip that message.

/David

May be we could create a special relation between templates – like
“Russian nesting dolls”, or template inheritance.

If template T3 ← T2 ← T1 then if message M3 has been T3-encoded,
it then can be decoded by T2 and or T1.

What you get when you decode M3 following T2 or T1 is not specified.

That would solve Hanno’s problem. That would also allow the notion of
template specialization etc.

What do you think???

Hanno,

There is no way to know how long the message is without the correct
template, so a new message with an unknown or ‘extended’ template
can’t be fully decoded (a result of the decision to not include a
message length in FAST or SCP).

Perhaps an optional length for messages requiring a new template
would maintain the integrity of the transport layer without
affecting the message contents.

Then you could create a revised template (instead of 7 it would
7.1?) that could have additional fields that would be ignored until
you get a hold of the new template.

Greg.

Hanno, this is an interesting problem.

The short answer is no. There is no way for the receiver to detect
that there is another field in the message. As you mention, you
could put a placeholder into the template, maybe an optional field
at the end of the template. This would maybe work in some
instances, but you would be restricted in choice of operator etc.

We did discuss this some when we were updating the spec from
1.0 to
1.1, but nothing came out of that discussion if I recall
correctly.

It’s worth discussing again, though. Anyone has ideas on this?

/Rolf

Rolf,

I have a related question. Is there a way to send a new message
without having sent a new template definition, i.e. if I add a
field to my message, is there a way for the receiver to ignore
that new field until he receives a new template definition from
me, perhaps by appending it to the end of my message and having
a placeholder in my initial template definition? Or is it a
strict requirement to always send a new template definition
prior to sending a message for it.

Thanks, Hanno.

[davidrosenborg30_x]

Yes, it would work in some situations, but it requires you to use blocks in general which might not be desired. Furthermore, you’d have to restrict a block to contain exactly one message (at least in situations where you know you’re using extended templates (as percieved by the receiver, which might not always be possible to know)).

Also, the constraint list would have to be extended to prohibit the bytes that constitute the extension from looking like the start of a message, which is rather tricky in the general case. It’s probably solvable by always adding two pad bytes at the start of the extension the first always having the value 0xc0, and the second having the value of some unused tid.

This last constraint is not necessary if you enforce the one message per block constraint for all messages. But if a decoder make use of this constraint it will suddenly behave in a non standard way.

/David

I think the easiest approach is to create a FAST best practices guide
that includes instructions for achieving support for multiple message
template versions at the same time. Summarizing what has been said
before, this would require:

1. Length prefixed blocks
2. Fields cannot be added to sequences
3. Fields can only be added to a
   group if the group is the last field instruction in a template and
   the added fields do not affect the groups use of a presence map (e.g.
   a group that does not need a presence map can only have fields added
   that don’t require entries in the presence map).
4. New fields cannot use operators that would affect shared
   dictionary values.

Jake

[fplpo_x]

[ original email was from Greg Orsini - greg.orsini@orcsoftware.com ]
As I mentioned earlier, an optional length prefix to the message would easily ensure message separation without the block stuff. Would it be simpler?

Greg.

Yes, it would work in some situations, but it requires you to use blocks
in general which might not be desired. Furthermore, you’d have to
restrict a block to contain exactly one message (at least in situations
where you know you’re using extended templates (as percieved by the
receiver, which might not always be possible to know)).

Also, the constraint list would have to be extended to prohibit the
bytes that constitute the extension from looking like the start of a
message, which is rather tricky in the general case. It’s probably
solvable by always adding two pad bytes at the start of the extension
the first always having the value 0xc0, and the second having the value
of some unused tid.

This last constraint is not necessary if you enforce the one message per
block constraint for all messages. But if a decoder make use of this
constraint it will suddenly behave in a non standard way.

/David

I think the easiest approach is to create a FAST best practices guide
that includes instructions for achieving support for multiple message
template versions at the same time. Summarizing what has been said
before, this would require:

1. Length prefixed blocks
2. Fields cannot be added to sequences
3. Fields can only be added to a group if the group is the last field
   instruction in a template and the added fields do not affect the
   groups use of a presence map (e.g. a group that does not need a
   presence map can only have fields added that don’t require entries
   in the presence map).
4. New fields cannot use operators that would affect shared dictionary
   values.

Jake

[davidrosenborg30_x]

I’m not sure I understand. What do you mean by optional? Optional per message or optional per application? If you mean per message, how do you discover the absence of a length preamble on a particular message?

/David

As I mentioned earlier, an optional length prefix to the message
would easily ensure message separation without the block stuff. Would
it be simpler?

Greg.

Yes, it would work in some situations, but it requires you to
use blocks in general which might not be desired. Furthermore,
you’d have to restrict a block to contain exactly one message
(at least in situations where you know you’re using extended
templates (as percieved by the receiver, which might not always
be possible to know)).

Also, the constraint list would have to be extended to prohibit the
bytes that constitute the extension from looking like the start of a
message, which is rather tricky in the general case. It’s probably
solvable by always adding two pad bytes at the start of the extension
the first always having the value 0xc0, and the second having the
value of some unused tid.

This last constraint is not necessary if you enforce the one message
per block constraint for all messages. But if a decoder make use of
this constraint it will suddenly behave in a non standard way.

/David

I think the easiest approach is to create a FAST best practices
guide that includes instructions for achieving support for multiple
message template versions at the same time. Summarizing what has
been said before, this would require:

1. Length prefixed blocks
2. Fields cannot be added to sequences
3. Fields can only be added to a group if the group is the last
   field instruction in a template and the added fields do not
   affect the groups use of a presence map (e.g. a group that does
   not need a presence map can only have fields added that don’t
   require entries in the presence map).
4. New fields cannot use operators that would affect shared
   dictionary values.

Jake

[fplpo_x]

[ original email was from Greg Orsini - greg.orsini@orcsoftware.com ]
Application. Makes it simpler to separate code layers as well since the buffered socket reader can read the length byte(s) and read the next ‘n’ bytes to pass up to the next layer.

Greg.

I’m not sure I understand. What do you mean by optional? Optional per
message or optional per application? If you mean per message, how do you
discover the absence of a length preamble on a particular message?

/David

As I mentioned earlier, an optional length prefix to the message would
easily ensure message separation without the block stuff. Would it be
simpler?

Greg.

Yes, it would work in some situations, but it requires you to use
blocks in general which might not be desired. Furthermore, you’d
have to restrict a block to contain exactly one message (at least in
situations where you know you’re using extended templates (as
percieved by the receiver, which might not always be possible to
know)).

Also, the constraint list would have to be extended to prohibit the
bytes that constitute the extension from looking like the start of a
message, which is rather tricky in the general case. It’s probably
solvable by always adding two pad bytes at the start of the
extension the first always having the value 0xc0, and the second
having the value of some unused tid.

This last constraint is not necessary if you enforce the one message
per block constraint for all messages. But if a decoder make use of
this constraint it will suddenly behave in a non standard way.

/David

I think the easiest approach is to create a FAST best practices
guide that includes instructions for achieving support for
multiple message template versions at the same time. Summarizing
what has been said before, this would require:

1. Length prefixed blocks
2. Fields cannot be added to sequences
3. Fields can only be added to a group if the group is the last
   field instruction in a template and the added fields do not
   affect the groups use of a presence map (e.g. a group that does
   not need a presence map can only have fields added that don’t
   require entries in the presence map).
4. New fields cannot use operators that would affect shared
   dictionary values.

Jake

[davidrosenborg30_x]

That is one of the purposes of the block construct already defined in the FAST 1.1 specification.

My point though, was that blocks are not fully suitable for handling template extensions unless you restrict them to containing exactly on message, and having a length preamble per message might not be feasible from a compaction point of view, especially if you have small messages.

/David

Application. Makes it simpler to separate code layers as well since the
buffered socket reader can read the length byte(s) and read the next ‘n’
bytes to pass up to the next layer.

Greg.

[jsulmon_x]

Cheers for this.

As long as there is no compatibility between template version (which boils down to Hanno’s question), then I don’t understand how what you describe works.

Say template foo, version associated with tid=7 and version 2 associated with tid=27, as per your example.
If I receive a message with tid=27 and I have not build the corresponding codec, then I’m doomed. Unless I know that 27 the id map to the next incarnation for template_id=7

I was maybe a bit terse.

By using immutable template ids (in the short term) you can relax the
requirements on synchrony between template definitions and messages
using template ids. As a receiver, you still need a new template
definition before you can decode a message with the new template id, but
the existing definitions don’t change (within a time window).

This allows for a lower overhead template distribution solution, which
is what Dimitry was asking about, I think.

An example; Let’s assume you have a template named “foo/1” where 1 is
the version of the template. Further let’s assume the Template Name
“foo/1” is associated with the Id 7. Now, if you want to modify this
template (extend or change), you can create “foo/2” which differs in
some aspect from “foo/1” and, depending on other Ids in use your first
unused Id is 27, so you associate the name “foo/2” with the id 27.

Now receivers may see different representations of a message type.
Template id 7 will still refer to the old layout and id 27 will be
unknown until receivers see / retrieve the definition for id 27. The
important difference is that you can relax strict ordering as id 7 is
immutable for some period of time before it may be reused. Not seeing
the new definition of id 27 can be detected.

You still need a level of synchronization between templates and messages
in order to consume messages with newly created templates, but you don’t
need a strict ordering; new definitions can be sent out earlier as there
is no (short-term) change to the meaning of an existing template.

There needs to be additional synchronization if several sources are
allowed to allocate template ids, but that is a separate issue.

/Rolf

The remaining problem is then to make sure that a receiver recognizes
and acts on template id rollover. Implementors need to make sure that
senders receivers agree on the current meaning of a template id (ie.
the current association between a template name and a template id).
It is still an instance of the reliable messaging problem, but with
drastically lower latency requirements.

That’s precisely the hard part: IMHO this is more than an instance of
the Reliable messaging problem – because it involves some ordering
considerations. Given a sender and a possibly large number of
receivers: no receiver should deliver a message prior to delivering
its related template.
e.g., … [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1] …

Not sure that’s an instance of the distributed consensus, but rather
instance of the virtual synchrony problem. That is, something (SCP)
should guarantee that template changes within a group of receiver, are
observed in the same order by all receivers. Furthermore, template
changes in SCP must be totally ordered with respect to all regular
FAST messages.

BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3 third
item one should read “…defined in 6.1” and not “defined in 5.1”

I couldn’t find mention of this in the SCP spec.

[jsulmon_x]

Could anyone please kindly respond to my question – I am not saying there is a problem but rather that I don’t understand

Cheers for this.

As long as there is no compatibility between template version (which
boils down to Hanno’s question), then I don’t understand how what you
describe works.

Say template foo, version associated with tid=7 and version 2 associated
with tid=27, as per your example. If I receive a message with tid=27 and
I have not build the corresponding codec, then I’m doomed. Unless I know
that 27 the id map to the next incarnation for template_id=7

I was maybe a bit terse.

By using immutable template ids (in the short term) you can relax the
requirements on synchrony between template definitions and messages
using template ids. As a receiver, you still need a new template
definition before you can decode a message with the new template id,
but the existing definitions don’t change (within a time window).

This allows for a lower overhead template distribution solution, which
is what Dimitry was asking about, I think.

An example; Let’s assume you have a template named “foo/1” where 1 is
the version of the template. Further let’s assume the Template Name
“foo/1” is associated with the Id 7. Now, if you want to modify this
template (extend or change), you can create “foo/2” which differs in
some aspect from “foo/1” and, depending on other Ids in use your first
unused Id is 27, so you associate the name “foo/2” with the id 27.

Now receivers may see different representations of a message type.
Template id 7 will still refer to the old layout and id 27 will be
unknown until receivers see / retrieve the definition for id 27. The
important difference is that you can relax strict ordering as id 7 is
immutable for some period of time before it may be reused. Not seeing
the new definition of id 27 can be detected.

You still need a level of synchronization between templates and
messages in order to consume messages with newly created templates,
but you don’t need a strict ordering; new definitions can be sent out
earlier as there is no (short-term) change to the meaning of an
existing template.

There needs to be additional synchronization if several sources are
allowed to allocate template ids, but that is a separate issue.

/Rolf

The remaining problem is then to make sure that a receiver
recognizes and acts on template id rollover. Implementors need to
make sure that senders receivers agree on the current meaning of a
template id (ie. the current association between a template name
and a template id). It is still an instance of the reliable
messaging problem, but with drastically lower latency requirements.

That’s precisely the hard part: IMHO this is more than an instance
of the Reliable messaging problem – because it involves some
ordering considerations. Given a sender and a possibly large number
of receivers: no receiver should deliver a message prior to
delivering its related template.
e.g., … [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1] …

Not sure that’s an instance of the distributed consensus, but rather
instance of the virtual synchrony problem. That is, something (SCP)
should guarantee that template changes within a group of receiver,
are observed in the same order by all receivers. Furthermore,
template changes in SCP must be totally ordered with respect to all
regular FAST messages.

BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3 third
item one should read “…defined in 6.1” and not “defined in 5.1”

I couldn’t find mention of this in the SCP spec.

[davidrosenborg30_x]

When the decoder sees a tid it hasn’t seen before, it must resolve that tid into a template name and further resolve that template name into the actual template definition before it can continue.

The exact steps involved in the resolution are dependent on how the particular service is setup. In a pure multicast scenario I guess you’d have a “template beacon” periodically scrolling through the current set of templates. Then you’d have to suspend your main decoder until the template decoder sees the requested template definition pass by if it hasn’t seen it already.

In other setups you might have a TCP back channel or FTP or web site where you can request new templates (think this was what Darshan recommended).

/David

Cheers for this.

As long as there is no compatibility between template version (which
boils down to Hanno’s question), then I don’t understand how what you
describe works.

Say template foo, version associated with tid=7 and version 2 associated
with tid=27, as per your example. If I receive a message with tid=27 and
I have not build the corresponding codec, then I’m doomed. Unless I know
that 27 the id map to the next incarnation for template_id=7

I was maybe a bit terse.

By using immutable template ids (in the short term) you can relax the
requirements on synchrony between template definitions and messages
using template ids. As a receiver, you still need a new template
definition before you can decode a message with the new template id,
but the existing definitions don’t change (within a time window).

This allows for a lower overhead template distribution solution, which
is what Dimitry was asking about, I think.

An example; Let’s assume you have a template named “foo/1” where 1 is
the version of the template. Further let’s assume the Template Name
“foo/1” is associated with the Id 7. Now, if you want to modify this
template (extend or change), you can create “foo/2” which differs in
some aspect from “foo/1” and, depending on other Ids in use your first
unused Id is 27, so you associate the name “foo/2” with the id 27.

Now receivers may see different representations of a message type.
Template id 7 will still refer to the old layout and id 27 will be
unknown until receivers see / retrieve the definition for id 27. The
important difference is that you can relax strict ordering as id 7 is
immutable for some period of time before it may be reused. Not seeing
the new definition of id 27 can be detected.

You still need a level of synchronization between templates and
messages in order to consume messages with newly created templates,
but you don’t need a strict ordering; new definitions can be sent out
earlier as there is no (short-term) change to the meaning of an
existing template.

There needs to be additional synchronization if several sources are
allowed to allocate template ids, but that is a separate issue.

/Rolf

The remaining problem is then to make sure that a receiver
recognizes and acts on template id rollover. Implementors need to
make sure that senders receivers agree on the current meaning of a
template id (ie. the current association between a template name
and a template id). It is still an instance of the reliable
messaging problem, but with drastically lower latency requirements.

That’s precisely the hard part: IMHO this is more than an instance
of the Reliable messaging problem – because it involves some
ordering considerations. Given a sender and a possibly large number
of receivers: no receiver should deliver a message prior to
delivering its related template.
e.g., … [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1] …

Not sure that’s an instance of the distributed consensus, but rather
instance of the virtual synchrony problem. That is, something (SCP)
should guarantee that template changes within a group of receiver,
are observed in the same order by all receivers. Furthermore,
template changes in SCP must be totally ordered with respect to all
regular FAST messages.

BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3 third
item one should read “…defined in 6.1” and not “defined in 5.1”

I couldn’t find mention of this in the SCP spec.

[jsulmon_x]

Thank you David – this is what I thought. If this is acceptable for receiver, then I see indeed no reasons to bother with virtual synchrony etc.

When the decoder sees a tid it hasn’t seen before, it must resolve that
tid into a template name and further resolve that template name into the
actual template definition before it can continue.

The exact steps involved in the resolution are dependent on how the
particular service is setup. In a pure multicast scenario I guess you’d
have a “template beacon” periodically scrolling through the current set
of templates. Then you’d have to suspend your main decoder until the
template decoder sees the requested template definition pass by if it
hasn’t seen it already.

In other setups you might have a TCP back channel or FTP or web site
where you can request new templates (think this was what Darshan
recommended).

/David

Cheers for this.

As long as there is no compatibility between template version (which
boils down to Hanno’s question), then I don’t understand how what you
describe works.

Say template foo, version associated with tid=7 and version 2
associated with tid=27, as per your example. If I receive a message
with tid=27 and I have not build the corresponding codec, then I’m
doomed. Unless I know that 27 the id map to the next incarnation for
template_id=7

I was maybe a bit terse.

By using immutable template ids (in the short term) you can relax
the requirements on synchrony between template definitions and
messages using template ids. As a receiver, you still need a new
template definition before you can decode a message with the new
template id, but the existing definitions don’t change (within a
time window).

This allows for a lower overhead template distribution solution,
which is what Dimitry was asking about, I think.

An example; Let’s assume you have a template named “foo/1” where 1
is the version of the template. Further let’s assume the Template
Name “foo/1” is associated with the Id 7. Now, if you want to modify
this template (extend or change), you can create “foo/2” which
differs in some aspect from “foo/1” and, depending on other Ids in
use your first unused Id is 27, so you associate the name “foo/2”
with the id 27.

Now receivers may see different representations of a message type.
Template id 7 will still refer to the old layout and id 27 will be
unknown until receivers see / retrieve the definition for id 27. The
important difference is that you can relax strict ordering as id 7
is immutable for some period of time before it may be reused. Not
seeing the new definition of id 27 can be detected.

You still need a level of synchronization between templates and
messages in order to consume messages with newly created templates,
but you don’t need a strict ordering; new definitions can be sent
out earlier as there is no (short-term) change to the meaning of an
existing template.

There needs to be additional synchronization if several sources are
allowed to allocate template ids, but that is a separate issue.

/Rolf

The remaining problem is then to make sure that a receiver
recognizes and acts on template id rollover. Implementors need to
make sure that senders receivers agree on the current meaning of
a template id (ie. the current association between a template
name and a template id). It is still an instance of the reliable
messaging problem, but with drastically lower latency
requirements.

That’s precisely the hard part: IMHO this is more than an instance
of the Reliable messaging problem – because it involves some
ordering considerations. Given a sender and a possibly large
number of receivers: no receiver should deliver a message prior to
delivering its related template.
e.g., … [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1] …

Not sure that’s an instance of the distributed consensus, but
rather instance of the virtual synchrony problem. That is,
something (SCP) should guarantee that template changes within a
group of receiver, are observed in the same order by all
receivers. Furthermore, template changes in SCP must be totally
ordered with respect to all regular FAST messages.

BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3
third item one should read “…defined in 6.1” and not “defined in
5.1”

I couldn’t find mention of this in the SCP spec.

[dimitrylondon_x]

Thanks everyone for responses.

Like Rolf noted, I am looking for a template distribution mechanism with minimum amount of overhead. It is also important not to restrict certain areas of template to be changed (end of the group, sequence, etc) - in practice, it hard to predict what part of the template will change in future.

I like the idea of using template id/template name resolution, with a new template id requiring a corresponding template structure. However, it is also important not to tie the template distribution mechanism to a particular transmission protocol.

Is SCP the right mechanism for the actual template distribution in both UDP and TCP environments?

Thanks,
Dimitry

Thank you David – this is what I thought. If this is acceptable for
receiver, then I see indeed no reasons to bother with virtual
synchrony etc.

When the decoder sees a tid it hasn’t seen before, it must resolve
that tid into a template name and further resolve that template name
into the actual template definition before it can continue.

The exact steps involved in the resolution are dependent on how the
particular service is setup. In a pure multicast scenario I guess
you’d have a “template beacon” periodically scrolling through the
current set of templates. Then you’d have to suspend your main decoder
until the template decoder sees the requested template definition pass
by if it hasn’t seen it already.

In other setups you might have a TCP back channel or FTP or web site
where you can request new templates (think this was what Darshan
recommended).

/David

Cheers for this.

As long as there is no compatibility between template version (which
boils down to Hanno’s question), then I don’t understand how what
you describe works.

Say template foo, version associated with tid=7 and version 2
associated with tid=27, as per your example. If I receive a message
with tid=27 and I have not build the corresponding codec, then I’m
doomed. Unless I know that 27 the id map to the next incarnation for
template_id=7

I was maybe a bit terse.

By using immutable template ids (in the short term) you can relax
the requirements on synchrony between template definitions and
messages using template ids. As a receiver, you still need a new
template definition before you can decode a message with the new
template id, but the existing definitions don’t change (within a
time window).

This allows for a lower overhead template distribution solution,
which is what Dimitry was asking about, I think.

An example; Let’s assume you have a template named “foo/1” where 1
is the version of the template. Further let’s assume the Template
Name “foo/1” is associated with the Id 7. Now, if you want to
modify this template (extend or change), you can create “foo/2”
which differs in some aspect from “foo/1” and, depending on other
Ids in use your first unused Id is 27, so you associate the name
“foo/2” with the id 27.

Now receivers may see different representations of a message type.
Template id 7 will still refer to the old layout and id 27 will be
unknown until receivers see / retrieve the definition for id 27.
The important difference is that you can relax strict ordering as
id 7 is immutable for some period of time before it may be reused.
Not seeing the new definition of id 27 can be detected.

You still need a level of synchronization between templates and
messages in order to consume messages with newly created
templates, but you don’t need a strict ordering; new definitions
can be sent out earlier as there is no (short-term) change to the
meaning of an existing template.

There needs to be additional synchronization if several sources
are allowed to allocate template ids, but that is a separate
issue.

/Rolf

The remaining problem is then to make sure that a receiver
recognizes and acts on template id rollover. Implementors need
to make sure that senders receivers agree on the current
meaning of a template id (ie. the current association between a
template name and a template id). It is still an instance of
the reliable messaging problem, but with drastically lower
latency requirements.

That’s precisely the hard part: IMHO this is more than an
instance of the Reliable messaging problem – because it
involves some ordering considerations. Given a sender and a
possibly large number of receivers: no receiver should deliver a
message prior to delivering its related template.
e.g., … [T_i] [M_i_1] … [M_i_j] … [T_(i+1)] [M_(i+1)_1]
…

Not sure that’s an instance of the distributed consensus, but
rather instance of the virtual synchrony problem. That is,
something (SCP) should guarantee that template changes within a
group of receiver, are observed in the same order by all
receivers. Furthermore, template changes in SCP must be totally
ordered with respect to all regular FAST messages.

BTW there is a typo in Fast SCP draft 1.1: page 15 section 7.3
third item one should read “…defined in 6.1” and not “defined
in
5.1”

I couldn’t find mention of this in the SCP spec.