Imported from previous forum
[ original email was from Jim Northey - jnorthey@jandj.com ]
For FAST 1.0 does Copy encoding of field values (using a previous value if one is not provided) span frame boundaries? In other words if the first occurrence of a field in a message in a new frame when not specified set to the last value received in a previous frame?
[ original email was from Greg Orsini - go@cameronsystems.com ]
> For FAST 1.0 does Copy encoding of field values (using a previous value
if one is not provided) span frame boundaries? In other words if the
first occurrence of a field in a message in a new frame when not
specified set to the last value received in a previous frame?
This is different to streams, where copy encoding can occur for an entire FIX session. Frame encoding implies a multi-cast environment which typically will not provide transmission with TCP’s “reliable” semantics. This means you may not receive a previous frame from which to copy a value.
Copy encoding should only be used AFTER the first occurence within a frame.
[ original email was from Sitaram Guruswamy - sitaram.guruswamy@transacttools.net ]
> > For FAST 1.0 does Copy encoding of field values (using a previous
value if one is not provided) span frame boundaries? In other words if
the first occurrence of a field in a message in a new frame when not
specified set to the last value received in a previous frame?This is different to streams, where copy encoding can occur for an
entire FIX session. Frame encoding implies a multi-cast environment
which typically will not provide transmission with TCP’s “reliable”
semantics. This means you may not receive a previous frame from which to
copy a value.Copy encoding should only be used AFTER the first occurence
within a frame.
The SCP spec provides for a FastReset Message. This is a one-way field-encoding reset notification. This is an indication to the receiving end that the sender has reset its outgoing field encoding history and so the receiver should reset its incoming field encoding history. Rather than imply the resetting of field encoding with every frame, every multicast frame (e.g. UDP datagram) could be transmitted with the FastReset as the first message. The receiver can then process messsages in a wire transport-agnostic codepath.
Framing is necessary and implied in a multicast UDP environment. Jim, are you also asking about framing in a streaming transport like TCP? There has been a lot of discussion as to whether this is required or optional, or even necessary.
[ original email was from Jim Northey - jnorthey@jandj.com ]
I was trying to inventory what others believed was in the specification. From your and Greg’s email - I am thinking that one of the two alternative is what we should go with:
-
omit framing as a concept when FAST is used with a streaming protocol
-
Or as Greg indicated for a streaming protocol - the session is the frame and to restart another frame in extraordinary circumstances the FastReset must be used.
For the datagram protocol - each packet is a frame
I think your points are related, not one or the other. Framing isn’t needed for encoding in a streaming transport like TCP since messages are ordered. But you still should be able to send a FastReset whenever you want, although you don’t specifically need to in TCP.
However, if the rule was “the sender should send a FastReset at the beginning of each UDP frame” then the recipient can remain agnostic as to whether the transport is TCP or UDP. The recipient just knows to reset incoming state whenever he sees a FastReset.
David Rhodes
TransactTools
I was trying to inventory what others believed was in the specification. From your and Greg’s email - I am thinking that one of the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming protocol
Or as Greg indicated for a streaming protocol - the session is the
frame and to restart another frame in extraordinary circumstances the
FastReset must be used.For the datagram protocol - each packet is a frame
[ original email was from Jim Northey - jnorthey@jandj.com ]
Not sure why we would want or need to send a FAST Reset in the UDP case?
Seems unnecessary and inefficient - I could be missing something here.
Are you referring to the use of UDP for bi-directional communication - exchanging messages? Even so - not sure there is a need for the FAST Reset in either case. For UDP the Frame should map to the UDP message size - making each message a Frame - making the FAST Reset unneeded in the UDP case as a way of communicating framing. The other concerns is having to push another packet out between frames like alot of overhead for no apparent benefit. Again, I am probably missing something important here as usual.
I think your points are related, not one or the other. Framing isn’t
needed for encoding in a streaming transport like TCP since messages are
ordered. But you still should be able to send a FastReset whenever you
want, although you don’t specifically need to in TCP.However, if the rule was “the sender should send a FastReset at the
beginning of each UDP frame” then the recipient can remain agnostic as
to whether the transport is TCP or UDP. The recipient just knows to
reset incoming state whenever he sees a FastReset.David Rhodes TransactTools
I was trying to inventory what others believed was in the
specification. From your and Greg’s email - I am thinking that one of
the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming
protocolOr as Greg indicated for a streaming protocol - the session is the
frame and to restart another frame in extraordinary circumstances
the FastReset must be used.For the datagram protocol - each packet is a frame
There’s an implementation choice at the receiving end:
A UDP reader may signal to the decoder to reset history using;
- some (internal) API flag telling the decoder to reset, or,
- adding a reset in front of the received datagram.
If the sender adds a reset as the first message, then the UDP
datagram receiver and the decoder will work without having to
worry about frame boundaries.
The overhead of a reset (with the currently proposed coding)
is one byte. The overhead of underlying protocol headers is
roughly 42 bytes (ETH 14, IP 20, UDP 8) so the overhead of
the reset message would be less than 2.5% in any case.
With a payload size of 100 bytes the overhead of the reset
message drops to 1/(42+1+100) = 1/143 => ~0.7%
We may want to consider recommending or specifying that the
sender should add a reset to simplify processing at the
receiving end.
Thx,
Rolf
Not sure why we would want or need to send a FAST Reset in the UDP case?
Seems unnecessary and inefficient - I could be missing something here.
Are you referring to the use of UDP for bi-directional communication -
exchanging messages? Even so - not sure there is a need for the FAST
Reset in either case. For UDP the Frame should map to the UDP message
size - making each message a Frame - making the FAST Reset unneeded in
the UDP case as a way of communicating framing. The other concerns is
having to push another packet out between frames like alot of overhead
for no apparent benefit. Again, I am probably missing something
important here as usual.I think your points are related, not one or the other. Framing isn’t
needed for encoding in a streaming transport like TCP since messages
are ordered. But you still should be able to send a FastReset whenever
you want, although you don’t specifically need to in TCP.However, if the rule was “the sender should send a FastReset at the
beginning of each UDP frame” then the recipient can remain agnostic as
to whether the transport is TCP or UDP. The recipient just knows to
reset incoming state whenever he sees a FastReset.David Rhodes TransactTools
I was trying to inventory what others believed was in the
specification. From your and Greg’s email - I am thinking that one
of the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming
protocolOr as Greg indicated for a streaming protocol - the session is
the frame and to restart another frame in extraordinary
circumstances the FastReset must be used.For the datagram protocol - each packet is a frame
Fast 1.0 in the most simple state contains no SCP or framing. A minimal valid FAST message is a one BYTE pmap. There is never a need to send a reset message unless you are using DELTA encoding. If you are just using COPY, NONE, or INCREMENT encoding, one simply needs to send a fully populated message to “seed” the values. The decoder would detect an error if a value defined with COPY encoding was not present in a message, and there was no previous value available.
DELTA encoding presents a challenge as there is no way to distinguish between the original “whole” value or the “delta” within a field without knowing if the message has been reset. Using the transport boundary (multicast datagram’s, etc.) as logical frame solves this.
/Daniel
I was trying to inventory what others believed was in the specification. From your and Greg’s email - I am thinking that one of the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming protocol
Or as Greg indicated for a streaming protocol - the session is the
frame and to restart another frame in extraordinary circumstances the
FastReset must be used.For the datagram protocol - each packet is a frame
[ original email was from Jim Northey - jnorthey@jandj.com ]
The previous discussion I believe captures a discussion of framing and the requirement for reset.
If in the case of a stream (TCP) - and the Session == Frame, unless reset for some extraordinary reason
For UDP - Frame == a packet
Fast 1.0 in the most simple state contains no SCP or framing. A minimal
valid FAST message is a one BYTE pmap. There is never a need to send a
reset message unless you are using DELTA encoding. If you are just using
COPY, NONE, or INCREMENT encoding, one simply needs to send a fully
populated message to “seed” the values. The decoder would detect an
error if a value defined with COPY encoding was not present in a
message, and there was no previous value available.DELTA encoding presents a challenge as there is no way to distinguish
between the original “whole” value or the “delta” within a field without
knowing if the message has been reset. Using the transport boundary
(multicast datagram’s, etc.) as logical frame solves this./Daniel
I was trying to inventory what others believed was in the
specification. From your and Greg’s email - I am thinking that one of
the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming
protocolOr as Greg indicated for a streaming protocol - the session is the
frame and to restart another frame in extraordinary circumstances
the FastReset must be used.For the datagram protocol - each packet is a frame
Guys,
The text below is from an e-mail I sent a while back to explain
why I think there is reason to support FastReset in a TCP based
environment. The original idea came up during a discusion with
Kevin Mueller of Archipelago on supporting late joiners in an
efficient manner. IIRC, Dan was also involved in the discussion
at the time.
… snip …
FastReset enables support for late joiners.
Consider the following setup:
Producer → Sender → Client(s)
The sender may well preserve frame boundaries even if TCP
as a stream transport doesn’t. Given that a client connects
mid-day, the sender may wait for a start of a new frame
before starting to send data to the new client (while the
sender may continue to send data to other clients already
connected).
A producer of information that provides the sender with a
framed message stream may periodically reset the encoding
history and add a FastReset. The sender can then duplicate
this data to multiple clients via their (individual) TCP
sessions.
This way, several clients can share one encoded stream and
the sender can synchronize simply by checking if the next
segment of the data stream begins with a reset (other ways
of deducing that there’s a new frame are available).
Obviously, some kind on internal framing is needed if the
producer and the sender are connected via TCP (but that is
another issue).
… snip …
Dan, Kevin, please feel free to chip in of there was something
that you think I left out from the description.
Thx,
Rolf
The previous discussion I believe captures a discussion of framing and
the requirement for reset.If in the case of a stream (TCP) - and the Session == Frame, unless
reset for some extraordinary reasonFor UDP - Frame == a packet
Fast 1.0 in the most simple state contains no SCP or framing. A
minimal valid FAST message is a one BYTE pmap. There is never a need
to send a reset message unless you are using DELTA encoding. If you
are just using COPY, NONE, or INCREMENT encoding, one simply needs to
send a fully populated message to “seed” the values. The decoder would
detect an error if a value defined with COPY encoding was not present
in a message, and there was no previous value available.DELTA encoding presents a challenge as there is no way to distinguish
between the original “whole” value or the “delta” within a field
without knowing if the message has been reset. Using the transport
boundary (multicast datagram’s, etc.) as logical frame solves this./Daniel
I was trying to inventory what others believed was in the
specification. From your and Greg’s email - I am thinking that one
of the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming
protocolOr as Greg indicated for a streaming protocol - the session is
the frame and to restart another frame in extraordinary
circumstances the FastReset must be used.For the datagram protocol - each packet is a frame
[ original email was from Jim Northey - jnorthey@jandj.com ]
We ended up encoding Rolf’s recommendation as part of the Session Control Protocol Version 1.0 Specification that is being reviewed on Wednesday April 19th.
In general for FAST 1.0:
Datagram (UDP): Frame == UDP Datagram
TCP (two party or individual sessions): Frame == Session, until reset using the FastReset message
TCP (transmission of same packets to multiple clients) Frame is reset at an application defined interval using the FastReset. This provides a way for late joiners (and rejoiners) to join back in and sync up.
Guys,
The text below is from an e-mail I sent a while back to explain why I
think there is reason to support FastReset in a TCP based environment.
The original idea came up during a discusion with Kevin Mueller of
Archipelago on supporting late joiners in an efficient manner. IIRC, Dan
was also involved in the discussion at the time.… snip …
FastReset enables support for late joiners.
Consider the following setup: Producer → Sender → Client(s)
The sender may well preserve frame boundaries even if TCP as a stream
transport doesn’t. Given that a client connects mid-day, the sender may
wait for a start of a new frame before starting to send data to the new
client (while the sender may continue to send data to other clients
already connected).A producer of information that provides the sender with a framed message
stream may periodically reset the encoding history and add a FastReset.
The sender can then duplicate this data to multiple clients via their
(individual) TCP sessions.This way, several clients can share one encoded stream and the sender
can synchronize simply by checking if the next segment of the data
stream begins with a reset (other ways of deducing that there’s a new
frame are available).Obviously, some kind on internal framing is needed if the producer and
the sender are connected via TCP (but that is another issue).… snip …
Dan, Kevin, please feel free to chip in of there was something that you
think I left out from the description.Thx, Rolf
The previous discussion I believe captures a discussion of framing and
the requirement for reset.If in the case of a stream (TCP) - and the Session == Frame, unless
reset for some extraordinary reasonFor UDP - Frame == a packet
Fast 1.0 in the most simple state contains no SCP or framing. A
minimal valid FAST message is a one BYTE pmap. There is never a need
to send a reset message unless you are using DELTA encoding. If you
are just using COPY, NONE, or INCREMENT encoding, one simply needs
to send a fully populated message to “seed” the values. The decoder
would detect an error if a value defined with COPY encoding was not
present in a message, and there was no previous value available.DELTA encoding presents a challenge as there is no way to
distinguish between the original “whole” value or the “delta” within
a field without knowing if the message has been reset. Using the
transport boundary (multicast datagram’s, etc.) as logical frame
solves this./Daniel
I was trying to inventory what others believed was in the
specification. From your and Greg’s email - I am thinking that one
of the two alternative is what we should go with:
omit framing as a concept when FAST is used with a streaming
protocolOr as Greg indicated for a streaming protocol - the session is
the frame and to restart another frame in extraordinary
circumstances the FastReset must be used.For the datagram protocol - each packet is a frame