Do we need to clear [reset] the dictionary?

, ,
1

Imported from previous forum

2

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

please help me to make this clear out.

thanks,

3

If you can reliably send packets in sequence you do not need to reset, e.g. with TCP/IP or withiin a single file. With UDP this is not the case and you should have a reset at the beginning of every packet. if you omit this and the sequence gets garbled then the recipient will be unable to decode your messages properly.

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

please help me to make this clear out.

thanks,

4

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

It depends on what transport protocol is being used. For Multicast, the protocol most commonly used with FAST, you statement is correct, but if FAST is used with a reliable protocol like TCP/IP it may never be necessary to reset the dictionaries.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

The impact is the decoded data is incorrect, or occasionally the decoding process fails. Having the Encoder & Decoder agree on when to reset the dictionaries is essential.

In FAST 1.1 this is done as part of the Rules of Engagement between counterparties. SCP adds the ability for the encoder to indicate explicitly when a reset should occur by using a template with the reset attribute.

HTH,

Dale

Principal Software Developer, Object Computing, Inc.
Lead Developer of QuickFAST

5

Hi Guyz,

Really thanks for your quick response.

So can I have the following conclusion.

For multicast you need to reset the dictionary for each packet. otherwise if sequence get mess up decoder will not be able to decode properly.

But in TCP reliable transmission transfer encoding will be effected across packets. So as sequence is not getting mess up you should not rest the cache.

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

It depends on what transport protocol is being used. For Multicast, the protocol most commonly used with FAST, you statement is correct, but if FAST is used with a reliable protocol like TCP/IP it may never be necessary to reset the dictionaries.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

The impact is the decoded data is incorrect, or occasionally the decoding process fails. Having the Encoder & Decoder agree on when to reset the dictionaries is essential.

In FAST 1.1 this is done as part of the Rules of Engagement between counterparties. SCP adds the ability for the encoder to indicate explicitly when a reset should occur by using a template with the reset attribute.

Can I have a simple example or explanation how to do this?

Thanks again.

HTH,

Dale

Principal Software Developer, Object Computing, Inc.
Lead Developer of QuickFAST

6

Hi Thilina,

that would be correct, although a sender of a FAST stream over TCP may want to send reset messages at points in the stream to synchronize between multiple TCP streams carrying the same FAST-encoded data. In this case the receiver must be prepared to receive reset messages in the stream.

An example when this scenario would be relevant is when a venue sends the same market data to multiple clients over TCP and wants to support late joiners.

Best,
Rolf

Hi Guyz,

Really thanks for your quick response.

So can I have the following conclusion.

For multicast you need to reset the dictionary for each packet. otherwise if sequence get mess up decoder will not be able to decode properly.

But in TCP reliable transmission transfer encoding will be effected across packets. So as sequence is not getting mess up you should not rest the cache.

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

It depends on what transport protocol is being used. For Multicast, the protocol most commonly used with FAST, you statement is correct, but if FAST is used with a reliable protocol like TCP/IP it may never be necessary to reset the dictionaries.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

The impact is the decoded data is incorrect, or occasionally the decoding process fails. Having the Encoder & Decoder agree on when to reset the dictionaries is essential.

In FAST 1.1 this is done as part of the Rules of Engagement between counterparties. SCP adds the ability for the encoder to indicate explicitly when a reset should occur by using a template with the reset attribute.

Can I have a simple example or explanation how to do this?

Thanks again.

HTH,

Dale

Principal Software Developer, Object Computing, Inc.
Lead Developer of QuickFAST

7

Hi Rolf,

Thanks for the explanation.
I now have a clear idea on this.

Any way can any of you please explain how to send reset messages.

thanks.

Hi Thilina,

that would be correct, although a sender of a FAST stream over TCP may want to send reset messages at points in the stream to synchronize between multiple TCP streams carrying the same FAST-encoded data. In this case the receiver must be prepared to receive reset messages in the stream.

An example when this scenario would be relevant is when a venue sends the same market data to multiple clients over TCP and wants to support late joiners.

Best,
Rolf

Hi Guyz,

Really thanks for your quick response.

So can I have the following conclusion.

For multicast you need to reset the dictionary for each packet. otherwise if sequence get mess up decoder will not be able to decode properly.

But in TCP reliable transmission transfer encoding will be effected across packets. So as sequence is not getting mess up you should not rest the cache.

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

It depends on what transport protocol is being used. For Multicast, the protocol most commonly used with FAST, you statement is correct, but if FAST is used with a reliable protocol like TCP/IP it may never be necessary to reset the dictionaries.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

The impact is the decoded data is incorrect, or occasionally the decoding process fails. Having the Encoder & Decoder agree on when to reset the dictionaries is essential.

In FAST 1.1 this is done as part of the Rules of Engagement between counterparties. SCP adds the ability for the encoder to indicate explicitly when a reset should occur by using a template with the reset attribute.

Can I have a simple example or explanation how to do this?

Thanks again.

HTH,

Dale

Principal Software Developer, Object Computing, Inc.
Lead Developer of QuickFAST

8

Hi,

The simplest way is to put two bytes on the wire: 0xC0 0xF8, and you must also remember to clear any encoder dictionary state.

If you are using a FAST library you’ll have to check with the corresponding documentation how to achieve this.

/David

Hi Rolf,

Thanks for the explanation.
I now have a clear idea on this.

Any way can any of you please explain how to send reset messages.

thanks.

Hi Thilina,

that would be correct, although a sender of a FAST stream over TCP may want to send reset messages at points in the stream to synchronize between multiple TCP streams carrying the same FAST-encoded data. In this case the receiver must be prepared to receive reset messages in the stream.

An example when this scenario would be relevant is when a venue sends the same market data to multiple clients over TCP and wants to support late joiners.

Best,
Rolf

Hi Guyz,

Really thanks for your quick response.

So can I have the following conclusion.

For multicast you need to reset the dictionary for each packet. otherwise if sequence get mess up decoder will not be able to decode properly.

But in TCP reliable transmission transfer encoding will be effected across packets. So as sequence is not getting mess up you should not rest the cache.

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

It depends on what transport protocol is being used. For Multicast, the protocol most commonly used with FAST, you statement is correct, but if FAST is used with a reliable protocol like TCP/IP it may never be necessary to reset the dictionaries.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

The impact is the decoded data is incorrect, or occasionally the decoding process fails. Having the Encoder & Decoder agree on when to reset the dictionaries is essential.

In FAST 1.1 this is done as part of the Rules of Engagement between counterparties. SCP adds the ability for the encoder to indicate explicitly when a reset should occur by using a template with the reset attribute.

Can I have a simple example or explanation how to do this?

Thanks again.

HTH,

Dale

Principal Software Developer, Object Computing, Inc.
Lead Developer of QuickFAST

9

Hi,

Thanks David.

Hi,

The simplest way is to put two bytes on the wire: 0xC0 0xF8, and you must also remember to clear any encoder dictionary state.

If you are using a FAST library you’ll have to check with the corresponding documentation how to achieve this.

/David

Hi Rolf,

Thanks for the explanation.
I now have a clear idea on this.

Any way can any of you please explain how to send reset messages.

thanks.

Hi Thilina,

that would be correct, although a sender of a FAST stream over TCP may want to send reset messages at points in the stream to synchronize between multiple TCP streams carrying the same FAST-encoded data. In this case the receiver must be prepared to receive reset messages in the stream.

An example when this scenario would be relevant is when a venue sends the same market data to multiple clients over TCP and wants to support late joiners.

Best,
Rolf

Hi Guyz,

Really thanks for your quick response.

So can I have the following conclusion.

For multicast you need to reset the dictionary for each packet. otherwise if sequence get mess up decoder will not be able to decode properly.

But in TCP reliable transmission transfer encoding will be effected across packets. So as sequence is not getting mess up you should not rest the cache.

According to my knowledge FAST operators are applied only for a packet. So transfer encoding will not happen across packets. Please correct me if I’m wrong.

It depends on what transport protocol is being used. For Multicast, the protocol most commonly used with FAST, you statement is correct, but if FAST is used with a reliable protocol like TCP/IP it may never be necessary to reset the dictionaries.

So if it is the case do we need to clear [reset] the cache for each packet. Is there any impact not doing so?

The impact is the decoded data is incorrect, or occasionally the decoding process fails. Having the Encoder & Decoder agree on when to reset the dictionaries is essential.

In FAST 1.1 this is done as part of the Rules of Engagement between counterparties. SCP adds the ability for the encoder to indicate explicitly when a reset should occur by using a template with the reset attribute.

Can I have a simple example or explanation how to do this?

Thanks again.

HTH,

Dale

Principal Software Developer, Object Computing, Inc.
Lead Developer of QuickFAST

10

The simplest way is to put two bytes on the wire: 0xC0 0xF8, and you must also remember to clear any encoder dictionary state.

That might work sometimes or it might not. If those two bytes happen to appear in exactly the right place in the data stream, they might be an empty message with a template ID of 120. For some feeds that is considered a reset message because the templates associated with the feed include a template definition that looks like

[template id=“120” reset=“yes”][/template]

Other feeds assume such a template definition exists and don’t bother to include it in their template definition file (which is nonstandard but common.)

For other feeds template ID 120 is not special in any way and may not even exist.

==================

If this is an established FAST feed, the simplest way to do things correctly is to consult the documentation for the feed describing how reset is handled for this feed and conform to the documentation.

If this is a new feed that you are designing, then the simplest way to do things correctly is to read the FAST specification documents including SCP which describes the reset attribute of a template (NOT a reset message!); design a system which conforms to these specifications; and document the design. It is quite likely that such a design would include template 120, but certainly not required.

Dale

–
Principal Software Engineer, Object Computing, Inc.
Lead Developer for QuickFAST (http://www.quickfast.org/)