length, checksum, implict

, ,
1

Imported from previous forum

2

At the FAST summit in April, there was mention of further compressing FIX by not sending FIX’s length and checksum fields. Now that the implicit operator has been retired, does this mean that length and checksum elimination was too risky in practice?

3

[ original email was from Matt Simpson - msimpson@cme.com ]
The primary reason for retiring the implicit operator was that it didn’t fit well with the core concept of removing redundancies by capitalizing on data affinities. Additionally, implying the length or checksum are things that can be done in the application layer.

When using FAST to send market data over a multicast transport there is little benefit to sending checksum and body length since it is unlikely that the data will be corrupted in transmission - especially when the encoding state spans only a single packet.

It’s a different story when using a FIX session over FAST and the encoding state spans the entire session. Checksum and body length are excellent safeguards for determining that the decoded result matches the original unencoded message. See the FIX over FAST Recommendation at the following link which does a good job illustrating the inherent riskiness of sending encoded data - and how Checksum can be used to reduce that risk.
http://www.fixprotocol.org/documents/2786/FIX_over_FAST_Specification_draft.doc

At the FAST summit in April, there was mention of further compressing
FIX by not sending FIX’s length and checksum fields. Now that the
implicit operator has been retired, does this mean that length and
checksum elimination was too risky in practice?

4

The primary reason for retiring the implicit operator was that it didn’t
fit well with the core concept of removing redundancies by capitalizing
on data affinities. Additionally, implying the length or checksum are
things that can be done in the application layer.
[…]

At the FAST summit in April, there was mention of further compressing
FIX by not sending FIX’s length and checksum fields. Now that the
implicit operator has been retired, does this mean that length and
checksum elimination was too risky in practice?

The retirement of the implicit operator from the spec does not imply (!) that length and checksum need to be passed now, only that the template is not explicit about them being dropped when encoding a FIX message using FAST. If the exact FIX tag/value input is to be regenerated at the receiving side, the decoder will recalculate those fields.
So, if you would like redundancy, you can pass length and checksum over FAST or you can leave them out in the template. If left out, the receiver has to know how to create them on the fly.
I think that the FIX over FAST document should specify these requirements on a receiver.

5

I would like to take this issue further. For the decoder to verify the checksum on the receiver side, the encoder would have send it (the length could be implicitly calculated). Furthermore, the encoder would have to calculate the length of the “implied” fields - i.e. constants, copy-coded away, default, etc. Doesn’t this calculation make sense only if the data is being encoded from the original FIX message into FAST format, but not if the data is encoded into FAST format using FastFix template specification directly(i.e, if MDEntryPX is being encoded directly into FAST message)?

Also, the reference C API does not appear to provide means of calculating length and checksum as it returns the lentgh of the encoded fields, not implied fields.

Thanks,
Dimitry

The primary reason for retiring the implicit operator was that it
didn’t fit well with the core concept of removing redundancies by
capitalizing on data affinities. Additionally, implying the length or
checksum are things that can be done in the application layer. […]

At the FAST summit in April, there was mention of further
compressing FIX by not sending FIX’s length and checksum fields. Now
that the implicit operator has been retired, does this mean that
length and checksum elimination was too risky in practice?

The retirement of the implicit operator from the spec does not imply (!)
that length and checksum need to be passed now, only that the template
is not explicit about them being dropped when encoding a FIX message
using FAST. If the exact FIX tag/value input is to be regenerated at the
receiving side, the decoder will recalculate those fields. So, if you
would like redundancy, you can pass length and checksum over FAST or you
can leave them out in the template. If left out, the receiver has to
know how to create them on the fly. I think that the FIX over FAST
document should specify these requirements on a receiver.

6

Dimitry, pls find my comments inlined below

As Anders comments in his original post, the FIX over FAST doc
may benefit from some clarifications.

/Rolf

I would like to take this issue further. For the decoder to verify
the checksum on the receiver side, the encoder would have send it
(the length could be implicitly calculated).

That’s correct

Furthermore, the encoder would have to calculate the length of the
“implied” fields - i.e. constants, copy-coded away, default, etc.

Yes

Doesn’t this calculation make sense only if the data is being
encoded from the original FIX message into FAST format, but not if
the data is encoded into FAST format using FastFix template
specification directly(i.e, if MDEntryPX is being encoded directly
into FAST message)?

It makes obvious sense if you do FIX-FAST-FIX, but the important
question here is if the receiver should be able to convert to FIX.
If you want to offer the ability to calculate and verify a FIX
checksum at the receiving end, then you would have to provide the
checksum from the sending side. Otherwise, as you point out, there
will be no checksum to verify against.

So, you could have use for a checksum in an ANY-FAST-FIX setup.

That said, I can think a few alternatives to using the checksum
field of classic FIX. Maybe it’s time to discuss this in the
mdowg.

Also, the reference C API does not appear to provide means of
calculating length and checksum as it returns the lentgh of the
encoded fields, not implied fields.

That’s correct

Thanks, Dimitry

The primary reason for retiring the implicit operator was that it
didn’t fit well with the core concept of removing redundancies by
capitalizing on data affinities. Additionally, implying the
length or checksum are things that can be done in the application
layer. […]

At the FAST summit in April, there was mention of further
compressing FIX by not sending FIX’s length and checksum fields.
Now that the implicit operator has been retired, does this mean
that length and checksum elimination was too risky in practice?

The retirement of the implicit operator from the spec does not
imply (!) that length and checksum need to be passed now, only
that the template is not explicit about them being dropped when
encoding a FIX message using FAST.

If the exact FIX tag/value input is to be regenerated at the
receiving side, the decoder will recalculate those fields.

So, if you would like redundancy, you can pass length and
checksum over FAST or you can leave them out in the template.

If left out, the receiver has to know how to create them on
the fly. I think that the FIX over FAST document should specify
these requirements on a receiver.

7

Rolf, thanks very much. I think it is a good idea to discuss alternatives to the FIX checksum in the FAST world.

Dimitry

Dimitry, pls find my comments inlined below

As Anders comments in his original post, the FIX over FAST doc may
benefit from some clarifications.

/Rolf

I would like to take this issue further. For the decoder to verify the
checksum on the receiver side, the encoder would have send it (the
length could be implicitly calculated).

That’s correct

Furthermore, the encoder would have to calculate the length of the
“implied” fields - i.e. constants, copy-coded away, default, etc.

Yes

Doesn’t this calculation make sense only if the data is being encoded
from the original FIX message into FAST format, but not if the data is
encoded into FAST format using FastFix template specification
directly(i.e, if MDEntryPX is being encoded directly into FAST
message)?

It makes obvious sense if you do FIX-FAST-FIX, but the important
question here is if the receiver should be able to convert to FIX. If
you want to offer the ability to calculate and verify a FIX checksum at
the receiving end, then you would have to provide the checksum from the
sending side. Otherwise, as you point out, there will be no checksum to
verify against.

So, you could have use for a checksum in an ANY-FAST-FIX setup.

That said, I can think a few alternatives to using the checksum field of
classic FIX. Maybe it’s time to discuss this in the mdowg.

Also, the reference C API does not appear to provide means of
calculating length and checksum as it returns the lentgh of the
encoded fields, not implied fields.

That’s correct

Thanks, Dimitry

The primary reason for retiring the implicit operator was that it
didn’t fit well with the core concept of removing redundancies by
capitalizing on data affinities. Additionally, implying the
length or checksum are things that can be done in the application
layer. […]

At the FAST summit in April, there was mention of further
compressing FIX by not sending FIX’s length and checksum fields.
Now that the implicit operator has been retired, does this mean
that length and checksum elimination was too risky in practice?

The retirement of the implicit operator from the spec does not imply
(!) that length and checksum need to be passed now, only that the
template is not explicit about them being dropped when encoding a FIX
message using FAST.

If the exact FIX tag/value input is to be regenerated at the
receiving side, the decoder will recalculate those fields.

So, if you would like redundancy, you can pass length and checksum
over FAST or you can leave them out in the template.

If left out, the receiver has to know how to create them on the fly.
I think that the FIX over FAST document should specify these
requirements on a receiver.

8

Hi,

Could somebody please help me get the FIX_over_FAST_Specification?

http://www.fixprotocol.org/documents/2786/FIX_over_FAST_Specification_draft.doc

It requests authorization in the portal. Is there any other way to access this document.

Rgds,
Thaya.

The primary reason for retiring the implicit operator was that it didn’t
fit well with the core concept of removing redundancies by capitalizing
on data affinities. Additionally, implying the length or checksum are
things that can be done in the application layer.

When using FAST to send market data over a multicast transport there is
little benefit to sending checksum and body length since it is unlikely
that the data will be corrupted in transmission - especially when the
encoding state spans only a single packet.

It’s a different story when using a FIX session over FAST and the
encoding state spans the entire session. Checksum and body length are
excellent safeguards for determining that the decoded result matches the
original unencoded message. See the FIX over FAST Recommendation at the
following link which does a good job illustrating the inherent riskiness
of sending encoded data - and how Checksum can be used to reduce that
risk. http://www.fixprotocol.org/documents/2786/FIX_over_FAST_Specifica-
tion_draft.doc

At the FAST summit in April, there was mention of further compressing
FIX by not sending FIX’s length and checksum fields. Now that the
implicit operator has been retired, does this mean that length and
checksum elimination was too risky in practice?