TCP frame size in absence of BodyLength?

, ,
1

Imported from previous forum

2

I am experimenting with a mini FIX engine. The code I have written so far operates on a text file containing a FIX message per line. I completely separate messages by using newline as the delimiter. Obviously I can’t do that for messages which come over the network.

For well-formed messages, I can extract the body length, which will tell me how big the message is. What if the body length field is missing or out of order? How do I know when the current, garbled message ends and a new one starts (this problem is worse if the next message is just as garbled). As far as I can tell, there are no end-of-message markers in the FIX spec.

How is this normally done?

Thanks

3

[ original email was from Ajay Kamdar - ajay.kamdar@thomson.com ]
Actually, if what you are writing is more than just an experiment, you really can’t use a newline as a delimiter even when reading from a file. Tag 58 could have a newline character in it. There can also be encoded fields that contain a newline.

If you encounter a garbled message then one strategy is to look for the start of the next message by skipping forward until you find the next “8=FIX.4.49=” (substitute FIX.4.4 with a value appropriate for your FIX session).

I am experimenting with a mini FIX engine. The code I have written so
far operates on a text file containing a FIX message per line. I
completely separate messages by using newline as the delimiter.
Obviously I can’t do that for messages which come over the network.

For well-formed messages, I can extract the body length, which will tell
me how big the message is. What if the body length field is missing or
out of order? How do I know when the current, garbled message ends and a
new one starts (this problem is worse if the next message is just as
garbled). As far as I can tell, there are no end-of-message markers in
the FIX spec.

How is this normally done?

Thanks

4

[ original email was from John Prewett - jprewett@lavatrading.com ]
Typically, once a “framing error” is encountered, the session is nonviable and should be disconnected and further Logon attempts should be denied until manual intervention overrides this.

The reason why the previous poster’s suggestion of skipping forwards until you see “8=FIX.4.4” is not viable is because you would ultimately issue a ResendRequest for the missing messages and presumably the garbage that you encountered before would be retransmitted to you again. So you would end up with an endlessly repeating problem.

So any “framing error” should be treated as a fatal error on the session and it should be disconnected and disallowed pending manual intervention.

There are many types of framing errors:

  1. Message doesn’t start with 8=FIX.4.4
  2. BodyLength doesn’t immediately follow after BeginString or is mal-formed.
  3. CheckSum field not where BodyLength value indicates it should be or is mal-formed.
  4. Calculated checksum doesn’t match CheckSum value.
  5. BodyLength longer than maximum allowed by FIX engine (I received one once that indicated the mesage was more than 1 billion bytes long).
    I daresay there are a few more I have omitted.

I hope this helps.

JohnP

5

The FIX spec says that if checksum is wrong (and a host of other parsing problems), the message should be considered “garbled” and it should be quietly ignored (no rejects, no use of seqnos, no dropped connection).

How are ‘garbled’ messages and framing errors handled in actual implementations used in the industry?

Thanks

Typically, once a “framing error” is encountered, the session is
nonviable and should be disconnected and further Logon attempts should
be denied until manual intervention overrides this.

The reason why the previous poster’s suggestion of skipping forwards
until you see “8=FIX.4.4” is not viable is because you would
ultimately issue a ResendRequest for the missing messages and presumably
the garbage that you encountered before would be retransmitted to you
again. So you would end up with an endlessly repeating problem.

So any “framing error” should be treated as a fatal error on the session
and it should be disconnected and disallowed pending manual
intervention.

There are many types of framing errors:

  1. Message doesn’t start with 8=FIX.4.4
  2. BodyLength doesn’t immediately follow after BeginString or is
    mal-formed.
  3. CheckSum field not where BodyLength value indicates it should be or
    is mal-formed.
  4. Calculated checksum doesn’t match CheckSum value. 5. BodyLength
    longer than maximum allowed by FIX engine (I received one once that
    indicated the mesage was more than 1 billion bytes long). I daresay
    there are a few more I have omitted.

I hope this helps.

JohnP

6

[ original email was from Ajay Kamdar - ajay.kamdar@thomson.com ]
> Typically, once a “framing error” is encountered, the session is

nonviable and should be disconnected and further Logon attempts should
be denied until manual intervention overrides this.

The reason why the previous poster’s suggestion of skipping forwards
until you see “8=FIX.4.4” is not viable is because you would
ultimately issue a ResendRequest for the missing messages and presumably
the garbage that you encountered before would be retransmitted to you
again. So you would end up with an endlessly repeating problem.

An endless loop of resend-request/skip-garbled-message would occur only if the counter party sent a garbage message. If the message got garbled due to a transient glitch then skipping to the next well formed message and requesting a resend of the missed messages would allow the session to recover without manual intervention.

You are correct that if care is not exercised, then the session could go into an endless loop. I know of at least one commercial FIX engine which a significant market share that uses this strategy for skipping past garbled messages. That engine also has a cap on how many times it will make a resend request for a message before it gives up and terminates the FIX session, which ensures it will never go into an endless loop due to garbage messages originating from the counter party. While this approach may not be pure, it works well in practice and reduces the need for manual intervention for transient errors that the engine can automatically recover from.

Regards.

7

Thanks, I need to be more careful about newlines!

Actually, if what you are writing is more than just an experiment, you
really can’t use a newline as a delimiter even when reading from a file.
Tag 58 could have a newline character in it. There can also be encoded
fields that contain a newline.

If you encounter a garbled message then one strategy is to look for the
start of the next message by skipping forward until you find the next
“8=FIX.4.49=” (substitute FIX.4.4 with a value appropriate for your
FIX session).

I am experimenting with a mini FIX engine. The code I have written so
far operates on a text file containing a FIX message per line. I
completely separate messages by using newline as the delimiter.
Obviously I can’t do that for messages which come over the network.

For well-formed messages, I can extract the body length, which will
tell me how big the message is. What if the body length field is
missing or out of order? How do I know when the current, garbled
message ends and a new one starts (this problem is worse if the next
message is just as garbled). As far as I can tell, there are no end-of-
message markers in the FIX spec.

How is this normally done?

Thanks

8

Shahbaz,

The end of every FIX message is checksum field ie ^10=NNN^ where ^ is SOH delimiter.

Regards,
Mahesh

I am experimenting with a mini FIX engine. The code I have written so
far operates on a text file containing a FIX message per line. I
completely separate messages by using newline as the delimiter.
Obviously I can’t do that for messages which come over the network.

For well-formed messages, I can extract the body length, which will tell
me how big the message is. What if the body length field is missing or
out of order? How do I know when the current, garbled message ends and a
new one starts (this problem is worse if the next message is just as
garbled). As far as I can tell, there are no end-of-message markers in
the FIX spec.

How is this normally done?

Thanks