Imported from previous forum
Page 6 of strawman document says “Easily processed message no overhead in accessing message content”. By this, am I correct in assuming that the decoder is able to easily seperate the header, body and trailer ?
If we add three tags
Length of Header - first tag after all mandatory initial tags of FIX message like 8, 9, 35, etc (http://fixprotocol.org/discuss/read/90566451)
Length of Body - first tag after all header fields and before any body tags.
Length of trailer - first tag after all body fields and before any trailer tags.
then body can be easily accessed and decoder can split the header, body and trailer. Presently, only after encountering the firt body tag does the decoder know that header is over.
Regards,
Mahesh
If we add three tags
Length of Header - first tag after all mandatory initial tags of FIX message like 8, 9, 35, etc (http://fixprotocol.org/discuss/read/90566451)
Length of Body - first tag after all header fields and before any body tags.
Length of trailer - first tag after all body fields and before any trailer tags.
then body can be easily accessed and decoder can split the header, body and trailer. Presently, only after encountering the firt body tag does the decoder know that header is over.
Regards,
Mahesh
This is an important property of the new protocol, and I think we should push this
even further. By providing direct access to all fields in a message it should be possible
to avoid some data copying in the decoding process.
There are several ways to accomplish this. One is using fixed size fields. Another is
having a compressed offset table in the message in conjunction with a predefined
message layout (something like FAST templates or Prepared Messages). I’m sure
this group can produce additional alternatives.
Regards,
/Peter
Separation of header and body, possibly different parts of the header and body as well, is in scope.
I don’t think new tags is the way to go here. We need to use a message structure that goes beyond the linear tag=value syntax. That said, I’m not entirely sure how to do it as there are numerous trade-offs to consider.
Page 6 of strawman document says “Easily processed message no overhead in accessing message content”. By this, am I correct in assuming that the decoder is able to easily seperate the header, body and trailer ?
If we add three tags
Length of Header - first tag after all mandatory initial tags of FIX message like 8, 9, 35, etc (http://fixprotocol.org/discuss/read/90566451)
Length of Body - first tag after all header fields and before any body tags.
Length of trailer - first tag after all body fields and before any trailer tags.
then body can be easily accessed and decoder can split the header, body and trailer. Presently, only after encountering the firt body tag does the decoder know that header is over.
Regards,
Mahesh
I proposed three new tags with the goal of splitting a message into Header, Body and Trailer only. Adding tags to locate sub-parts of the message beyond a certain coarseness of granularity would not work.
Also I was not restricting to only Tag=Value encoding, presence of these three tags in Compact Binary Format CBF encoding proposed by Georges Gomes @ Ullink could benefit equally. If CBF emerges as an additional encoding standard, then any changes in Tag=Value FIX could have a parallel rippling change in CBF with its associated benefit.
If three is a large number of tags to add, we could make do with one tag - Length of Header. This approach has two benefits compared to three Tag approach :-
- Space saving of two tags
- Computation savings = two additions and one comparision to validate that BodyLength(9) value is consistent with values in new Tags Header Length(x), BodyBody Length(y) and Trailer Length(z).
Using the one new Tag for Header length, once Header is split, the remaining is body + trailer. As it appears, CheckSum 10=NNN^ could be on the way out. If this happens, then Message minus Header gives Body.
Adding one new tag to specify Header Length gives a large speed advantage - the decoder could use two parallel Threads to process them after split assuming there are no validation dependencies between Header and Body fields.
Separation of header and body, possibly different parts of the header and body as well, is in scope.
I don’t think new tags is the way to go here. We need to use a message structure that goes beyond the linear tag=value syntax. That said, I’m not entirely sure how to do it as there are numerous trade-offs to consider.
Page 6 of strawman document says “Easily processed message no overhead in accessing message content”. By this, am I correct in assuming that the decoder is able to easily seperate the header, body and trailer ?
If we add three tags
Length of Header - first tag after all mandatory initial tags of FIX message like 8, 9, 35, etc (http://fixprotocol.org/discuss/read/90566451)
Length of Body - first tag after all header fields and before any body tags.
Length of trailer - first tag after all body fields and before any trailer tags.
then body can be easily accessed and decoder can split the header, body and trailer. Presently, only after encountering the firt body tag does the decoder know that header is over.
Regards,
Mahesh
I was a bit terse in my previous post: the ‘tag=value’ was meant to be interpreted as any tag/value representation. The way I see it, there is little reason for tagging the two lengths/offsets. If we implement them, they should be part of the message structure, probably residing at a fixed offset from the beginning of the message.
I proposed three new tags with the goal of splitting a message into Header, Body and Trailer only. Adding tags to locate sub-parts of the message beyond a certain coarseness of granularity would not work.
Also I was not restricting to only Tag=Value encoding, presence of these three tags in Compact Binary Format CBF encoding proposed by Georges Gomes @ Ullink could benefit equally. If CBF emerges as an additional encoding standard, then any changes in Tag=Value FIX could have a parallel rippling change in CBF with its associated benefit.
If three is a large number of tags to add, we could make do with one tag - Length of Header. This approach has two benefits compared to three Tag approach :-
- Space saving of two tags
- Computation savings = two additions and one comparision to validate that BodyLength(9) value is consistent with values in new Tags Header Length(x), BodyBody Length(y) and Trailer Length(z).
Using the one new Tag for Header length, once Header is split, the remaining is body + trailer. As it appears, CheckSum 10=NNN^ could be on the way out. If this happens, then Message minus Header gives Body.
Adding one new tag to specify Header Length gives a large speed advantage - the decoder could use two parallel Threads to process them after split assuming there are no validation dependencies between Header and Body fields.
Separation of header and body, possibly different parts of the header and body as well, is in scope.
I don’t think new tags is the way to go here. We need to use a message structure that goes beyond the linear tag=value syntax. That said, I’m not entirely sure how to do it as there are numerous trade-offs to consider.
Fixed Offset(s) would make the message structure inflexible because
- Many Fields have variable length values
- Presence / absence of optional fields
So the length of message or its top level sub-parts (Header, Body) would be difficult to standardize at protocol design stage.
The way I see it, there is little reason for tagging the two lengths/offsets. If we implement them, they should be part of the message structure, probably residing at a fixed offset from the beginning of the message.
I managed to be too terse once again. I was referring to the two length/offset fields, not the rest of the data fields. For example:
| HdrLen | BodyLen | Hdr … | Body … |
where MsgLen = HdrLen + BodyLen
would allow a decoder to access the body using the HdrLen to skip over the hdr. My point is that there is probably little reason to use tag/value to represent the length/offset fields. In the example above, HdrLen and BodyLen are part of the message layout and wouldn’t necessarily need to be represented as other fields. Another variant could be:
| MsgLen | HdrLen | Hdr … | Body … |
where BodyLen = MsgLen - HdrLen and BodyOffset = HdrLen …
Fixed Offset(s) would make the message structure inflexible because
- Many Fields have variable length values
- Presence / absence of optional fields
So the length of message or its top level sub-parts (Header, Body) would be difficult to standardize at protocol design stage.
The way I see it, there is little reason for tagging the two lengths/offsets. If we implement them, they should be part of the message structure, probably residing at a fixed offset from the beginning of the message.
Rolf,
You weren’t terse, sorry I was thinking out aloud (just being tooo clear to myself 
To me, this variant
| HdrLen | BodyLen | Hdr … | Body … |
looks worse than
| MsgLen | HdrLen | Hdr … | Body … |
In the 2nd variant (MsgLen + HdrLen), once decoder encounters MsgLen, it can split MsgLen bytes and spin it off into a different worker thread which only knows about
| HdrLen | Hdr … | Body … |
but in the 1st variant (HdrLen + BodyLen), the decoder has to read both of these and add them to get size of message to split message out from the byte stream, then it would have to pass atleast one additional parameter HdrLen + maybe BodyLen for the worker thread to parse the bytes into header and body.
Also from a wholisticsness persepective, its better to use | MsgLen | HdrLen | Hdr … | Body … | layout because the first few bytes identify the length of the message and the basic unit of HFT transmission is Message (a Header + a body is NOT the basic unit).
Regards,
K. Mahesh
I managed to be too terse once again. I was referring to the two length/offset fields, not the rest of the data fields. For example:
| HdrLen | BodyLen | Hdr … | Body … |
where MsgLen = HdrLen + BodyLen
would allow a decoder to access the body using the HdrLen to skip over the hdr. My point is that there is probably little reason to use tag/value to represent the length/offset fields. In the example above, HdrLen and BodyLen are part of the message layout and wouldn’t necessarily need to be represented as other fields. Another variant could be:
| MsgLen | HdrLen | Hdr … | Body … |
where BodyLen = MsgLen - HdrLen and BodyOffset = HdrLen …
Fixed Offset(s) would make the message structure inflexible because
- Many Fields have variable length values
- Presence / absence of optional fields
So the length of message or its top level sub-parts (Header, Body) would be difficult to standardize at protocol design stage.
The way I see it, there is little reason for tagging the two lengths/offsets. If we implement them, they should be part of the message structure, probably residing at a fixed offset from the beginning of the message.