Imported from previous forum
In the fix4.1 spec first page FIX MESSAGE FORMAT AND DELIVERY. The first paragraph states that the general format of a FIX messages is a standard header followed by the message body fields and terminated with a standard trailer. But the bold paragraph states that fields can be in any order except the first three fields and the checksum. Also the first paragraph of the message header description, it states that Each admin or application message is preceded by a standard header.
So can a header field(not the first 3 fields, e.q. field 142) mix inside the message body fields? Or does it mean the fields inside the header can be in any other but not mix with the message body?
[ original email was from Yevgeniy Tovshteyn - yevgeniy@javtech.com ]
Our FIX engine - “Coppelia” handles the case then all fields, header’s and body’s (besides ones that must be in its place), mixed in any order. We think that since FIX protocol doesn’t prohibit that, FIX Engine should be able receive such messages.
> In the fix4.1 spec first page FIX MESSAGE FORMAT AND DELIVERY. The first paragraph states that the general format of a FIX messages is a standard header followed by the message body fields and terminated with a standard trailer. But the bold paragraph states that fields can be in any order except the first three fields and the checksum. Also the first paragraph of the message header description, it states that Each admin or application message is preceded by a standard header.
>
> So can a header field(not the first 3 fields, e.q. field 142) mix inside the message body fields? Or does it mean the fields inside the header can be in any other but not mix with the message body?
>
I think most engines could parse standard header fields mixed within the body. Logically construction consists of the header + body + trailer and I would certainly recommended that one construct messages that way.
> In the fix4.1 spec first page FIX MESSAGE FORMAT AND DELIVERY. The first paragraph states that the general format of a FIX messages is a standard header followed by the message body fields and terminated with a standard trailer. But the bold paragraph states that fields can be in any order except the first three fields and the checksum. Also the first paragraph of the message header description, it states that Each admin or application message is preceded by a standard header.
>
> So can a header field(not the first 3 fields, e.q. field 142) mix inside the message body fields? Or does it mean the fields inside the header can be in any other but not mix with the message body?
>
[ original email was from Zul Kagalwalla - zul@mintco.com ]
> I think most engines could parse standard header fields mixed within the body. Logically construction consists of the header + body + trailer and I would certainly recommended that one construct messages that way.
>
> > In the fix4.1 spec first page FIX MESSAGE FORMAT AND DELIVERY. The first paragraph states that the general format of a FIX messages is a standard header followed by the message body fields and terminated with a standard trailer. But the bold paragraph states that fields can be in any order except the first three fields and the checksum. Also the first paragraph of the message header description, it states that Each admin or application message is preceded by a standard header.
> >
> > So can a header field(not the first 3 fields, e.q. field 142) mix inside the message body fields? Or does it mean the fields inside the header can be in any other but not mix with the message body?
> >
> Scott,
Thanks For you comments.
The following excrepts of the FIX Protocol very clearly state that the message should be constructed as Header+BODY+Trailer.
Page 14
Each administrative or application message, is preceded by a standard header. The header identifies the message type, length, destination, sequence number, origination point and time.
Page 22
The exchange of business related information is accomplished through the passing of application messages. The application message is composed of the standard header followed by the message body and trailer.
Descriptions and formats of the specific messages follow.
Also each Message Type 8, J and P. Specifically in document is presented as Standard Header , Body and Trailer.
Is it not correct to say that the parsers that are parsing the messages which have header and body mixed are an exception and not the rule.
Your comments would be very much appreciated. Thanks in advance
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
I would say that bold paragraph takes lower priority than any statement in the spec requiring field order, especially for things like repeating groups which often have an explicit requirement that a required field appears at the start of each group. So one could make the case that the proper way is header + body + trailer.
But a point could also be made that one should code defensively, and so someone slipping a header field after a body field shouldn’t necessarily cause a problem.
I think it most prudent to stick to sending only header + body + trailer but accepting header fields other than the first three in the body should your trading partners do otherwise.
I also recommend that future spec versions should clarify the language, adding something like "Except where otherwise noted in this specification…" to the bold paragraph.
> In the fix4.1 spec first page FIX MESSAGE FORMAT AND DELIVERY. The first paragraph states that the general format of a FIX messages is a standard header followed by the message body fields and terminated with a standard trailer. But the bold paragraph states that fields can be in any order except the first three fields and the checksum. Also the first paragraph of the message header description, it states that Each admin or application message is preceded by a standard header.
>
> So can a header field(not the first 3 fields, e.q. field 142) mix inside the message body fields? Or does it mean the fields inside the header can be in any other but not mix with the message body?
>
[ original email was from Nikhil Bose - assistsoft@yahoo.com ]
Our FIX engine, AssistFIX also doesn’t mind header fields being mixed with the body. I agree with the person who said that its better to program defensively. If the opposite side doesn’t mix header and body fields, that’s great. But if they you don’t want to throw a fit. Much better to handle it gracefully especially since it doesn’t cost you anything extra, i.e. it doesn’t provide you any great benefit to assume that the header and body fields are separate.
I suppose this is a result of my interpretation of the Header section. The way I looked at it is: Apart from Begin String, BodyLength, and MsgType, the rest of the header doesn’t have to be at the head of the message. Think of it as a grouping of all tags that are common to all message types, and nothing more. Hence they can be mixed with body tags. Only the first three fields are the “real” header.
Just my thoughts,
Nikhil Bose
AssistSoft Corporation
> In the fix4.1 spec first page FIX MESSAGE FORMAT AND DELIVERY. The first paragraph states that the general format of a FIX messages is a standard header followed by the message body fields and terminated with a standard trailer. But the bold paragraph states that fields can be in any order except the first three fields and the checksum. Also the first paragraph of the message header description, it states that Each admin or application message is preceded by a standard header.
>
> So can a header field(not the first 3 fields, e.q. field 142) mix inside the message body fields? Or does it mean the fields inside the header can be in any other but not mix with the message body?
>
The order of fields does matter when you save the message in an audit log or a message warehouse. You want to store the header in one place so that it can be easily queried. Rearranging the message for this purpose breaks the integrity of the audit log.
Each organization must have a process to deal with badly structured messages. Typically, the message is accepted and the partner is strongly encouraged to fix the problem.
[ original email was from Danny Shobrook - dzshobrook@dial.pipex.com ]
> The order of fields does matter when you save the message in an audit log or a message warehouse. You want to store the header in one place so that it can be easily queried. Rearranging the message for this purpose breaks the integrity of the audit log.
>
So strip out the header and store it separately from the original message.
The FIX spec only requires that the order be begin string, body length, msg type with checksum on the end. Some tags that are related need to be in order - eg signature length then signature. If you require more, you have not written a compliant engine, end of story.
> Each organization must have a process to deal with badly structured messages. Typically, the message is accepted and the partner is strongly encouraged to fix the problem.
>
I know of one engine that always has the sender sub ID near the end of the message. It is still a compliant if maybe eccentric engine. Therefore the message should be accepted as it is well structured. If our engine did not accept it then it is our problem and our problem to fix.
> Is it not correct to say that the parsers that are parsing the messages which have header and body mixed are an exception and not the rule.
>
Having only written one FIX engine myself, I guess I don’t really know how many engines require it as header + body + trailer way or have the capability to parse the fields mixed in any order. 
We have obviously pointed out an opportunity for clarification of language in the spec related to this. It seems logical that the message be presented as header + body + trailer and it appears from this discussion that some implementations require it in this fashion.
My main question is: Is (vs. would) someone currently having a problem with a counterparty constructing a message with header or trailer fields mixed within the body fields? If so, is there a reason for constructing it that way?
[ original email was from Zul Kagalwalla - zul@mintco.com ]
> > Is it not correct to say that the parsers that are parsing the messages which have header and body mixed are an exception and not the rule.
> >
>
> Having only written one FIX engine myself, I guess I don’t really know how many engines require it as header + body + trailer way or have the capability to parse the fields mixed in any order.
>
> We have obviously pointed out an opportunity for clarification of language in the spec related to this. It seems logical that the message be presented as header + body + trailer and it appears from this discussion that some implementations require it in this fashion.
>
> My main question is: Is (vs. would) someone currently having a problem with a counterparty constructing a message with header or trailer fields mixed within the body fields? If so, is there a reason for constructing it that way?
>
>
[ original email was from John Armstrong - johna@ms.com ]
Typically protocols define a header as the
essential information needed to parse a message
e.g. version, length, and perhaps message type.
I think we’ve extended the meaning of header in FIX to include routing/identification fields
common to all messages.
To answer your question, I see no reason why one
HAS to require the additional header fields to
be before all body fields.
And we also have to consider if the FIX allowance
for duplicating a field(one encrypted, one unencrypted) in an encrypted message would cause
problems if we were to say all header fields must
precede any body fields.
>
> My main question is: Is (vs. would) someone currently having a problem with a counterparty constructing a message with header or trailer fields mixed within the body fields? If so, is there a reason for constructing it that way?
>
>
[ original email was from Zul Kagalwalla - zul@mintco.com ]
> > Is it not correct to say that the parsers that are parsing the messages which have header and body mixed are an exception and not the rule.
> >
>
> Having only written one FIX engine myself, I guess I don’t really know how many engines require it as header + body + trailer way or have the capability to parse the fields mixed in any order.
>
> We have obviously pointed out an opportunity for clarification of language in the spec related to this. It seems logical that the message be presented as header + body + trailer and it appears from this discussion that some implementations require it in this fashion.
>
> My main question is: Is (vs. would) someone currently having a problem with a counterparty constructing a message with header or trailer fields mixed within the body fields? If so, is there a reason for constructing it that way?
> Yes we have problem parsing the incomming FIX messages that have Body and Text mixed together.
More Important issue is what is right? What happens in Future if FIX standard enforces that the messages strictly follow the Standards. What would that standard be.
You have correctly pointed out that there should
>be clarification in the language. Would this clarification enforce that the FIX standard have messages in HEADER+BODY+TRAILER format.
If we do that then does it implie some organizations will have to change their definitions in a later version of FIX ??
Thanks
[ original email was from Nikhil Bose - assistsoft@yahoo.com ]
> My main question is: Is (vs. would) someone currently having a problem with a counterparty constructing a message with header or trailer fields mixed within the body fields? If so, is there a reason for constructing it that way?
>
One reason for this might be the view we take of message creation and transmission. The user creates a message with only the body fields without worrying about the sender, sequence number, and any of the header fields except for the message type. After he has created the message with his application-level tags, he uses a Session object to send it to a specific party. The Session object fills in the sequence number, the body length, check sum, and all the header fields that are common to that Session (party). In this model, it is more convenient to attach the header fields to the end of the message since one view of the message is a Vector. So except for the 4 fields mandated to be at specific positions, it would be more convenient to put the common (to the session/party) fields at the end.
This is not how we do it, but it is a plausible scanario. We do put all the header fields at the top, as most people seem to expect it there.
Nikhil Bose
AssistSoft Corporation
[ original email was from Yevgeniy Tovshteyn - yevgeniy@javtech.com ]
The other thing is SecureData - it contains fields from header and body and when one unpack it, he might not be willing to separate them to header and body portions. If fix engine serves as a router(like in Thompson network), it’s job might be not only rout messages, but encrypt-decrypt them.
We do separate fields on decryption -"but it is a plausible scanario"
> > My main question is: Is (vs. would) someone currently having a problem with a counterparty constructing a message with header or trailer fields mixed within the body fields? If so, is there a reason for constructing it that way?
> >
>
> One reason for this might be the view we take of message creation and transmission. The user creates a message with only the body fields without worrying about the sender, sequence number, and any of the header fields except for the message type. After he has created the message with his application-level tags, he uses a Session object to send it to a specific party. The Session object fills in the sequence number, the body length, check sum, and all the header fields that are common to that Session (party). In this model, it is more convenient to attach the header fields to the end of the message since one view of the message is a Vector. So except for the 4 fields mandated to be at specific positions, it would be more convenient to put the common (to the session/party) fields at the end.
>
> This is not how we do it, but it is a plausible scanario. We do put all the header fields at the top, as most people seem to expect it there.
>
> Nikhil Bose
> AssistSoft Corporation
>
>