Imported from previous forum
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
A few questions were raised about the Transport Independence Framework at the Technology Day at the end of October, but they don’t seem to have been addressed in the released specification.
-
The Standard Message Header and Standard Message Trailer are defined in the FIXT (fix session transport) document, but they are referenced in the FIX (fix application) document as required components of every application message. It’s not clear whether the header and trailer are intended to belong to FIXT or to FIX. I’d suggest that some of their contents belong to FIXT while others belong to FIX. The Standard Message Header requires the fields BeginString, BodyLength, and SequenceNumber. The Standard Message Trailer requires Checksum. These fields are not intrinsic to FIX application messages. They are artifacts of transporting these messages over FIXT. Case in point - the only valid BeginString is FIXT.1.1. Furthermore, message-oriented middleware (MOM) architectures are commonly equipped with their own facilities for managing message integrity and sequencing. They don’t necessarily require this information in the application message.
-
The Standard Message Header is also a part of each session message. Should the ApplVerID field be permitted on session messages? Are session messages versioned with the FIXT version or with the FIX version?
-
How does the specification anticipate the introduction of new core features? For instance, if FPL decides upon a better semantic for repeating group parsing, what can be used to indicate the change to a parser? In the past, the BeginString field could be used for this purpose. It was, and still is, specified as the first field in the message. FIX 5.0 has two options – BeginString and ApplVerID. Tag-value parsing is a feature of the FIX application layer. It describes how one can understand the bytes that make up an application message. This being the case, BeginString is no longer an option. BeginString (as I suggested above) belongs to the FIXT layer. Therefore, it cannot be used to version application messages. The corresponding field in the application layer is ApplVerID. However, the placement of ApplVerID in a message is not specified, except that it belongs in the header. This means that one has to parse the entire header searching for the ApplVerID field in order to determine how to parse the message. To complicate matters, the header itself contains a repeating group. It sounds funny, but you end up in a scenario where you have to parse the header in order to figure out how to parse the header. I suggest that the specification is very clear that ApplVerID needs to be the very first field in the message – a replacement for BeginString.
Steven Grossman
TransactTools - NYC
http://www.transacttools.net
[ original email was from Matt Simpson - msimpson@cme.com ]
Steve - thanks for the inquiries. I think it would be useful to address some of these issues in greater detail in a GTC meeting.
On 1 - While there are fields in the standard header that gravitate more toward either the session or application layer it is difficult to draw a definitive line. The recommendation of the GTC is that that best practices be established for each transport (Web Services, MQSeries, UDP Multicast, etc) based on an accumulation of analysis and implementation experience. For example, header fields such as PossDup and PossResend would not be appropriate for a MOM but it may useful to use SenderCompID to identify the entity sending messages.
On 2 - session messages are versioned with the FIXT version rather than the FIX application version. In general, session messages should not carry the ApplVerID tag.
On 3 - a change to the tag=value syntax may make it necessary to release a new version of both the FIX transport as well as the FIX application layer since both would be impacted. The new version would be reflected in BeginString as well as ApplVerID. Regarding the placement of ApplVerID, the GTC is recommending that it be used as the fourth field (BeginString, BodyLength, MsgType, ApplVerID) when used with the FIX session and the third field (BodyLength, MsgType, ApplVerID) when used with in an transport independence scenario. This arrangement should provide a deterministic approach by which to parse the message. We need a bit more discussion on this point and agree that the spec needs to be very clear.
A few questions were raised about the Transport Independence Framework
at the Technology Day at the end of October, but they don’t seem to have
been addressed in the released specification.
The Standard Message Header and Standard Message Trailer are defined
in the FIXT (fix session transport) document, but they are referenced in
the FIX (fix application) document as required components of every
application message. It’s not clear whether the header and trailer are
intended to belong to FIXT or to FIX. I’d suggest that some of their
contents belong to FIXT while others belong to FIX. The Standard Message
Header requires the fields BeginString, BodyLength, and SequenceNumber.
The Standard Message Trailer requires Checksum. These fields are not
intrinsic to FIX application messages. They are artifacts of
transporting these messages over FIXT. Case in point - the only valid
BeginString is FIXT.1.1. Furthermore, message-oriented middleware (MOM)
architectures are commonly equipped with their own facilities for
managing message integrity and sequencing. They don’t necessarily
require this information in the application message.The Standard Message Header is also a part of each session
message. Should the ApplVerID field be permitted on session
messages? Are session messages versioned with the FIXT version or
with the FIX version?How does the specification anticipate the introduction of new core
features? For instance, if FPL decides upon a better semantic for
repeating group parsing, what can be used to indicate the change to a
parser? In the past, the BeginString field could be used for this
purpose. It was, and still is, specified as the first field in the
message. FIX 5.0 has two options – BeginString and ApplVerID. Tag-value
parsing is a feature of the FIX application layer. It describes how one
can understand the bytes that make up an application message. This being
the case, BeginString is no longer an option. BeginString (as I
suggested above) belongs to the FIXT layer. Therefore, it cannot be used
to version application messages. The corresponding field in the
application layer is ApplVerID. However, the placement of ApplVerID in a
message is not specified, except that it belongs in the header. This
means that one has to parse the entire header searching for the
ApplVerID field in order to determine how to parse the message. To
complicate matters, the header itself contains a repeating group. It
sounds funny, but you end up in a scenario where you have to parse the
header in order to figure out how to parse the header. I suggest that
the specification is very clear that ApplVerID needs to be the very
first field in the message – a replacement for BeginString.Steven Grossman TransactTools - NYC http://www.transacttools.net
Hi Steven, Matt, GTC…
I have tried two ways of doing FIX-MIX and they are detailed as follows :-
The structure of a FIX Tag=Value^ message is as follows :-
FIX Message = Header + Body + Trailer
Header = StartOfFIXMessage + MesgSeqNum(34) + LastMsgSeqNumProc(369) + OtherHeaderFields
StartOfFIXMessage is defined below in two ways
OtherHeaderFields = CompIds(In any order) + DupFlags + Times + Secures + MessageEncoding(347) + Hops
DupFlags = PossDupFlag(43) + PossResendFlag(97)
Times = SendingTime(52) + OrigSendingTime(122)
Hops = NoHops(627) + HopCompID(628) + HopSendingTime(629) + HopRefID(630)
Secures = SecureDataLen(90) + SecureData(91)
XMLDataLen(212) and XMLData(213) may be moved to body. Anyway if XML/FIXML content is to be sent, we can use MesgType(35)=n
Trailer = SignatureLength(93) + Signature(89) + CheckSum(10)
Body is defined in two ways below
First Way
StartOfFIXMessage = BeginString(8) + Body length(9) + MesgType(35)
Body = ApplVerID(1128) + CstmApplVerID(1129) + THEMessage
Second Way
StartOfFIXMessage = BeginString(8) + Body length(9)
Body = MesgType(35) + ApplVerID(1128) + CstmApplVerID(1129) + THEMessage
THEMessage is what may be called the “payload”, i.e. for which the remaining parts of the message are present, THEMessage is not present for the envolope / remaining part(s).
My comparision
1st Way, is inline with Standard FIX by having MesgType(35) as third tag in message, 2nd way moves MesgType(35) to first tag in body of message.
If MesgType(35) is a Session Message, then ApplVerID(1128) + CstmApplVerID(1129) are absent and THEMessage contains the relevant tags.
If MesgType(35) is a Business Message, then ApplVerID(1128) + CstmApplVerID(1129) is/are present and THEMessage contains the relevant tags.
2nd way, there may be more than one THEMessage, i.e. THEMessage is a sequence of AMessages. This is an idea like repeating group, its repeating messages and these messages may be different MesgTypes and different ApplVerIds / CstmApplVerIDs. Here there is no tag to indicate number of Amessages, it is delimited by MesgType(35) + ApplVerID(1128) + CstmApplVerID(1129) where at least one of ApplVerID(1128) or CstmApplVerID(1129) is present but both may be present. Also in this case, one AMessage may be FIXML and another Tag=Value^
2nd way, if MesgType(35) is session then should THEMessage be one and only one AMessage ? Maybe Maynotbe ! For example, if I permit multiple AMessages, then I could in a single message send a test request and a resend request ( Case of Long time No See + I think I missed some messages from you ).
1st way, THEMessage is a single AMessage.
Other points
As per current FIX, CompIds though part of Header, can occur intermixed with Body fields.
Like a postal worker looking at address information and delivering mail and not caring about contents(payload), the transport layer just looks at moving the message and not bothering about THEMessage. I think the CompIds are like the details written on a postal envolope ( who is sending to whom ) and hence these belong to the transport layer which is the envolope and should not be mixed with Message body. The message body should only contain application / business part.
In the 2nd way the payload and envolope are better seperated, so multiple different payloads may be present in same envolope.
Presence of DupFlags = PossDupFlag(43) + PossResendFlag(97) is will not hinder transport over MOM, WebServices etc.
The fields in the header can occur in any order because message body starts with ApplVerID(1st Way) or MesgType(2nd Way)
To cut a long story short, since FIX is a protocol ( a language ) for communication between systems, a lot depends on calling a cat a cat ( Just you and me may decide to call a cat a rat a rat a bat and a bat a cat, but then the world would not understand what we are speaking / blabbering ??? “The rat was waiting with a cat to hit the bat” - You and me can still talk in this minglish 
my english 
Just my 2 cents…MyFIX
Regards,
K. Mahesh
+1-203-252-4039
Steve - thanks for the inquiries. I think it would be useful to address
some of these issues in greater detail in a GTC meeting.On 1 - While there are fields in the standard header that gravitate more
toward either the session or application layer it is difficult to draw a
definitive line. The recommendation of the GTC is that that best
practices be established for each transport (Web Services, MQSeries, UDP
Multicast, etc) based on an accumulation of analysis and implementation
experience. For example, header fields such as PossDup and PossResend
would not be appropriate for a MOM but it may useful to use SenderCompID
to identify the entity sending messages.On 2 - session messages are versioned with the FIXT version rather than
the FIX application version. In general, session messages should not
carry the ApplVerID tag.On 3 - a change to the tag=value syntax may make it necessary to release
a new version of both the FIX transport as well as the FIX application
layer since both would be impacted. The new version would be reflected
in BeginString as well as ApplVerID. Regarding the placement of
ApplVerID, the GTC is recommending that it be used as the fourth field
(BeginString, BodyLength, MsgType, ApplVerID) when used with the FIX
session and the third field (BodyLength, MsgType, ApplVerID) when used
with in an transport independence scenario. This arrangement should
provide a deterministic approach by which to parse the message. We need
a bit more discussion on this point and agree that the spec needs to be
very clear.A few questions were raised about the Transport Independence Framework
at the Technology Day at the end of October, but they don’t seem to
have been addressed in the released specification.
The Standard Message Header and Standard Message Trailer are
defined in the FIXT (fix session transport) document, but they are
referenced in the FIX (fix application) document as required
components of every application message. It’s not clear whether the
header and trailer are intended to belong to FIXT or to FIX. I’d
suggest that some of their contents belong to FIXT while others
belong to FIX. The Standard Message Header requires the fields
BeginString, BodyLength, and SequenceNumber. The Standard Message
Trailer requires Checksum. These fields are not intrinsic to FIX
application messages. They are artifacts of transporting these
messages over FIXT. Case in point - the only valid BeginString is
FIXT.1.1. Furthermore, message-oriented middleware (MOM)
architectures are commonly equipped with their own facilities for
managing message integrity and sequencing. They don’t necessarily
require this information in the application message.The Standard Message Header is also a part of each session message.
Should the ApplVerID field be permitted on session messages? Are
session messages versioned with the FIXT version or with the FIX
version?How does the specification anticipate the introduction of new core
features? For instance, if FPL decides upon a better semantic for
repeating group parsing, what can be used to indicate the change to
a parser? In the past, the BeginString field could be used for this
purpose. It was, and still is, specified as the first field in the
message. FIX 5.0 has two options – BeginString and ApplVerID. Tag-
value parsing is a feature of the FIX application layer. It
describes how one can understand the bytes that make up an
application message. This being the case, BeginString is no longer
an option. BeginString (as I suggested above) belongs to the FIXT
layer. Therefore, it cannot be used to version application
messages. The corresponding field in the application layer is
ApplVerID. However, the placement of ApplVerID in a message is not
specified, except that it belongs in the header. This means that
one has to parse the entire header searching for the ApplVerID
field in order to determine how to parse the message. To complicate
matters, the header itself contains a repeating group. It sounds
funny, but you end up in a scenario where you have to parse the
header in order to figure out how to parse the header. I suggest
that the specification is very clear that ApplVerID needs to be the
very first field in the message – a replacement for BeginString.Steven Grossman TransactTools - NYC http://www.transacttools.net
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
1 – I’m not convinced that a definitive line cannot be drawn. With careful analysis of the header fields, I am confident that we can determine which fields belong to the transport (FIXT) and which belong to the application (FIX). The inability to create such a separation suggests that neither specification is independent of the other.
2 – If ApplVerID should not be used for session messages, shouldn’t it be removed from the session message’s header?
3a – while both FIX and FIXT may change, transport independence suggests that they don’t need to change in tandem. I should be able to use the old tag=value format for FIXT while transporting a new format for the FIX content.
3b – Fixing the placement of the ApplVerID field alleviates most of my concern. However, I still believe that it should be the first field in the application message. (I feel that BodyLength belongs to FIXT, so I’ll address my comments to MsgType.) The version defines how the message is parsed and understood. I see no reason to exclude MsgType from this versioning. Placing ApplVerID before MsgType allows FIX the flexibility to construct a message that does not contain a MsgType field as it is currently represented (tags may be re-numbered, message type may be derived from multiple fields, …). What’s the downside of ApplVerID proceeding MsgType?
Steve - thanks for the inquiries. I think it would be useful to address
some of these issues in greater detail in a GTC meeting.On 1 - While there are fields in the standard header that gravitate more
toward either the session or application layer it is difficult to draw a
definitive line. The recommendation of the GTC is that that best
practices be established for each transport (Web Services, MQSeries, UDP
Multicast, etc) based on an accumulation of analysis and implementation
experience. For example, header fields such as PossDup and PossResend
would not be appropriate for a MOM but it may useful to use SenderCompID
to identify the entity sending messages.On 2 - session messages are versioned with the FIXT version rather than
the FIX application version. In general, session messages should not
carry the ApplVerID tag.On 3 - a change to the tag=value syntax may make it necessary to release
a new version of both the FIX transport as well as the FIX application
layer since both would be impacted. The new version would be reflected
in BeginString as well as ApplVerID. Regarding the placement of
ApplVerID, the GTC is recommending that it be used as the fourth field
(BeginString, BodyLength, MsgType, ApplVerID) when used with the FIX
session and the third field (BodyLength, MsgType, ApplVerID) when used
with in an transport independence scenario. This arrangement should
provide a deterministic approach by which to parse the message. We need
a bit more discussion on this point and agree that the spec needs to be
very clear.A few questions were raised about the Transport Independence Framework
at the Technology Day at the end of October, but they don’t seem to
have been addressed in the released specification.
The Standard Message Header and Standard Message Trailer are
defined in the FIXT (fix session transport) document, but they are
referenced in the FIX (fix application) document as required
components of every application message. It’s not clear whether the
header and trailer are intended to belong to FIXT or to FIX. I’d
suggest that some of their contents belong to FIXT while others
belong to FIX. The Standard Message Header requires the fields
BeginString, BodyLength, and SequenceNumber. The Standard Message
Trailer requires Checksum. These fields are not intrinsic to FIX
application messages. They are artifacts of transporting these
messages over FIXT. Case in point - the only valid BeginString is
FIXT.1.1. Furthermore, message-oriented middleware (MOM)
architectures are commonly equipped with their own facilities for
managing message integrity and sequencing. They don’t necessarily
require this information in the application message.The Standard Message Header is also a part of each session message.
Should the ApplVerID field be permitted on session messages? Are
session messages versioned with the FIXT version or with the FIX
version?How does the specification anticipate the introduction of new core
features? For instance, if FPL decides upon a better semantic for
repeating group parsing, what can be used to indicate the change to
a parser? In the past, the BeginString field could be used for this
purpose. It was, and still is, specified as the first field in the
message. FIX 5.0 has two options – BeginString and ApplVerID. Tag-
value parsing is a feature of the FIX application layer. It
describes how one can understand the bytes that make up an
application message. This being the case, BeginString is no longer
an option. BeginString (as I suggested above) belongs to the FIXT
layer. Therefore, it cannot be used to version application
messages. The corresponding field in the application layer is
ApplVerID. However, the placement of ApplVerID in a message is not
specified, except that it belongs in the header. This means that
one has to parse the entire header searching for the ApplVerID
field in order to determine how to parse the message. To complicate
matters, the header itself contains a repeating group. It sounds
funny, but you end up in a scenario where you have to parse the
header in order to figure out how to parse the header. I suggest
that the specification is very clear that ApplVerID needs to be the
very first field in the message – a replacement for BeginString.Steven Grossman TransactTools - NYC http://www.transacttools.net
1 - A definitive line should be drawn - either a field belongs to Transport layer ( what I call the envelope ) or it belongs to the Application layer ( what I call the payload ). Drawing a clear line would make the two layers independent of each other.
2 - In my previous posting in this thread of discussion
http://fixprotocol.org/discuss/read/d5962a4b
I suggest that ApplVerID be part of Body of the message
3.a - Once the Session layer and Application layer are clearly seperated, the application layer can contain message in any format while the session layer is tag=value^
3.b.1 - BodyLength is part of Session layer.
3.b.2 - MsgType and ApplVerID should come next to each other and these two fields becomes part of body. Both of these fields are used to parse the message and validate its contents.
3.b.3 - If MsgType is moved to body, then multiple Application messages in different formats of different versions can be sent in a single Session message. Presently since MsgType is part of header, one FIX message contains only one Application message. By moving MsgType to Message Body, multiple Application messages can be sent in a single FIX message i.e. a single envelope can contain multiple payloads. This way FIX tag=value^ and FIXML can be sent in a single message.
Regards,
K. Mahesh
+1-203-252-4039
Has there been any more discussion on this matter?
I think if this issue remains unresolved (i.e. keeping the current writing) FIX 5.0 removes some important optimization opportunities that are present in < 5.0-parsing. Also, leaving the placement of ApplVerID unspecified makes processing more complex for implementations that does not use the simple but not so performant “bag-of-tags” style of processing.
I think there are two separate issues that have been vetted here
- Whether ApplVerID and MsgType belong together or not
- The placement of this pair in the overall structure
My take on those issues are that
-
They definitely belong together. Both are needed to uniquely identify the message type. The ApplVerID can be defaulted to the DefaultApplVerID passed in the Logon, but this knowledge should also materialize (that is, detecting the absence of an ApplVerID) in conjunction with the MsgType.
-
Keep them part of the header for compatibility and optimization reasons.
As it is now, in the general case, a 5.0 implementation must process fields until it encounters the first non-FIXT.1.1 header field before it knows the actual message type. In < 5.0 the message type is always known after reading the first three fields, which is a good thing. So if we could extend this to be the first 4 fields for >= 5.0 it would be much better.
(I guess the above reasoning should be extended to include CstmApplVerID too)
Kind Regards,
David
1 - A definitive line should be drawn - either a field belongs to
Transport layer ( what I call the envelope ) or it belongs to the
Application layer ( what I call the payload ). Drawing a clear line
would make the two layers independent of each other.2 - In my previous posting in this thread of discussion
http://fixprotocol.org/discuss/read/d5962a4b
I suggest that ApplVerID be part of Body of the message
3.a - Once the Session layer and Application layer are clearly
seperated, the application layer can contain message in any format
while the session layer is tag=value^3.a.1 - BodyLength is part of Session layer.
3.b.2 - MsgType and ApplVerID should come next to each other and these
two fields becomes part of body. Both of these fields are used to
parse the message and validate its contents.3.c.3 - If MsgType is moved to body, then multiple Application messages
in different formats of different versions can be sent in a single
Session message. Presently since MsgType is part of header, one FIX
message contains only one Application message. By moving MsgType to
Message Body, multiple Application messages can be sent in a single
FIX message i.e. a single envelope can contain multiple payloads.
This way FIX tag=value^ and FIXML can be sent in a single message.Regards,
K. Mahesh +1-203-252-4039
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
Hi David,
Has there been any more discussion on this matter?
The GTC FIX 5.0 Usage Subcommittee (http://www.fixprotocol.org/committees/fix5.0usage) was formed in June to begin addressing these issues. We have been focusing on the first issue mentioned at the top of this thread (and that you re-iterate in your later posting http://www.fixprotocol.org/discuss/read/4ed37190) – the dependency of both the application and session on the same data repository. We are working on creating independent repositories for application and session data.
I think there are two separate issues that have been vetted here
- Whether ApplVerID and MsgType belong together or not
- The placement of this pair in the overall structure
My take on those issues are that
They definitely belong together. Both are needed to uniquely
identify the message type. The ApplVerID can be defaulted to the
DefaultApplVerID passed in the Logon, but this knowledge should also
materialize (that is, detecting the absence of an ApplVerID) in
conjunction with the MsgType.Keep them part of the header for compatibility and
optimization reasons.
While I agree with you on both counts, it’s important to note that this is not something that the FIX 5.0 specification should address. The way in which fields are marshalled is transport specific. FIXT is responsible for defining this format.
- steven