Imported from previous forum
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
I have created a Working Draft of a specification for book messages which is now posted on the FIX web site under "Specifications - Drafts"
The draft lists several open issues. I am looking for recommendations on how to address these issues, as well as comments on the draft itself.
One key reason for Book messages is the ammount of information & hence bandwidth that an exchange can generate/use.
Regardless of exchange usage, I would suggest that are many points in FIX where this will become a problem. The high (and increasing) number of IOI’s being generated is an example.
Adding compression in a similar way to security seems to be a generic way to solve this problem.
Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
Do people feel there is value in this extension?
Regards,
Paul
> I have created a Working Draft of a specification for book messages which is now posted on the FIX web site under "Specifications - Drafts"
>
> The draft lists several open issues. I am looking for recommendations on how to address these issues, as well as comments on the draft itself.
>
>
[ original email was from Yevgeniy Tovshteyn - yevgeniy@javtech.com ]
Why couldn’t we use RawData and RawDataLen for this purpose?
> One key reason for Book messages is the ammount of information & hence bandwidth that an exchange can generate/use.
>
> Regardless of exchange usage, I would suggest that are many points in FIX where this will become a problem. The high (and increasing) number of IOI’s being generated is an example.
>
> Adding compression in a similar way to security seems to be a generic way to solve this problem.
>
> Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
>
> Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
>
> Do people feel there is value in this extension?
>
> Regards,
> Paul
>
> > I have created a Working Draft of a specification for book messages which is now posted on the FIX web site under “Specifications - Drafts”
> >
> > The draft lists several open issues. I am looking for recommendations on how to address these issues, as well as comments on the draft itself.
> >
> >
>
[ original email was from Yevgeniy Tovshteyn - yevgeniy@javtech.com ]
ps: this field just must be transformed to the Header in order all messages would have it. Or it should be in the messages that need compression (now it is just in Logon, News and Email)
> Why couldn’t we use RawData and RawDataLen for this purpose?
>
> > One key reason for Book messages is the ammount of information & hence bandwidth that an exchange can generate/use.
> >
> > Regardless of exchange usage, I would suggest that are many points in FIX where this will become a problem. The high (and increasing) number of IOI’s being generated is an example.
> >
> > Adding compression in a similar way to security seems to be a generic way to solve this problem.
> >
> > Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
> >
> > Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
> >
> > Do people feel there is value in this extension?
> >
> > Regards,
> > Paul
> >
> > > I have created a Working Draft of a specification for book messages which is now posted on the FIX web site under “Specifications - Drafts”
> > >
> > > The draft lists several open issues. I am looking for recommendations on how to address these issues, as well as comments on the draft itself.
> > >
> > >
> >
>
If the recommendation from this working group is to support encryption, I, personally, would be more in favor of adding CompressedDataLen and CompressedData to the header and CompressionType to the Logon than trying to use/re-use RawData which is a body field of certain messages today. The compression algorithm(s) would need to be specified and agreed to.
> ps: this field just must be transformed to the Header in order all messages would have it. Or it should be in the messages that need compression (now it is just in Logon, News and Email)
>
> > Why couldn’t we use RawData and RawDataLen for this purpose?
> >
> > > One key reason for Book messages is the ammount of information & hence bandwidth that an exchange can generate/use.
> > >
> > > Regardless of exchange usage, I would suggest that are many points in FIX where this will become a problem. The high (and increasing) number of IOI’s being generated is an example.
> > >
> > > Adding compression in a similar way to security seems to be a generic way to solve this problem.
> > >
> > > Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
> > >
> > > Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
> > >
> > > Do people feel there is value in this extension?
> > >
> > > Regards,
> > > Paul
> > >
> > > > I have created a Working Draft of a specification for book messages which is now posted on the FIX web site under “Specifications - Drafts”
> > > >
> > > > The draft lists several open issues. I am looking for recommendations on how to address these issues, as well as comments on the draft itself.
> > > >
> > > >
> > >
> >
>
[ original email was from John Lopez - john.e.lopez@usa.net ]
I think compression is at the network transport layer and should be excluded from convoluting the specification.
John
> If the recommendation from this working group is to support encryption, I, personally, would be more in favor of adding CompressedDataLen and CompressedData to the header and CompressionType to the Logon than trying to use/re-use RawData which is a body field of certain messages today. The compression algorithm(s) would need to be specified and agreed to.
>
>
> > ps: this field just must be transformed to the Header in order all messages would have it. Or it should be in the messages that need compression (now it is just in Logon, News and Email)
> >
> > > Why couldn’t we use RawData and RawDataLen for this purpose?
> > >
> > > > One key reason for Book messages is the ammount of information & hence bandwidth that an exchange can generate/use.
> > > >
> > > > Regardless of exchange usage, I would suggest that are many points in FIX where this will become a problem. The high (and increasing) number of IOI’s being generated is an example.
> > > >
> > > > Adding compression in a similar way to security seems to be a generic way to solve this problem.
> > > >
> > > > Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
> > > >
> > > > Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
> > > >
> > > > Do people feel there is value in this extension?
> > > >
> > > > Regards,
> > > > Paul
> > > >
> > > > > I have created a Working Draft of a specification for book messages which is now posted on the FIX web site under “Specifications - Drafts”
> > > > >
> > > > > The draft lists several open issues. I am looking for recommendations on how to address these issues, as well as comments on the draft itself.
> > > > >
> > > > >
> > > >
> > >
> >
>
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> Adding compression in a similar way to security seems to be a generic way to solve this problem.
I agree compression is needed in FIX.
> Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
>
> Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
I think this is the wrong way to go about doing it, as it either will result in poor compression, or will repeat a serious mistake made with encryption.
The issue is whether encryption or compression works with one message at a time, or in the context of the whole stream of messages.
If a message is compressed or encrypted independently of others, then a CompressedDataLen / CompressedData style approach embedded in a normal FIX message will work just fine.
But using an encryption approach like DES Cypher Block Chaining, which uses the previous message as the initialization vector for the next message, or treating the messages as one long stream that is compressed, and then embedding the compressed/encrypted data in the FIX message causes serious problems should messages be dropped or otherwise experience abnormalities that FIX would otherwise be able to recover.
Compression works by reducing redundancy. Compressing just the FIX message independently of other messages results in less compression by far than compressing a stream of FIX messages, since all the Book messages will share similarities with each other. In addition, compressing a stream of messages provides much savings on repetitive headers; just think how much savings would be had given that all messages messages must begin with the same 12 bytes "8=FIX.4.1[SOH]9="
This is why the general view of the Encryption working group was to use SSL or TLS and put the encryption outside of and below the FIX protocol, and then address signatures and end-to-end authentication within each message. I would think the same should be true of compression.
Either both counterparties could agree to use a specific type of compression for the entire session, or some negotiation can happen in an uncompressed Logon, and then the whole stream would be compressed.
If you wish to use both encryption and compression, you must compress the data first, as any encryption algorithm worth its salt produces cyphertext that cannot be compressed.
TLS does support different compression types, so it has the potential to handle both the encryption and compression, however it looks like the only compression type defined at present is none. If compression gets standardized into TLS and widely implemented, then it may be possible to use TLS just to negotiate compression with null encryption.
[ original email was from Yevgeniy Tovshteyn - yevgeniy@javtech.com ]
I think compression shouldn’t be performed on the entire session, but on selected messages and fields. Because compression compress messages bigger than a certain size; it might make small messages longer.
> > Adding compression in a similar way to security seems to be a generic way to solve this problem.
>
> I agree compression is needed in FIX.
>
> > Adding CompressedDataLen and CompressedData fields to FIX will allow any FIX message to be compressed.
> >
> > Furthermore, if security and compression are required, CompressedDataLen and CompressedData fields can themselves be embedded within SecureDataLen and SecureData fields.
>
> I think this is the wrong way to go about doing it, as it either will result in poor compression, or will repeat a serious mistake made with encryption.
>
> The issue is whether encryption or compression works with one message at a time, or in the context of the whole stream of messages.
>
> If a message is compressed or encrypted independently of others, then a CompressedDataLen / CompressedData style approach embedded in a normal FIX message will work just fine.
>
> But using an encryption approach like DES Cypher Block Chaining, which uses the previous message as the initialization vector for the next message, or treating the messages as one long stream that is compressed, and then embedding the compressed/encrypted data in the FIX message causes serious problems should messages be dropped or otherwise experience abnormalities that FIX would otherwise be able to recover.
>
> Compression works by reducing redundancy. Compressing just the FIX message independently of other messages results in less compression by far than compressing a stream of FIX messages, since all the Book messages will share similarities with each other. In addition, compressing a stream of messages provides much savings on repetitive headers; just think how much savings would be had given that all messages messages must begin with the same 12 bytes "8=FIX.4.1[SOH]9="
>
> This is why the general view of the Encryption working group was to use SSL or TLS and put the encryption outside of and below the FIX protocol, and then address signatures and end-to-end authentication within each message. I would think the same should be true of compression.
>
> Either both counterparties could agree to use a specific type of compression for the entire session, or some negotiation can happen in an uncompressed Logon, and then the whole stream would be compressed.
>
> If you wish to use both encryption and compression, you must compress the data first, as any encryption algorithm worth its salt produces cyphertext that cannot be compressed.
>
> TLS does support different compression types, so it has the potential to handle both the encryption and compression, however it looks like the only compression type defined at present is none. If compression gets standardized into TLS and widely implemented, then it may be possible to use TLS just to negotiate compression with null encryption.
>
>
[ original email was from Nikhil Bose - assistsoft@yahoo.com ]
Hello,
Sorry I have nothing to say about compression or encryption. I have a couple of questions about the draft, "Book Messages":
-
Why do two different messages, Book (Snapshot / Full Refresh), and Book (Incremental Refresh), have the same MsgType of "UB"? Is this just a typo, or is there a reason for it?
-
Why was tag 5013 left unused? Again, is there a reason for it?
Finally, when will this draft become a standard? Any information on its implementation status would be greatly appreciated.
Thanks,
Nikhil
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> 1) Why do two different messages, Book (Snapshot / Full Refresh), and Book (Incremental Refresh), have the same MsgType of "UB"? Is this just a typo, or is there a reason for it?
This was done because MsgTypes are currently scarce. People would like to stay with 1-character message types as long as possible. When/if this merges into the final FIX spec, each of these messages would be assigned a real MsgType as opposed to the user defined “U*” message space. Already the proposal requires 3 MsgType’s; I didn’t want to ask for 4.
The idea was that based on the BookReqID, one could determine which of the Book formats to use. The second format is used only if the Book Request had BookReqType = 1 and BookUpdateType = 1.
If people believe it would be better to use 4 MsgType’s, I’d have no objection.
> 2) Why was tag 5013 left unused? Again, is there a reason for it?
When preparing the draft, an extra field was thrown in which shouldn’t have been included. It was removed, and rather than renumber all fields after it, we left a gap. When/if this is incorporated into a released spec, real tags will be assigned, and there will not be a gap.
> Finally, when will this draft become a standard? Any information on its implementation status would be greatly appreciated.
Its status is a FIX Technical Committee Working Draft. This means:
"They are draft documents and may be updated, replaced, or obsoleted by other documents at any time. The Working Groups responsible for these documents will not allow early implementation to constrain their ability to make changes to these specifications prior to final release. It is inappropriate to use FIX Technical Committee Working Drafts as reference material or to cite them as other than ‘works in progress.’ "
The draft has some open issues, highlighted in the draft, which need to be addressed prior to inclusion in the spec.
I must stress that it is certainly open to comment and revision. The whole point of doing this is to create something that as many people as possible find useable.
People certainly can make implementations of the draft, with the understanding that the draft is a work in progress subject to change. Such early implementations would help ensure that the draft stands up under real world conditions and meets the needs of its users.
I would appreciate being kept informed of such implementations, as well as any modifications people find necessary, so that if they are useful to others they can be incorporated into a later version of the draft.
My understanding is that once the draft is refined, the FIX Technical Committee would decide whether to include it in the next upcoming FIX specification.
[ original email was from Nikhil Bose - assistsoft@yahoo.com ]
Thanks Ryan for an extremely clear explanation.
For what its worth, I am very much in favour of 4 MsgType’s instead of 3. Complicating the message type detection process is something I am not in favour of.
We will run out of 1-character MsgType codes sooner or later. We might as well bite that bullet sooner rather than later. Increasing the complexity of message type detection to save one code is not worth it, IMHO. For implementors, its more important to keep the rules simple, uniform, and easy to state as possible.
Simplicity is one of the biggest virtues of FIX, and IMHO it would be good to preserve that as long as practical. Swift on the other hand, puts lots of semantics into its CODE WORD / QUALIFIER, which makes the new message types essentially kitchen sinks with all kinds of stuff thrown in.
Regards,
Nikhil
> > 1) Why do two different messages, Book (Snapshot / Full Refresh), and Book (Incremental Refresh), have the same MsgType of “UB”? Is this just a typo, or is there a reason for it?
>
> This was done because MsgTypes are currently scarce. People would like to stay with 1-character message types as long as possible. When/if this merges into the final FIX spec, each of these messages would be assigned a real MsgType as opposed to the user defined “U*” message space. Already the proposal requires 3 MsgType’s; I didn’t want to ask for 4.
>
> The idea was that based on the BookReqID, one could determine which of the Book formats to use. The second format is used only if the Book Request had BookReqType = 1 and BookUpdateType = 1.
>
> If people believe it would be better to use 4 MsgType’s, I’d have no objection.
>
> > 2) Why was tag 5013 left unused? Again, is there a reason for it?
>
> When preparing the draft, an extra field was thrown in which shouldn’t have been included. It was removed, and rather than renumber all fields after it, we left a gap. When/if this is incorporated into a released spec, real tags will be assigned, and there will not be a gap.
>
> > Finally, when will this draft become a standard? Any information on its implementation status would be greatly appreciated.
>
> Its status is a FIX Technical Committee Working Draft. This means:
>
> "They are draft documents and may be updated, replaced, or obsoleted by other documents at any time. The Working Groups responsible for these documents will not allow early implementation to constrain their ability to make changes to these specifications prior to final release. It is inappropriate to use FIX Technical Committee Working Drafts as reference material or to cite them as other than ‘works in progress.’ "
>
> The draft has some open issues, highlighted in the draft, which need to be addressed prior to inclusion in the spec.
>
> I must stress that it is certainly open to comment and revision. The whole point of doing this is to create something that as many people as possible find useable.
>
> People certainly can make implementations of the draft, with the understanding that the draft is a work in progress subject to change. Such early implementations would help ensure that the draft stands up under real world conditions and meets the needs of its users.
>
> I would appreciate being kept informed of such implementations, as well as any modifications people find necessary, so that if they are useful to others they can be incorporated into a later version of the draft.
>
> My understanding is that once the draft is refined, the FIX Technical Committee would decide whether to include it in the next upcoming FIX specification.
>
>
[ original email was from Nikhil Bose - assistsoft@yahoo.com ]
Hello,
Sorry to follow up to my own post. But in the process of implementing the new Book messages, I realized a couple other reasons that distinct MsgType’s might be more convenient:
-
In this particular case, when an engine receives a Book message it has to retrieve the original Book Request message that this was Book message was a response to, to determine the values of the BookReqType and BookUpdateType fields. Only then will it know which message it is receiving. Which means the engine won’t know what fields (tags) to expect unless it holds on to all Book Request messages. This is inconvenient for implementors of pure FIX engines, that simply pass Application level FIX messages on to backend application servers.
-
Since fields in FIX messages are not ordered, the field that determines the template (true message type) of a message being received might occur at the end of the message. So an engine might have to receive an entire message before it knows what message it received. This is not a fatal problem. But its very convenient to know right at the start of receiving a message, what message is coming in, and what tags to expect. This makes debugging, diagnostics, and error tracing a little more convenient.
Just my thoughts,
Nikhil
> Thanks Ryan for an extremely clear explanation.
>
> For what its worth, I am very much in favour of 4 MsgType’s instead of 3. Complicating the message type detection process is something I am not in favour of.
>
> We will run out of 1-character MsgType codes sooner or later. We might as well bite that bullet sooner rather than later. Increasing the complexity of message type detection to save one code is not worth it, IMHO. For implementors, its more important to keep the rules simple, uniform, and easy to state as possible.
>
> Simplicity is one of the biggest virtues of FIX, and IMHO it would be good to preserve that as long as practical. Swift on the other hand, puts lots of semantics into its CODE WORD / QUALIFIER, which makes the new message types essentially kitchen sinks with all kinds of stuff thrown in.
>
> Regards,
> Nikhil
>
>
>
>
>
>
> > > 1) Why do two different messages, Book (Snapshot / Full Refresh), and Book (Incremental Refresh), have the same MsgType of “UB”? Is this just a typo, or is there a reason for it?
> >
> > This was done because MsgTypes are currently scarce. People would like to stay with 1-character message types as long as possible. When/if this merges into the final FIX spec, each of these messages would be assigned a real MsgType as opposed to the user defined “U*” message space. Already the proposal requires 3 MsgType’s; I didn’t want to ask for 4.
> >
> > The idea was that based on the BookReqID, one could determine which of the Book formats to use. The second format is used only if the Book Request had BookReqType = 1 and BookUpdateType = 1.
> >
> > If people believe it would be better to use 4 MsgType’s, I’d have no objection.
> >
> > > 2) Why was tag 5013 left unused? Again, is there a reason for it?
> >
> > When preparing the draft, an extra field was thrown in which shouldn’t have been included. It was removed, and rather than renumber all fields after it, we left a gap. When/if this is incorporated into a released spec, real tags will be assigned, and there will not be a gap.
> >
> > > Finally, when will this draft become a standard? Any information on its implementation status would be greatly appreciated.
> >
> > Its status is a FIX Technical Committee Working Draft. This means:
> >
> > "They are draft documents and may be updated, replaced, or obsoleted by other documents at any time. The Working Groups responsible for these documents will not allow early implementation to constrain their ability to make changes to these specifications prior to final release. It is inappropriate to use FIX Technical Committee Working Drafts as reference material or to cite them as other than ‘works in progress.’ "
> >
> > The draft has some open issues, highlighted in the draft, which need to be addressed prior to inclusion in the spec.
> >
> > I must stress that it is certainly open to comment and revision. The whole point of doing this is to create something that as many people as possible find useable.
> >
> > People certainly can make implementations of the draft, with the understanding that the draft is a work in progress subject to change. Such early implementations would help ensure that the draft stands up under real world conditions and meets the needs of its users.
> >
> > I would appreciate being kept informed of such implementations, as well as any modifications people find necessary, so that if they are useful to others they can be incorporated into a later version of the draft.
> >
> > My understanding is that once the draft is refined, the FIX Technical Committee would decide whether to include it in the next upcoming FIX specification.
> >
> >
>
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> Sorry to follow up to my own post. But in the process of implementing the new Book messages, I realized a couple other reasons that distinct MsgType’s might be more convenient:
[snip]
Okay, okay, I am now thoroughly convinced.
I am assuming that because this is in the working draft stage, it can still be freely changed. I plan on releasing a new version of the draft shortly, with the two message response formats separated. To maintain the order in the document, the incremental refresh Book message would be type UC, and the Book Request Reject would be UD, assuming that UD has not been assigned to any other working draft in progress.
I also plan on adding the closing and settlement price requests, and provisional and final price responses as specified in the Options discussion group at the same time.
Is there anything else I’m neglecting that should be added or changed?
[ original email was from Ryan Pierce - rpierce@taltrade.com ]
> I think compression shouldn’t be performed on the entire session, but on selected messages and fields. Because compression compress messages bigger than a certain size; it might make small messages longer.
First, I must state I am not a compression algorithm guru.
But my understanding is that even the smallest of FIX messages, the Heartbeat, contains a whole lot of repetition from one message to the next; only a few bytes really change. I would imagine that a compression algorithm which works on the entire FIX message stream could do quite a nice job at seeing the similarity between the messages.
The individual messages themselves don’t have as much internal repetition, so I’d imagine they would be individually less compressible than a whole stream of similar-looking messages.
[ original email was from Yevgeniy Tovshteyn - yevgeniy@javtech.com ]
What you say is right, but how are you going to let compression algorithm analize the entire session to perform good compression if the session is not yet completed.
I think most compression algorithms have to analize the entire data-to-be-compressed before performing compression. May be in our case compression would be efficient if an algorithm is armed only with past data analizes since big portions of many messages are alike. But I don’t know if there are compression algorithms that could do that. Does anybody know?
> > I think compression shouldn’t be performed on the entire session, but on selected messages and fields. Because compression compress messages bigger than a certain size; it might make small messages longer.
>
> First, I must state I am not a compression algorithm guru.
>
> But my understanding is that even the smallest of FIX messages, the Heartbeat, contains a whole lot of repetition from one message to the next; only a few bytes really change. I would imagine that a compression algorithm which works on the entire FIX message stream could do quite a nice job at seeing the similarity between the messages.
>
> The individual messages themselves don’t have as much internal repetition, so I’d imagine they would be individually less compressible than a whole stream of similar-looking messages.
>
>
[ original email was from Nikhil Bose - assistsoft@yahoo.com ]
How about a FIX committee inventing a compression algorithm specifically for FIX. Clearly, different compression algorithms work best for different situations. There may be enough FIX-specific features (attributes) of FIX messages that it makes sense for FIX to have its compression algorithm. The only thing you need is that both sides understand and use the same algorithm. For instance, a couple of ideas that could be used are:
-
Integers such as tags could be transmitted in binary using the IIOP representation of int’s.
-
Many fields that have the same value for every (most) message in the same session can use a short code instead of a long string. An example would be SenderCompID, TargetCompID, etc. The two parties could exchange the code in advance.
-
The compression algorithm could be adaptive, and a protocol arranged for the two ends to exchange compression information before starting to use it. The fact that FIX practically guarantees delivery gives us a big benefit that we can use to exchange compression information adaptively, as the session progresses.
Regards,
Nikhil
> What you say is right, but how are you going to let compression algorithm analize the entire session to perform good compression if the session is not yet completed.
>
> I think most compression algorithms have to analize the entire data-to-be-compressed before performing compression. May be in our case compression would be efficient if an algorithm is armed only with past data analizes since big portions of many messages are alike. But I don’t know if there are compression algorithms that could do that. Does anybody know?
>