Imported from previous forum
Hi All,
As discussed during last meeting.
Please find hereafter some ideas for the working group.
Feel free to comment on this forum thread.
Kind regards
Merry Christmas
Georges
Hi Georges,
I read thru the section “Encode Fields in compact binary form” and wanted to get a clarification. On pages 25/48 and 26/48, for all formats other than String, the length of the value in bytes is known from the value of the “TBD” 7 bits. But in case of String, the parser will know its a String value, but would not know the length of the String and hence where next Field header starts.
In the 1 byte field header, presently 1st bit indicates the number of bytes for tag and 2nd to 8th bits is value encoded format.
For 2nd to 8th bits instead if we use as follows, the “Encoding Strings in compact binary form” problem could be solved with some limitations.
2nd bit indicates datatype
0 = String datatype
1 = Numeric datatype
3rd to 8th bits -
when 2nd bit = 0, these bits represent the length of String value i.e. 63 char long String values can be encoded.
when 2nd bit = 1, these bits represent one of the TBD values for numeric datatypes whose length is known.
Thus after reading and parsing a field header, the decoder knows exactly how many bytes forward to read for this field (tag + value) and how to interpret the value.
Regards,
K. Mahesh
Hi All,
As discussed during last meeting.
Please find hereafter some ideas for the working group.ULLINK - Analysis and proposal for FIX HFT - December 2010 | PPT
Feel free to comment on this forum thread.
Kind regards
Merry Christmas
Georges
Mahesh,
this is reminiscent of one of the early mdowg proposals at the end of January 2005. What we tried to create at the time was a sort of virtual machine instruction set that when executed generated unpacked data. The proposal was called “OPF - Opcode Binary Format” and may be available in some archive. The proposal was later superceeded by “BMF - Byte Mapped Format” that later in an enhanced form became FAST.
The primary reason for not going with OPF was that BMF was more compact. Contrary to what we guessed at the outset, the software implementation of OPF wasn’t faster than BMF. Both OPF and BMF were byte oriented formats (as is FAST).
There was some discussion early on about going to a bit-oriented format but there was considerable push-back and early testing showed increased complexity, much lower readability (yes, that was indeed an argument), and higher encoding and decoding processing overhead.
/Rolf
…
In the 1 byte field header, presently 1st bit indicates the number
of bytes for tag and 2nd to 8th bits is value encoded format.For 2nd to 8th bits instead if we use as follows, the “Encoding
Strings in compact binary form” problem could be solved with some
limitations.2nd bit indicates datatype
0 = String datatype
1 = Numeric datatype3rd to 8th bits -
when 2nd bit = 0, these bits represent the length of String value i.e. 63 char long String values can be encoded. when 2nd bit = 1, these bits represent one of the TBD values for numeric datatypes whose length is known.
Rolf,
Since FAST is a byte oriented format (as is BMF and OPF), should HFT try the bit oriented format because any further work on byte oriented optimizations could be carried out under FAST improvements.
Or should we revisit FIX over FAST and see why FOF does not suffice for HFT ?
Or could we try a different byte oriented encoding format ?
Regards,
Mahesh
Mahesh,
this is reminiscent of one of the early mdowg proposals at the end of January 2005. What we tried to create at the time was a sort of virtual machine instruction set that when executed generated unpacked data. The proposal was called “OPF - Opcode Binary Format” and may be available in some archive. The proposal was later superceeded by “BMF - Byte Mapped Format” that later in an enhanced form became FAST.
The primary reason for not going with OPF was that BMF was more compact. Contrary to what we guessed at the outset, the software implementation of OPF wasn’t faster than BMF. Both OPF and BMF were byte oriented formats (as is FAST).
There was some discussion early on about going to a bit-oriented format but there was considerable push-back and early testing showed increased complexity, much lower readability (yes, that was indeed an argument), and higher encoding and decoding processing overhead.
/Rolf
…
In the 1 byte field header, presently 1st bit indicates the number
of bytes for tag and 2nd to 8th bits is value encoded format.For 2nd to 8th bits instead if we use as follows, the “Encoding
Strings in compact binary form” problem could be solved with some
limitations.2nd bit indicates datatype
0 = String datatype
1 = Numeric datatype3rd to 8th bits -
when 2nd bit = 0, these bits represent the length of String value i.e. 63 char long String values can be encoded. when 2nd bit = 1, these bits represent one of the TBD values for numeric datatypes whose length is known.
Mahesh,
these are important questions. I think the answers depend on how we want to trade off compactness, speed, processing overhead, implementation complexity (and possibly other factors).
My current thinking re each of your questions is as follows:
-
A bit-oriented approach increases the processing overhead and I haven’t (yet) seen a proposal that is worth the cost. Compactness is (still) needed but my guess is that the tipping point is before going to bits.
-
I believe that FAST is a good candidate to start with from an encoding perspective, but there is much more to it. We are currently doing some work within the mdowg to define a few minor extensions to FAST. The 1.3 extension proposal is a result of feedback from practioners (as so often in FIX). The proposed extension will most likely be of use for HFT, should we choose to use FAST encoding in whole or in part. We’ll see what we arrive at after design and validation.
-
A different encoding from FAST will most likely be similar to FAST. There are many genetic variations, but they share a common base. Most of what we do today has been done before.
(Sidenote: For example, you can find traces as far back as the 1960’s in the design components of FAST. This is not a coincidence since both Dan and I had worked with market data in the 80’s and early 90’s.)
Best,
Rolf
Since FAST is a byte oriented format (as is BMF and OPF), should HFT try the bit oriented format because any further work on byte oriented optimizations could be carried out under FAST improvements.
Or should we revisit FIX over FAST and see why FOF does not suffice for HFT ?
Or could we try a different byte oriented encoding format ?
Hi Mahesh,
I initialy thought of encoding string with an additional varint in front of the string to specify the size of the string. varint can bring up to 128 char len.
What you propose is interesting, it brings a compact len of the string without additional byte.
7 bits gives us 64 bytes of string which is usualy good enough for FIX tags…
This leaves also 64 others values for decimal/other encoding - may be enough actually.
Only a couple of tag are strings like “Symbol” and “ClOrdID”. And as Rolf was mentionning to me privately, "ClOrdID"s are frequently pure numeric and could be encoded in decimal. So the gain is very limited.
My only show stopper to add it into my proposal is: what do we do if string is >64 bytes?
Regards
Georges
Hi Georges,
I read thru the section “Encode Fields in compact binary form” and wanted to get a clarification. On pages 25/48 and 26/48, for all formats other than String, the length of the value in bytes is known from the value of the “TBD” 7 bits. But in case of String, the parser will know its a String value, but would not know the length of the String and hence where next Field header starts.
In the 1 byte field header, presently 1st bit indicates the number of bytes for tag and 2nd to 8th bits is value encoded format.
For 2nd to 8th bits instead if we use as follows, the “Encoding Strings in compact binary form” problem could be solved with some limitations.
2nd bit indicates datatype
0 = String datatype
1 = Numeric datatype3rd to 8th bits -
when 2nd bit = 0, these bits represent the length of String value i.e. 63 char long String values can be encoded. when 2nd bit = 1, these bits represent one of the TBD values for numeric datatypes whose length is known.Thus after reading and parsing a field header, the decoder knows exactly how many bytes forward to read for this field (tag + value) and how to interpret the value.
Regards,
K. MaheshHi All,
As discussed during last meeting.
Please find hereafter some ideas for the working group.ULLINK - Analysis and proposal for FIX HFT - December 2010 | PPT
Feel free to comment on this forum thread.
Kind regards
Merry Christmas
Georges
Georges,
re string lengths > 63 bytes
other (similar) encodings have multiple string types; in addition to the 0-63 byte string type, you could also define one or more types that can encode strings with >63 bytes. For example, you could assign a type that has a 4 byte length followed by the data:
(see for example amqp short (1-255) and long (1-2^32-1) strings)
/Rolf
Hi Mahesh,
I initialy thought of encoding string with an additional varint in front of the string to specify the size of the string. varint can bring up to 128 char len.
What you propose is interesting, it brings a compact len of the string without additional byte.
7 bits gives us 64 bytes of string which is usualy good enough for FIX tags…
This leaves also 64 others values for decimal/other encoding - may be enough actually.
Only a couple of tag are strings like “Symbol” and “ClOrdID”. And as Rolf was mentionning to me privately, "ClOrdID"s are frequently pure numeric and could be encoded in decimal. So the gain is very limited.
My only show stopper to add it into my proposal is: what do we do if string is >64 bytes?
Regards
Georges
Hi Georges,
When the String is longer than 63 bytes, then the length of the String in the field header is set to 000000 bits and the first byte(s) of the String contains the length of the String. I can think of two approaches here
-
Fixed number of bytes at the beginning of the String to represent the String value length as integer.
-
The first byte of the String contains the number of bytes which contain the length of the String and subsequent number of bytes contains the actual length of the String as integer.
Thus very large / infinetly large Strings can also be encoded / decoded.
Regards,
K. Mahesh
Georges,
re string lengths > 63 bytes
other (similar) encodings have multiple string types; in addition to the 0-63 byte string type, you could also define one or more types that can encode strings with >63 bytes. For example, you could assign a type that has a 4 byte length followed by the data:
(see for example amqp short (1-255) and long (1-2^32-1) strings)
/Rolf
Hi Mahesh,
I initialy thought of encoding string with an additional varint in front of the string to specify the size of the string. varint can bring up to 128 char len.
What you propose is interesting, it brings a compact len of the string without additional byte.
7 bits gives us 64 bytes of string which is usualy good enough for FIX tags…
This leaves also 64 others values for decimal/other encoding - may be enough actually.
Only a couple of tag are strings like “Symbol” and “ClOrdID”. And as Rolf was mentionning to me privately, "ClOrdID"s are frequently pure numeric and could be encoded in decimal. So the gain is very limited.
My only show stopper to add it into my proposal is: what do we do if string is >64 bytes?
Regards
Georges
Looks good.
I take that for next update of my document.
I add credits, don’t worry 
Thanks
Georges
Hi Georges,
When the String is longer than 63 bytes, then the length of the String in the field header is set to 000000 bits and the first byte(s) of the String contains the length of the String. I can think of two approaches here
Fixed number of bytes at the beginning of the String to represent the String value length as integer.
The first byte of the String contains the number of bytes which contain the length of the String and subsequent number of bytes contains the actual length of the String as integer.
Thus very large / infinetly large Strings can also be encoded / decoded.
Regards,
K. MaheshGeorges,
re string lengths > 63 bytes
other (similar) encodings have multiple string types; in addition to the 0-63 byte string type, you could also define one or more types that can encode strings with >63 bytes. For example, you could assign a type that has a 4 byte length followed by the data:
(see for example amqp short (1-255) and long (1-2^32-1) strings)
/Rolf
Hi Mahesh,
I initialy thought of encoding string with an additional varint in front of the string to specify the size of the string. varint can bring up to 128 char len.
What you propose is interesting, it brings a compact len of the string without additional byte.
7 bits gives us 64 bytes of string which is usualy good enough for FIX tags…
This leaves also 64 others values for decimal/other encoding - may be enough actually.
Only a couple of tag are strings like “Symbol” and “ClOrdID”. And as Rolf was mentionning to me privately, "ClOrdID"s are frequently pure numeric and could be encoded in decimal. So the gain is very limited.
My only show stopper to add it into my proposal is: what do we do if string is >64 bytes?
Regards
Georges
Georges,
This is great analysis. There are many useful ideas here on how to compress FIX messages and reduce processing time. What is not quite clear to me is which parts of FIX you are suggesting to keep and why. It appears that you aim to achieve performance gains by addressing message encoding while preserving FIX semantics in full, both session level and application level. I concede that simply switching to binary message format is perhaps the best way to improve performance. However, given that FIX engines will need to be rewritten to process new message formats anyway, it is relevant to ask whether more radical changes to FIX message structure and semantics is worth considering to attain even better performance.
For example, instead of spending a full byte per tag to encode its format, one can define small fixed size messages with fields in predefined formats for each message type. Various fixed extensions for most common options can also be considered. This approach reduces processing time not only by reducing message size but also by giving the decoder faster access to message fields by virtue of them being located at well-known rather than arbitrary offsets. Mandating field order (for most common cases) has an added benefit of reducing decoder complexity.
I do not insist that such approach is superior, although it does look attractive. My point is that you start with specific assumptions about which parts of FIX semantics you are keeping. I suggest re-examining these assumptions.
Thanks,
Vitali
Vitali,
your point on looking at more radical changes is well made. Whereas the syntactical level (encoding and decoding) is a necessary step I believe that the dramatic improvements will be found elsewhere.
Removing less useful fields (through message specialization), narrowing value domains (eg. using numbers instead of strings), simplifying recovery semantics (and therefore decreasing state overhead) will most likely bring much better performance especially in configurations with a large number of sessions.
/Rolf
Georges,
This is great analysis. There are many useful ideas here on how to compress FIX messages and reduce processing time. What is not quite clear to me is which parts of FIX you are suggesting to keep and why. It appears that you aim to achieve performance gains by addressing message encoding while preserving FIX semantics in full, both session level and application level. I concede that simply switching to binary message format is perhaps the best way to improve performance. However, given that FIX engines will need to be rewritten to process new message formats anyway, it is relevant to ask whether more radical changes to FIX message structure and semantics is worth considering to attain even better performance.
For example, instead of spending a full byte per tag to encode its format, one can define small fixed size messages with fields in predefined formats for each message type. Various fixed extensions for most common options can also be considered. This approach reduces processing time not only by reducing message size but also by giving the decoder faster access to message fields by virtue of them being located at well-known rather than arbitrary offsets. Mandating field order (for most common cases) has an added benefit of reducing decoder complexity.
I do not insist that such approach is superior, although it does look attractive. My point is that you start with specific assumptions about which parts of FIX semantics you are keeping. I suggest re-examining these assumptions.
Thanks,
Vitali
Hi Vitali,
My main concern was to preserve the FIX specifications. Because there is a lot of business value into it. Redesigning it would be a waist of time in my perspective. But maybe I sticked too much with the tag=value style.
Prepared messages (like templates in FAST) are a way to get away from this FIX semantic. Then you have two choses to use them: fixed size and variable size.
Fixed size format will get the best decoding performance, I agree with you. The only problem is that fixed size don’t give great ‘compression’. For exemple, for tag “quantity”, since you don’t know if it’s 100 or 100 000 you have to arbitrary encode it on 4bytes to cover any possible values.
With non-fixed size, “quantity” of 100 would be encoded in 1 unique byte (for exemple)
I wanted something that was a balance between message size and encoding/decoding speed. I’ve gone this route because I believe that recent processors are super fast when we play at the register level.
For me, message size is the bottleneck because many different layers in the network and in the system are slower if message is bigger. But it’s only a guess I haven’t compare both approach empirically.
Regards
Georges
Georges,
This is great analysis. There are many useful ideas here on how to compress FIX messages and reduce processing time. What is not quite clear to me is which parts of FIX you are suggesting to keep and why. It appears that you aim to achieve performance gains by addressing message encoding while preserving FIX semantics in full, both session level and application level. I concede that simply switching to binary message format is perhaps the best way to improve performance. However, given that FIX engines will need to be rewritten to process new message formats anyway, it is relevant to ask whether more radical changes to FIX message structure and semantics is worth considering to attain even better performance.
For example, instead of spending a full byte per tag to encode its format, one can define small fixed size messages with fields in predefined formats for each message type. Various fixed extensions for most common options can also be considered. This approach reduces processing time not only by reducing message size but also by giving the decoder faster access to message fields by virtue of them being located at well-known rather than arbitrary offsets. Mandating field order (for most common cases) has an added benefit of reducing decoder complexity.
I do not insist that such approach is superior, although it does look attractive. My point is that you start with specific assumptions about which parts of FIX semantics you are keeping. I suggest re-examining these assumptions.
Thanks,
Vitali
Hi Georges,
I agree with you regarding preserving the FIX specification because there is a lot of business value into it. One thing I would like to preserve in any HFT protocol design is the flexiblity FIX has to offer. For example, OUCH protocol of NASDAQ is faster but not flexible - the only thing that can be done using OUCH is trade on NASDAQ. But FIX can be used to trade across most geographies of the world on most asset classes using many diverse trading styles. FIX TCP Sessions can transport non FIX data reliably. FIXProtocol is much more than just a trading protocol.
This calls into question our group’s choices of flexibility versus performance. My present thought is that when FIX started as Tag=Value^ encoded bytes over TCP/IP, it was flexibility which led to its widespread adoption. Now any changes to FIX which make it faster by reducing flexibility would not be well received by the industry.
For example, if there is a new version of FIX which is faster but cannot accept Tag 58 Text messages longer than 63 bytes, many end users would say “there is no need to upgrade, we want the full tag 58 Text without truncation because our secret Algo parameters are coded in tag 58 text value” or “The text messages are very informative, we want it fully transmitted”.
With semantics and business workflows unchanged, if we are able to offer a new HFT encoding to make FIX faster with no loss of flexibility, it could receive wider industry adoption.
Regards,
K. Mahesh
Hi Vitali,
My main concern was to preserve the FIX specifications. Because there is a lot of business value into it. Redesigning it would be a waist of time in my perspective. But maybe I sticked too much with the tag=value style.
Prepared messages (like templates in FAST) are a way to get away from this FIX semantic. Then you have two choses to use them: fixed size and variable size.
Fixed size format will get the best decoding performance, I agree with you. The only problem is that fixed size don’t give great ‘compression’. For exemple, for tag “quantity”, since you don’t know if it’s 100 or 100 000 you have to arbitrary encode it on 4bytes to cover any possible values.
With non-fixed size, “quantity” of 100 would be encoded in 1 unique byte (for exemple)
I wanted something that was a balance between message size and encoding/decoding speed. I’ve gone this route because I believe that recent processors are super fast when we play at the register level.
For me, message size is the bottleneck because many different layers in the network and in the system are slower if message is bigger. But it’s only a guess I haven’t compare both approach empirically.
Regards
GeorgesGeorges,
This is great analysis. There are many useful ideas here on how to compress FIX messages and reduce processing time. What is not quite clear to me is which parts of FIX you are suggesting to keep and why. It appears that you aim to achieve performance gains by addressing message encoding while preserving FIX semantics in full, both session level and application level. I concede that simply switching to binary message format is perhaps the best way to improve performance. However, given that FIX engines will need to be rewritten to process new message formats anyway, it is relevant to ask whether more radical changes to FIX message structure and semantics is worth considering to attain even better performance.
For example, instead of spending a full byte per tag to encode its format, one can define small fixed size messages with fields in predefined formats for each message type. Various fixed extensions for most common options can also be considered. This approach reduces processing time not only by reducing message size but also by giving the decoder faster access to message fields by virtue of them being located at well-known rather than arbitrary offsets. Mandating field order (for most common cases) has an added benefit of reducing decoder complexity.
I do not insist that such approach is superior, although it does look attractive. My point is that you start with specific assumptions about which parts of FIX semantics you are keeping. I suggest re-examining these assumptions.
Thanks,
Vitali
[ original email was from Mark Reece - mark.reece@hsbcib.com ]
I agree that the business flexibility of FIX must be retained. Indeed, a lot of the FIX IPR is captured in these specifications which describe some very varied trading models.
Equally, the current rules for building up messages (irrespective of format) includes mandatory fields. Sometimes these are obvious/reasonable - FIX 4.2 New Order Single requires Tag 55 (Symbol) to be in the message. However, the corresponding Execution Reports are (all) required to also carry Tag 55.
While I can see that it might be useful to have the Symbol, I can also see that with an ClOrdID, some people might regard Tag 55 as unnecessary.
Another example might be ClOrdID and chaining. Many trading venues have some form of Order ID which exists for the life of the order. Cancel/Replace’s always act on the Order ID. Some even allow you to specify it. Where FIX has been added, it is likely to be a translation to the underlying Order ID.
It could be useful to permit non-chaining, life time Order IDs. This would simplify order processing (and establishing what happened to an order). It would also, then, be necessary to make OrigClOrdID (Tag 41) optional in the Order Cancel/Replace Request message.
I believe that in both of these examples, the flexibility and business integrity of FIX are retained, but the message building rules have been amended to ensure that information which parties do not wish to exchange does not have to be the messages. Equally, I would expect that participants wanting to retain order chaining, can and will get a more efficiently encoded message than with Tag=Value encoding - albeit, not as concise as someone who has moved to life-time Order IDs.
As in the Scope document, I suggest that there is a separate stream to review messages and message rules (primarily around required fields, but could include a view on order state etc).
Regards, Mark
Hi Georges,
I agree with you regarding preserving the FIX specification because there is a lot of business value into it. One thing I would like to preserve in any HFT protocol design is the flexiblity FIX has to offer. For example, OUCH protocol of NASDAQ is faster but not flexible - the only thing that can be done using OUCH is trade on NASDAQ. But FIX can be used to trade across most geographies of the world on most asset classes using many diverse trading styles. FIX TCP Sessions can transport non FIX data reliably. FIXProtocol is much more than just a trading protocol.
This calls into question our group’s choices of flexibility versus performance. My present thought is that when FIX started as Tag=Value^ encoded bytes over TCP/IP, it was flexibility which led to its widespread adoption. Now any changes to FIX which make it faster by reducing flexibility would not be well received by the industry.
For example, if there is a new version of FIX which is faster but cannot accept Tag 58 Text messages longer than 63 bytes, many end users would say “there is no need to upgrade, we want the full tag 58 Text without truncation because our secret Algo parameters are coded in tag 58 text value” or “The text messages are very informative, we want it fully transmitted”.
With semantics and business workflows unchanged, if we are able to offer a new HFT encoding to make FIX faster with no loss of flexibility, it could receive wider industry adoption.
Regards,
K. MaheshHi Vitali,
My main concern was to preserve the FIX specifications. Because there is a lot of business value into it. Redesigning it would be a waist of time in my perspective. But maybe I sticked too much with the tag=value style.
Prepared messages (like templates in FAST) are a way to get away from this FIX semantic. Then you have two choses to use them: fixed size and variable size.
Fixed size format will get the best decoding performance, I agree with you. The only problem is that fixed size don’t give great ‘compression’. For exemple, for tag “quantity”, since you don’t know if it’s 100 or 100 000 you have to arbitrary encode it on 4bytes to cover any possible values.
With non-fixed size, “quantity” of 100 would be encoded in 1 unique byte (for exemple)
I wanted something that was a balance between message size and encoding/decoding speed. I’ve gone this route because I believe that recent processors are super fast when we play at the register level.
For me, message size is the bottleneck because many different layers in the network and in the system are slower if message is bigger. But it’s only a guess I haven’t compare both approach empirically.
Regards
GeorgesGeorges,
This is great analysis. There are many useful ideas here on how to compress FIX messages and reduce processing time. What is not quite clear to me is which parts of FIX you are suggesting to keep and why. It appears that you aim to achieve performance gains by addressing message encoding while preserving FIX semantics in full, both session level and application level. I concede that simply switching to binary message format is perhaps the best way to improve performance. However, given that FIX engines will need to be rewritten to process new message formats anyway, it is relevant to ask whether more radical changes to FIX message structure and semantics is worth considering to attain even better performance.
For example, instead of spending a full byte per tag to encode its format, one can define small fixed size messages with fields in predefined formats for each message type. Various fixed extensions for most common options can also be considered. This approach reduces processing time not only by reducing message size but also by giving the decoder faster access to message fields by virtue of them being located at well-known rather than arbitrary offsets. Mandating field order (for most common cases) has an added benefit of reducing decoder complexity.
I do not insist that such approach is superior, although it does look attractive. My point is that you start with specific assumptions about which parts of FIX semantics you are keeping. I suggest re-examining these assumptions.
Thanks,
Vitali
Taking a somewhat different perspective, I would argue that we need to increase the flexibility as compared to FIX. This is possible by allowing increased configurability, re-using the good parts of FIX and relaxing some of the current limitations.
To re-iterate some points made in earlier discussions; The current syntax (tag=value encoding) is inflexible in that it doesn’t offer more specialized encodings of different application types (string, integer, decimal, …). A number of fields are superflous in some use cases and would be avoided if they were optional. The FIX recovery mechanism incurs a large overhead in some situations.
By making the HFT protocol more configurable, we will be able to express any FIX message exchange efficiently by allowing bi-lateral agreement of what features, fields, value ranges etc that are relevant for the current use case.
One user’s feature is another user’s anomaly …
Best,
Rolf
Hi Georges,
I agree with you regarding preserving the FIX specification because there is a lot of business value into it. One thing I would like to preserve in any HFT protocol design is the flexiblity FIX has to offer. For example, OUCH protocol of NASDAQ is faster but not flexible - the only thing that can be done using OUCH is trade on NASDAQ. But FIX can be used to trade across most geographies of the world on most asset classes using many diverse trading styles. FIX TCP Sessions can transport non FIX data reliably. FIXProtocol is much more than just a trading protocol.
This calls into question our group’s choices of flexibility versus performance. My present thought is that when FIX started as Tag=Value^ encoded bytes over TCP/IP, it was flexibility which led to its widespread adoption. Now any changes to FIX which make it faster by reducing flexibility would not be well received by the industry.
For example, if there is a new version of FIX which is faster but cannot accept Tag 58 Text messages longer than 63 bytes, many end users would say “there is no need to upgrade, we want the full tag 58 Text without truncation because our secret Algo parameters are coded in tag 58 text value” or “The text messages are very informative, we want it fully transmitted”.
With semantics and business workflows unchanged, if we are able to offer a new HFT encoding to make FIX faster with no loss of flexibility, it could receive wider industry adoption.
Regards,
K. Mahesh
finally somebody stated the obvios - the message length and check sum are arguably the worst/completely unnecessary “features” of Fix protocol for financial communications. First is just plain stupid - reading from a socket and parsing it byte by byte to figure out a length , while excluding the end of the message - only a business type person could come up with something moronic like that. Second is completely unnecessary for communications going over private networks over TCP(!!!) protocol.
Alex
Hi All,
As discussed during last meeting.
Please find hereafter some ideas for the working group.ULLINK - Analysis and proposal for FIX HFT - December 2010 | PPT
Feel free to comment on this forum thread.
Kind regards
Merry Christmas
Georges
Alex,
I totally agree with your technical point, but let’s avoid personal judgement regarding current proposals or about decisions made long ago under different circumstances.
Consider that Ethernet keeps length and check values, and no one calls its designers ‘moronic’. Agreed that we don’t need to duplicate message integrity guaranteed by an underlying transport.
Let’s think of this effort in terms of continuous improvement.
Don
finally somebody stated the obvios - the message length and check sum are arguably the worst/completely unnecessary “features” of Fix protocol for financial communications. First is just plain stupid - reading from a socket and parsing it byte by byte to figure out a length , while excluding the end of the message - only a business type person could come up with something moronic like that. Second is completely unnecessary for communications going over private networks over TCP(!!!) protocol.
Alex
Hi All,
As discussed during last meeting.
Please find hereafter some ideas for the working group.ULLINK - Analysis and proposal for FIX HFT - December 2010 | PPT
Feel free to comment on this forum thread.
Kind regards
Merry Christmas
Georges
Also it is worth remembering that in the dark ages of FIX (pre mass Internet circa '93) TCP/IP was not a given. There was work with X.25 if I’m not mistaken - Here’s looking at you, BLAM!
Any bit of software over a certain size and age is going to contain design decisions that look strange in light of current technology that were credible at the time. I would add the FIX encryption facilities to that list.
TCP Socket based communications allows only to send packets of data between applications. When a Java application opens a socket and starts reading the input stream, how can the application know how much to read for the current FIX message ? If both BodyLength and Checksum are removed, then how to breakup a stream of bytes into a series of messages? In message oriented middleware, the middleware delivers seperate messages to the application and hence a message BodyLength serves no useful purpose.
My guess is that BodyLength would be required on all FIX messaging over stream oriented network communication protocols which are not message based. The SocketReader reads the BodyLength field and then reads BodyLength bytes to complete the current message.
As per present FIX spec, a FIX Engine would also check
- The last field at the end of BodyLength bytes is CheckSum field
- The value of the CheckSum field in the message matches the values of checksum as calculated from the bytes of the message.
If either of the above two checks fail, the received bytes are declared garbled and discarded.
Since FIX has weak CheckSum, I agree CheckSum might have lost its utility and hence deprecated.
Alex,
I totally agree with your technical point, but let’s avoid personal judgement regarding current proposals or about decisions made long ago under different circumstances.
Consider that Ethernet keeps length and check values, and no one calls its designers ‘moronic’. Agreed that we don’t need to duplicate message integrity guaranteed by an underlying transport.
Let’s think of this effort in terms of continuous improvement.
Don
finally somebody stated the obvios - the message length and check sum are arguably the worst/completely unnecessary “features” of Fix protocol for financial communications. First is just plain stupid - reading from a socket and parsing it byte by byte to figure out a length , while excluding the end of the message - only a business type person could come up with something moronic like that. Second is completely unnecessary for communications going over private networks over TCP(!!!) protocol.
Alex
Hi All,
As discussed during last meeting.
Please find hereafter some ideas for the working group.ULLINK - Analysis and proposal for FIX HFT - December 2010 | PPT
Feel free to comment on this forum thread.
Kind regards
Merry Christmas
Georges