Imported from previous forum
[ original email was from John Greenan - john.greenan@alignment-systems.com ]
Does anyone have any views on the character set that should be supported by a regular FIX 4.2 session?
The 4.2 spec states that ASCII is the character set used - this indicates character codes 0-127.
I’ve seen a case where an order with a French character was sent from an OMS to an order routing hub and hence to a broker. The OMS did not prevent this and neither did the order routing hub. The broker rejected the order as it viewed the message as being non-FIX compliant.
The broker stated that the specific reason for rejection is that it would be impossible to calculate a FIX 4.2 checksum on the FIX message since one of the characters was outside of 0-127 - for example e accent grave, e accent aigu, e accent circonflexe.
It is of course possible to calculate a checksum but your checksum calculation is then not FIX 4.2 compliant.
What do folks think is the right behaviour here?
1 Like
When we do training we highlight and make sure that firms understand this is a 256 checksum and that the summation must include the full 8 bits of data in the calculation. However, the actual text of the checksum calculation appears under-specified and does not explicitly call for full 8 bit of each byte summation.
(refer to the appendix in Volume 2 of FIX.4.4 for the text or the FIXT.1.1 Errata specification for the checksum calculation description).
This point in requiring all 8 bits of each byte within the checksum calculation cannot be argued, because the FIX ‘data’ datatype (yes an absurd datatype name to be sure) is designed to support full 8 bit data and must be included in the checksum calculation, which implies that the full 8 bits of each byte must be included in the checksum calculation.
Could someone argue successfully that the String type and char type for that matter only contain 7 bit ASCII data? I suppose they could based upon the lack of documentation. But the practice clearly has been for much longer than a decade (more likely approaching two decades) to support extended ASCII in implementations. Clearly this point is under-specified in the specification.
The facts are - you must include all 8 bits of each byte in the checksum. The only character forbidden in the value part of the tag=value syntax for datatypes other than ‘data’ (great name huh?) is 0x01 (SOH), therefore there is no technical reason that extended ASCII characters should be excluded from FIX fields that support character data.
I have captured this in Jira SPEC-537 - asking if we should revise the checksum description to explicitly state that all 8 bits of each byte need to be included in the checksum calculation.
This is not without controversy and we should have other respondents, because one could make an argument that any extended ASCII characters should be placed into the appropriate data/Length field combinations. However, this narrow view runs up against well established market practices and would unnecessarily break many implementations.
It does get a bit more interesting when encoding FIX over FAST however, in that the FAST ASCII String type only permits 7 bit ASCII data, so the UNICODE (poorly named) String datatype must be used to represent 8 bits. However, this is an entirely different encoding schema. However, if one were to want to enforce consistency across the entire suite of encodings, FIXML, FIX, and FIX over FAST, one might easily conclude that the only way to fully mechanize a FIX over FAST encoding would be to enforce a 7-bit ASCII limitation on the String and char datatypes of the FIX layer. However, this again may be an over restriction, in that there is a general guideline on mapping FIX over FAST, but there is flexibility in which FAST datatypes are appropriate for each FIX datatype. In addition the FIX over FAST mapping document is not a FIX standard.
Does anyone have any views on the character set that should be supported by a regular FIX 4.2 session?
The 4.2 spec states that ASCII is the character set used - this indicates character codes 0-127.
I’ve seen a case where an order with a French character was sent from an OMS to an order routing hub and hence to a broker. The OMS did not prevent this and neither did the order routing hub. The broker rejected the order as it viewed the message as being non-FIX compliant.
The broker stated that the specific reason for rejection is that it would be impossible to calculate a FIX 4.2 checksum on the FIX message since one of the characters was outside of 0-127 - for example e accent grave, e accent aigu, e accent circonflexe.
It is of course possible to calculate a checksum but your checksum calculation is then not FIX 4.2 compliant.
What do folks think is the right behaviour here?
[ original email was from Andrei Goldchleger - agoldchleger@bvmf.com.br ]
Some of our customers have experienced problems with extended ASCII. As the documentation states ASCII I always assumed that extended values were not allowed.