Imported from previous forum
Hi,
We are looking to use FIXML (v5) as the message format for our front to back office systems (ie internal). The interface is looking to send insert/update/delete for trades.
I was planning to largely use the Execution Reports for this, but have now noticed that there are Trade Capture Reports. At first glance the TCRs seem like what we should be using.
Our interface is for Repo’s (bonds and equities).
Does that sound like the most appropriate message type?
Thanks in advance,
Chris
Chris,
the answer is “it depends”. ER was the only available message initially in FIX, TCR came later. The TCR has more fields to convey clearing and settlement related information and is able to represent both sides of a trade. The ER is focussed on the pure execution of an order or quote. Exchanges might conduct trade enrichment to add back office information and could be sending an ER initially, followed by one or more TCRs. Clearinghouses might only be sending TCRs but no ERs. The first question is thus what kind of event or message triggers your internal message to your back office system. The second question is then the type of content that needs to be conveyed and how that is covered by the ER vs TCR.
Regards,
Hanno.
Hi,
We are looking to use FIXML (v5) as the message format for our front to back office systems (ie internal). The interface is looking to send insert/update/delete for trades.
I was planning to largely use the Execution Reports for this, but have now noticed that there are Trade Capture Reports. At first glance the TCRs seem like what we should be using.
Our interface is for Repo’s (bonds and equities).
Does that sound like the most appropriate message type?
Thanks in advance,
Chris
TCR offers a nicer API for dealing with back office trades. I would use it over plain execution reports if you starting new.
Do not worry about how the TCRs are getting triggered in your system, as you may add counter parties later. Focus on getting your API right and then just write adapters.
I would avoid FIXML at all costs. It is a very poor protocol. Plain FIX or plain XML are better solutions.
Hi All,
Thanks for the advice.
I would avoid FIXML at all costs. It is a very poor protocol. Plain FIX or plain XML are better solutions.
I am really interested in this comment - why do you say that?
We are at an early stage, so could switch tack (FpML or totally custom XML)…
Thanks,
Chris
Chris,
I would not avoid FIXML for your postrade procesing since it is commonly used in the middle and back office - you will fine many trade feeds from exchanges and brokers use fixml, so in many cases you cant avoid it.
I think it would make most sense to use the TCR.
Regards
John
Hi All,
Thanks for the advice.
I would avoid FIXML at all costs. It is a very poor protocol. Plain FIX or plain XML are better solutions.
I am really interested in this comment - why do you say that?
We are at an early stage, so could switch tack (FpML or totally custom XML)…
Thanks,
Chris
Hi All,
Thanks again - for this internal interface, we are using an inhouse custom transport mechanism (MQ/XML-based) and that will be the wrapper around the FIXML.
Better get my head back into the TCR manual, find some examples and how to extend it (need to send internal book type stuff).
Regards,
Chris
I would not avoid FIXML for your postrade procesing since it is commonly used in the middle and back office - you will fine many trade feeds from exchanges and brokers use fixml, so in many cases you cant avoid it.
I think it would make most sense to use the TCR.
Regards
John
I would avoid FIXML at all costs. It is a very poor protocol. Plain FIX or plain XML are better solutions.
FPL is obviously interested in the background on that statement, especially comparing FIXML to plain XML. FIXML is heavily using in the clearing space and the FIA Post Trade Working Group has decided a few years ago to standardize clearing workflows based on FIXML.
Is FIXML “broken beyond repair” or can FIXML be “fixed”?
… Of course, you need a transport for moving FIXML around; some sort of message queueing approach seems common, for example, IBM Websphere MQ.
I would add AMQP as a transport here. Eurex has had a good experience with AMQP, even if it is not the 1.0 version yet (published but not formally completed yet).
Hi,
[Start Jim Northey’s Quote from http://www.fixprotocol.org/discuss/read/218c5d68]
I think the best way to communicate FIXML over FIX is to use FIXT.1.1 and use the XML_non_FIX (unfortunately named message) MsgType(tag 35)=‘n’.
[End Quote]
Regards,
Mahesh
Agreed - AMQP is just the job, although I would counsel to pick your implementation of AMQP carefully. (Feel like we’re going slightly OT now…)
… Of course, you need a transport for moving FIXML around; some sort of message queueing approach seems common, for example, IBM Websphere MQ.
I would add AMQP as a transport here. Eurex has had a good experience with AMQP, even if it is not the 1.0 version yet (published but not formally completed yet).
I second Jim’s quote here. If you are needing to communicate point to point XML payload, including scenarios that are heterogeneous in nature, FIXT.1.1 is the most practical, most well proven approach that exists.
The number of firms shipping XML payload around using FIX is quite astounding.
I think AMQP is almost getting old enough, the original idea was leaked in 2001 in a Security News story, to be considered. We have two separate streams and neither are the preverbial “1.0” gold standard, but AMQP is important in both product terms and technology terms. The potential for AMQP goes beyond mere point to point ordered bi-directional messaging streams.
I would stop short of recommending AMQP over FIXT.1.1 without knowing the context in which the question was asked. If I were a trading firm and I was using FIX and I just needed to send some FpML trade confirms to my prime broker - I would probably consider putting up a FIX.4.4 or FIXT.1.1 session and do it that way. If I were a major global derivatives clearing house looking to release the next generation clearing system, faced with no uptake of the FIX Session Layer in derivatives clearing, and the preponderance of a very expense and costly to administrate proprietary message queuing system, I would likely be embracing AMQP.
It all depends on the details… and don’t count the FIX session out - it is some solid, reliable, and cheap plumbing.
Hi,
[Start Jim Northey’s Quote from http://www.fixprotocol.org/discuss/read/218c5d68]
I think the best way to communicate FIXML over FIX is to use FIXT.1.1 and use the XML_non_FIX (unfortunately named message) MsgType(tag 35)=‘n’.
[End Quote]
Regards,
MaheshAgreed - AMQP is just the job, although I would counsel to pick your implementation of AMQP carefully. (Feel like we’re going slightly OT now…)
… Of course, you need a transport for moving FIXML around; some sort of message queueing approach seems common, for example, IBM Websphere MQ.
I would add AMQP as a transport here. Eurex has had a good experience with AMQP, even if it is not the 1.0 version yet (published but not formally completed yet).
Kenneth,
I would like to hear more specifics regarding “It is a very poor protocol”? This is a difficult statement to make outside of some specific usage context. We created FIXML to be a very usable application of XML to represent FIX. And the vast majority of users have taken it up quite happily, especially compared with other XML based message syntaxes in use in the financial services industry.
I can understand an argument that says XML, and as a result FIXML, is not a good choice for higher volume trading due to message size.
I could also understand an argument that said “the FIXML Schema is a bit overcomplicated”, but a general statement does “It is a very poor protocol” does not carry with it any context to either support or refute the argument.
Why do I ask? We have identified some issues with FIXML that can be improved and are likely, time permitting, to issue some enhancements to FIXML in the near future (12 to 18 months hopefully - no commitment made yet by GTC - just something I am pushing for right now). If there are specific actionable objections or issues we would definitely consider addressing them when we perform a round of FIXML maintenance.
Best,
Jim N
[ original email was from Andrei Goldchleger - agoldchleger@bmf.com.br ]
I have the impression that “It is a very poor protocol” might stand for “why define a XML rendering when we have a perfectly useable, more compact representation in place”. At least this is what I thought when I entered the FIX jungle six years ago 
But I know that you had your reasons (such as the whole industry pushing XML as the cure for all IT ailments).
Sorry for derailing the thread. FIXML has had success in middle and back office. It’s better that we have a sort-of standard for distribution between firms versus everyone rolling their own.
I just don’t see FIXML as an improvement over FIX. The promise of XML is human/machine readability, but FIXML fails at the human part. The field names are abbreviated in strange, sometimes indiscernible ways for those experienced with FIX, and the values are the encoded ones from FIX. I would expect SIDE=BUY for XML but instead it’s SIDE=1, rendering the document pretty pointless. And how was ‘SIDE’ chosen for instead of ‘SID’ or ‘SD’? You never know how these tag abbreviations are going to work. FIXML is an awkward compromise between XML and FIX, and it ends up with the worst of both worlds.
In more general terms, XML’s arbitrariness breaks ‘code=data’ meme that computer science has known for 50 years. There was an overreaction to XMLize everything in IT, but the (more quickly moving) web has completely abandoned XML for good reason - it’s overly verbose and lends itself to arbitrary structures.
A repeating group in FIXML is no more clear than the FIX itself, so why not just use FIX? This is just not good XML:
We saved two characters in Party and three in Role (which we give back via closing tags), I have no idea what Sub Type = “26” is without looking it up. It just seems like bizarre compromises are being made everywhere. If you are worried about message size, you would never use XML. If you want human/machine documents, FIXML won’t help you either. If you want object mappings, now you are stuck with a a property ‘R’ or the expanded ‘Role’ where you (again) have to look up the mapping (R=Role for Pty==Party group).
I understand there were pressures from all over that forged FIXML. I was just saying that I would not not use it as an internal API for a greenfield messaging system. Keep up the good work fix protocol 
Kenny
Not a thread derailment in my opinion and your comments below accurately reflect the state of FIXML. I will say this, we prioritized message size reduction over readability. You presented your perspective here very well and reflects legitimate concerns with our design choices.
We do have inconsistencies in abbreviations, we have addressed this by adding abbreviations to the FIX repository and we review the abbreviations as part of extension pack creation. But we are not planning and going back and causing breakage that would result from a cleanup.
I will say that my choice of “R” for “Role” and the my choice to optimize the InstrumentLegGrp and UnderlyingInstrumentGrp were definitely over the top and have been a real pain in the neck. But it is what it is now. We have found a way via Ryan Pierce’s proposal for Inlined Component References to resolve the Optimized component problem without breaking the specification, so some indirect redemption has been achieved, “inlined redemption” 
Nearly ten years on, I am struggling with why we all thought that XML was a good wire format, when other standards, such as ASN.1 already existed that provided access to metadata. I revised my opinion about 5 or 6 years ago that XML should not be used as a wire protocol. Why do we have JSON - largely to overcome XML verbosity. There are new binary XML wire formats being defined now, such as by the W3C, whether any of these will be usable remains to be seen.
Sorry for derailing the thread. FIXML has had success in middle and back office. It’s better that we have a sort-of standard for distribution between firms versus everyone rolling their own.
I just don’t see FIXML as an improvement over FIX. The promise of XML is human/machine readability, but FIXML fails at the human part. The field names are abbreviated in strange, sometimes indiscernible ways for those experienced with FIX, and the values are the encoded ones from FIX. I would expect SIDE=BUY for XML but instead it’s SIDE=1, rendering the document pretty pointless. And how was ‘SIDE’ chosen for instead of ‘SID’ or ‘SD’? You never know how these tag abbreviations are going to work. FIXML is an awkward compromise between XML and FIX, and it ends up with the worst of both worlds.
In more general terms, XML’s arbitrariness breaks ‘code=data’ meme that computer science has known for 50 years. There was an overreaction to XMLize everything in IT, but the (more quickly moving) web has completely abandoned XML for good reason - it’s overly verbose and lends itself to arbitrary structures.
A repeating group in FIXML is no more clear than the FIX itself, so why not just use FIX? This is just not good XML:
We saved two characters in Party and three in Role (which we give back via closing tags), I have no idea what Sub Type = “26” is without looking it up. It just seems like bizarre compromises are being made everywhere. If you are worried about message size, you would never use XML. If you want human/machine documents, FIXML won’t help you either. If you want object mappings, now you are stuck with a a property ‘R’ or the expanded ‘Role’ where you (again) have to look up the mapping (R=Role for Pty==Party group).
I understand there were pressures from all over that forged FIXML. I was just saying that I would not not use it as an internal API for a greenfield messaging system. Keep up the good work fix protocol
Kenny
Steve, Kenny
Your argument taken from a readability perspective is the fundamental problem with XML as a wire format. Transmission protocols and messaging applications are for computer to computer communication. Computers have no problem interpreting and understanding these messages.
Within the context of using XML as a wire protocol the readability argument falls flat. Are you really saying that:
This:
Is less readable and preferable to this from a readability perspective?
… 453=1^448=clearing_acct_2^452=24^802=1^523=1^803=26^ …
How many people who benefit daily from the hundreds of millions of FIX and FIXML messages they exchange need to see and read the raw messages?
When we used the industry standard element based approach for FIXML the then de facto industry standard trade message in TREX went from about 200+ bytes up to 9,000 bytes for a single trade. Via optimizing the format and converting to attributes and abbreviations we created a much more terse, simple, and smaller (about 400+ bytes) and arguably for those who have been using these messages in a production environment since 2003 have found them more readable than more complicated and verbose XML element based syntax’s. In fact, due to the inefficiency of XML as a wire format, other protocols, such as ISO 20022 have followed suit with using abbreviations.
FIXML is a practical compromise once the industry chose XML as a wire format. It is easier to adopt and implement efficiently than other XML messaging interfaces, so I don’t buy the readability over efficiency argument within this context.
If you want to take a step back as did the thought leaders behind ISO 20022 Version 1.5 and adopt something like ASN.1 as a metadata layer from which you can then choose multiple wire formats (including naively XML) you have that option, I am with you.
If you want to bemoan that fact that we did not create an XML encoding of FIX that would completely fail the non-functional requirements of the FIX Community, yet be a verbose, highly readable thing of beauty, then I am afraid I am not with you. FIX is about practicality. Most of the FIX Protocol is simple to the point of being crude. And in that lies both usability and a certain valuable design aesthetic as proven by adoption.
Our industry must move beyond XML as a wire format and also move beyond bespoke protocols and look at horizontal information technology standards that meet our functional and non-functional requirements.
Kenny - these are very fair criticisms - having worked with the first version of FIXML (which was much more verbose), it seems that in the move to the attribute-based style of 4.4 FIXML, we in FPL-Land erred rather too much on the side of improving efficiency rather than improving readability. Perhaps a missed opportunity…
You kidding - this was an incredible thread - we had an important discussion about the fundamental decisions that go into protocols. We added a new contributor who can articulate and defend their position - Kenneth. I recommended to Ed Mangles, Daniella, and Lisa T that this could be the basis for a future FIX Global Article - maybe in discussion or interview form. We did most of the work already in the thread. We were able to capture some of the important folks that contributed, we were able to really frame the argument on the use of XML in messaging.
Jim - didn’t mean to offend anyone by my remarks - I definitely think FIXML 4.4 was a huge improvement over FIXML 4.2. My point was that if the goal (as might conceivably be the case with an XML-based syntax) was human-readability, I’m not sure we made huge progress. As Hanno points out, it is a rare human that can hold all the FIX enumerated values in their head…
That said, the widespread adoption of FIXML is testimony to its usefulness and effectiveness to a class of problems where parsing FIX messages isn’t quite so natural. It clearly is a success in that respect, and it was not my intention to criticise anyone that was involved in the standards effort, or the output that resulted. Please accept my apologies if that was how it came across.
Steve.
Steve, Kenny
Your argument taken from a readability perspective is the fundamental problem with XML as a wire format. Transmission protocols and messaging applications are for computer to computer communication. Computers have no problem interpreting and understanding these messages.
Within the context of using XML as a wire protocol the readability argument falls flat. Are you really saying that:
This:
Is less readable and preferable to this from a readability perspective?
… 453=1^448=clearing_acct_2^452=24^802=1^523=1^803=26^ …
How many people who benefit daily from the hundreds of millions of FIX and FIXML messages they exchange need to see and read the raw messages?
When we used the industry standard element based approach for FIXML the then de facto industry standard trade message in TREX went from about 200+ bytes up to 9,000 bytes for a single trade. Via optimizing the format and converting to attributes and abbreviations we created a much more terse, simple, and smaller (about 400+ bytes) and arguably for those who have been using these messages in a production environment since 2003 have found them more readable than more complicated and verbose XML element based syntax’s. In fact, due to the inefficiency of XML as a wire format, other protocols, such as ISO 20022 have followed suit with using abbreviations.
FIXML is a practical compromise once the industry chose XML as a wire format. It is easier to adopt and implement efficiently than other XML messaging interfaces, so I don’t buy the readability over efficiency argument within this context.
If you want to take a step back as did the thought leaders behind ISO 20022 Version 1.5 and adopt something like ASN.1 as a metadata layer from which you can then choose multiple wire formats (including naively XML) you have that option, I am with you.
If you want to bemoan that fact that we did not create an XML encoding of FIX that would completely fail the non-functional requirements of the FIX Community, yet be a verbose, highly readable thing of beauty, then I am afraid I am not with you. FIX is about practicality. Most of the FIX Protocol is simple to the point of being crude. And in that lies both usability and a certain valuable design aesthetic as proven by adoption.
Our industry must move beyond XML as a wire format and also move beyond bespoke protocols and look at horizontal information technology standards that meet our functional and non-functional requirements.
Kenny - these are very fair criticisms - having worked with the first version of FIXML (which was much more verbose), it seems that in the move to the attribute-based style of 4.4 FIXML, we in FPL-Land erred rather too much on the side of improving efficiency rather than improving readability. Perhaps a missed opportunity…
Jim,
Now that you have mentioned a future FIX Global article, may I suggest that we consider the article as “FIX in End-to-End STP and the Future” starting with pre-trade, trade and post-trade phases of the trading life cycle covering front, middle to back office spaces seamless integration using FIX and its offsprings like FAST, FIXML etc. It could also look at how FIX is looking at more efficient information flow like HFT/HPIWG etc.
Regards,
K. Mahesh
You kidding - this was an incredible thread - we had an important discussion about the fundamental decisions that go into protocols. We added a new contributor who can articulate and defend their position - Kenneth. I recommended to Ed Mangles, Daniella, and Lisa T that this could be the basis for a future FIX Global Article - maybe in discussion or interview form. We did most of the work already in the thread. We were able to capture some of the important folks that contributed, we were able to really frame the argument on the use of XML in messaging.
I would like to add that FIXML was also driven by the availability of (free) standard XML tools that support development and testing. Regardless of syntax (tag=value, FIXML, FAST,…) our tools access a repository and produce a GUI or logfile output “SIDE=BUY”, i.e. we translate between the wire format and the human readable format. The translation effort depends on the wire format and the usage of 3rd party tools.
I also do not know most valid values by their number or char, i.e. I need this translation and am always annoyed when people post tag=value formats on the forum and I have to look up the values in FIXimate. I must admit that I am less inclined to answers posts that require this extra effort. “Side=1” is not much better than “54=1”. I used to admire people that were able to read hex dumps printed on endless paper after a mainframe crash but I think nowadays we should expect a tool layer between the raw format and the format for the human eye. The translation process needs to be automated to be efficient. Notepad is clearly not enough, Internet Explorer makes it a little better by using colors but is unable to change “Side=1” to “Side=Buy”. I am sure there ar FIX-specific XML tools out there (no, I am not asking people to solicit products here) that will do the job and can even be integrated into your application development environment.
Interesting to see how many people stood up to defend FIXML, albeit its deficiencies. That alone is a sign that it is usable as a standard which is also about overall effort for the industry and should not be compared to a perfect but proprietary solution.
Regards,
Hanno.
Sorry for derailing the thread. FIXML has had success in middle and back office. It’s better that we have a sort-of standard for distribution between firms versus everyone rolling their own.
I just don’t see FIXML as an improvement over FIX. The promise of XML is human/machine readability, but FIXML fails at the human part. The field names are abbreviated in strange, sometimes indiscernible ways for those experienced with FIX, and the values are the encoded ones from FIX. I would expect SIDE=BUY for XML but instead it’s SIDE=1, rendering the document pretty pointless. And how was ‘SIDE’ chosen for instead of ‘SID’ or ‘SD’? You never know how these tag abbreviations are going to work. FIXML is an awkward compromise between XML and FIX, and it ends up with the worst of both worlds.
In more general terms, XML’s arbitrariness breaks ‘code=data’ meme that computer science has known for 50 years. There was an overreaction to XMLize everything in IT, but the (more quickly moving) web has completely abandoned XML for good reason - it’s overly verbose and lends itself to arbitrary structures.
A repeating group in FIXML is no more clear than the FIX itself, so why not just use FIX? This is just not good XML:
We saved two characters in Party and three in Role (which we give back via closing tags), I have no idea what Sub Type = “26” is without looking it up. It just seems like bizarre compromises are being made everywhere. If you are worried about message size, you would never use XML. If you want human/machine documents, FIXML won’t help you either. If you want object mappings, now you are stuck with a a property ‘R’ or the expanded ‘Role’ where you (again) have to look up the mapping (R=Role for Pty==Party group).
I understand there were pressures from all over that forged FIXML. I was just saying that I would not not use it as an internal API for a greenfield messaging system. Keep up the good work fix protocol
Kenny
Andrei - I would say very much that we were responding during the first Internet tech bubble to the “everything will be in XML” hysteria and hyperbole. There must have been at least a dozen different XML based financial protocols announced at the same time. Of those XBRL, FpML, FIXML, ISO20022 have taken hold. The birth of FIXML, led by John Goeller (my co-chair of the America’s Region) in response to XML, but also in response to the Global Straight Through Processing Association (GSTPA) initiative. There was considerable momentum behind this initiative until the collapse of the first Internet bubble.
When the futures industry approached me for a recommendation, then acting as leader of the Futures and Options Working Group (having replaced Will Walter when we restarted that group) since there was no install base of FIX tag=value in derivatives clearing and the clearing messages tended to be very nested, we chose FIXML. The original version of FIXML followed design standards prevalent in the late 1990s and was based upon elements. This caused incredible message sizes and excessive “wordiness”. So working with CME, specifically Matt Simpson and Niranjana Sharma, we optimized the FIXML syntax as a proof of concept moving to attributes for fields and abbreviations for element and attribute naming. We were only looking for message size reduction, but we were pleased to find the message structures were simplified and actual readability improved.
We then worked with OCC (Kevin Kobets, Barry Galster (Teresa Simon did the much important job of managing yours truly and keeping me on task)) and CME (Matt Simpson, not sure if Todd Borro had joined yet he may have) to develop the FIXML Schema approach. The schema complexity was such because it was developed prior to the creation of the FIX Repository by Kevin Houstoun. I was looking for the FIXML Schema to contain metadata sufficient to represent FIX and to also provide a mechanism to be able to document implementation as a set of *impl.xsd files that could be exchanged by market participants. It worked well from a Schema perspective, but the xsd:redefine instruction is not supported by any (I think this is still true) language binding tools, such as JAXB, JIBX, etc. Then Kevin Houstoun donated the repository for use by FPL and also donated the first FIXML Schema generator. The combination of redefine not being supported and the ability to generate a schema from a repository, in this case the donated FIX Repository, rendered much of the FIXML Schema design unneeded.
At the time we also had all the clearing members consulted stating that they wanted to maintain MQ Series as the message transport and that if we left MQ Series in place, they would tolerate a change in the message format from the de facto US standard, TREX (largely invented and maintained by Dick Baker now of ICE among others), to FIXML.
When we moved from batch delivery of data to interactive messages of course there were characteristics of a reliable bi-directional protocol that are not supported without additional functionality using MQ Series.
So this is a long winded history lesson, if I had to do it all over again, I would have recommended FIX tag=value and the FIX session layer for listed derivatives clearing.
But, I can say without reservation, that FIXML is very usable, from a technical perspective even enjoyable should you have to use XML, and that it has an incredibly positive impact on the listed derivatives clearing industry and within financial services who have adopted for EA and SOA based infrastructure. I think FIXML is a valuable addition to the FIX standards suite. We are planning some enhancements to make it more usable as an integration technology for EA,MDA,SOA environments based upon the work done by Matthew Rawlings and Andy Key. NYSE Technology (John Lambe, Jaime Barra, Tim Hudson) working with Dean Kauffman identified some incompleteness around the area of error reporting that we also need to address. We identified some enhancements we would like to make in terms of namespace usage that we learned from FpML 5.0. But all of these are really minor enhancements that, with the exception of the namespace naming, will be fully backward compatible enhancements, and will not cause breakage.
[This should probably be over on the John Cameron’s donated FIXwiki as it is good to know how, to borrow a phrase from Gerald Weinberg, “things got that way.”]