Imported from previous forum
[ original email was from Tim Miller - timothy.miller@bankofamerica.com ]
If anyone is interested in speed, I have been carrying out some tests to assess the performance of FIXML in a Real-Time environment. The results have been quite interesting. I ran 100 FIXML messages through a C++ parser using DOM/MSXML. There is a significant impact on performance depending on the size of the DTD.
When using the FIXML 4.2 Version 1.0 DTD "as is" the processing time took about 60 seconds for 100 messages.
When stripping down the DTD to one message with its associated fields, the processing took about 7 seconds for 100 messages.
As is expected, there is a correlation between size of DTD and performance. However, the figures are quite dramatic and the first example that is approaching 1 second per message is quite a lengthy period for a Real-Time environments.
This may suggest that a modular approach to the DTD would bring performance benefits? I tried importing the FIX header and a FIX general entities as separate DTDs and this did not slow down the faster performance figures.
Alternatively, it may be best to keep moving forwards with an all-in-one DTD and to simply automate the process of transforming it to whichever smaller parts are relevant to a project?
I have been impressed with the Fiximate 4.2 utiity hosted by FIXprotocol.org. Perhaps someone could extend its functionality to produce a subset DTD for FIXML based on selected message types?
I believe everyone who uses FIX is interested in performance, and the figures stated look far from satisfactory. However, I question the suggestion of using a cut-down or modularised DTD.
The figures indicate that most of the processing time was spent parsing the DTD, and that it was re-parsed for each FIX message. This may be the way that most XML parsers currently work, but there is no rule that it must be done this way. It is easy to imagine a customised FIXML engine
incorporating a parser that re-uses an internalised DTD for each message.
These figures are also based on the assumption that each FIXML message should be validated. Personally I believe that validating each message is a good idea because it helps to assure the quality of messages coming from third-party organizations. However, this is not the universal case. In some other areas of e-commerce XML messages are only validated during the development stages and assumed to be valid in production systems.
Regards
Rob Lugt
ElCel Technology
> If anyone is interested in speed, I have been carrying out some tests to assess the performance of FIXML in a Real-Time environment. The results have been quite interesting. I ran 100 FIXML messages through a C++ parser using DOM/MSXML. There is a significant impact on performance depending on the size of the DTD.
>
> When using the FIXML 4.2 Version 1.0 DTD "as is" the processing time took about 60 seconds for 100 messages.
>
> When stripping down the DTD to one message with its associated fields, the processing took about 7 seconds for 100 messages.
>
> As is expected, there is a correlation between size of DTD and performance. However, the figures are quite dramatic and the first example that is approaching 1 second per message is quite a lengthy period for a Real-Time environments.
>
> This may suggest that a modular approach to the DTD would bring performance benefits? I tried importing the FIX header and a FIX general entities as separate DTDs and this did not slow down the faster performance figures.
>
> Alternatively, it may be best to keep moving forwards with an all-in-one DTD and to simply automate the process of transforming it to whichever smaller parts are relevant to a project?
>
> I have been impressed with the Fiximate 4.2 utiity hosted by FIXprotocol.org. Perhaps someone could extend its functionality to produce a subset DTD for FIXML based on selected message types?
>
>
Hi Tim,
your results are very interesting. The reason
for the impact to performance might be a worse
alogrithm to find element tags.
To boost the performance of your XML application
I would suggest tree things:
-
having two modes on your application - validating and
non-validating so that you can turn-off the DTD check
if possible. -
using the SAX algorithm, because DOM is very slow in
contrast to SAX. SAX is a lightweight event-driven appoach
compared to DOM which builds a complete and expensive tree
representation of the document. -
there are approaches generating language mappings for
DTD/XML-Schema. Such a mapping is a class representation
of the XML element types and give you an API to parse and
generate XML documents (based on the DTD/XML-Schema).
Mappings often use the prefered SAX algorithm, have a
efficient build-in validation and use pooling for memory
management.
Cheers,
Stefan
Stefan Neurohr
Shinka Technologies AG
stefan.neurohr@shinkatech.com
> If anyone is interested in speed, I have been carrying out some tests to assess the performance of FIXML in a Real-Time environment. The results have been quite interesting. I ran 100 FIXML messages through a C++ parser using DOM/MSXML. There is a significant impact on performance depending on the size of the DTD.
>
> When using the FIXML 4.2 Version 1.0 DTD "as is" the processing time took about 60 seconds for 100 messages.
>
> When stripping down the DTD to one message with its associated fields, the processing took about 7 seconds for 100 messages.
>
> As is expected, there is a correlation between size of DTD and performance. However, the figures are quite dramatic and the first example that is approaching 1 second per message is quite a lengthy period for a Real-Time environments.
>
> This may suggest that a modular approach to the DTD would bring performance benefits? I tried importing the FIX header and a FIX general entities as separate DTDs and this did not slow down the faster performance figures.
>
> Alternatively, it may be best to keep moving forwards with an all-in-one DTD and to simply automate the process of transforming it to whichever smaller parts are relevant to a project?
>
> I have been impressed with the Fiximate 4.2 utiity hosted by FIXprotocol.org. Perhaps someone could extend its functionality to produce a subset DTD for FIXML based on selected message types?
>
>
[ original email was from John Goeller - john.goeller@csfb.com ]
First off, thanks for providing the feedback. One of the challenges of FIXML is that we don’t know exactly how people are using it. We depend on the community to provide the implementation feedback and guidance.
Regarding DTD modularity, you may remember we took a modular approach in previous versions of the DTDs. Our approach was to have a main DTD (fixmlmain.dtd) with Entity references to the other DTDs (Order.DTD, IOI.DTD, etc). These references act like #include(s) for a C preprocessor. The end result of this approach from parser point of view was the same as having one big DTD. With 4.2, I abandoned the modular approach because the tool I was using (converts DTDs to XML Schemas) didn’t handle Entity statements well and it also became a challenge to figure out where the particular fields belong in the different DTDs.
There seems to be two schools of thought regarding DTD modularity. The combined DTD seems to be the more common approach although implementations like GSTPA have adopted the one-to-one message to DTD approach.
To a certain extent it depends on how you’re using the DTDs. I think the previous posts give some good implementation advice.
In addition, on the validation front, Extensibility has created a SAX validation implementation that might be worth a look.
http://216.122.205.184/solutions/xml_validate/
> If anyone is interested in speed, I have been carrying out some tests to assess the performance of FIXML in a Real-Time environment. The results have been quite interesting. I ran 100 FIXML messages through a C++ parser using DOM/MSXML. There is a significant impact on performance depending on the size of the DTD.
>
> When using the FIXML 4.2 Version 1.0 DTD "as is" the processing time took about 60 seconds for 100 messages.
>
> When stripping down the DTD to one message with its associated fields, the processing took about 7 seconds for 100 messages.
>
> As is expected, there is a correlation between size of DTD and performance. However, the figures are quite dramatic and the first example that is approaching 1 second per message is quite a lengthy period for a Real-Time environments.
>
> This may suggest that a modular approach to the DTD would bring performance benefits? I tried importing the FIX header and a FIX general entities as separate DTDs and this did not slow down the faster performance figures.
>
> Alternatively, it may be best to keep moving forwards with an all-in-one DTD and to simply automate the process of transforming it to whichever smaller parts are relevant to a project?
>
> I have been impressed with the Fiximate 4.2 utiity hosted by FIXprotocol.org. Perhaps someone could extend its functionality to produce a subset DTD for FIXML based on selected message types?
>
>
People have touched on this matter but have not been explicit…
The XML parser that we use (IBM’s XML4C, based on Apache’s Xerces) has a facility to allow you to cache the parsed DTD. I would imagine that other parsers provide this functionality too.
Large performance gains can be achieved by turning on the parser’s DTD caching.
For example:
Parsing 100 FixML Execution Report messages (DTD = fixml4.2v1.1.dtd)
Without caching:
Total time = 171 seconds
With caching:
Total time = 20 seconds
Interestingly, it takes two parses before the caching is fully “warmed up”.
e.g.
1.3s - Time to parse 1st message.
0.26s - Time to parse 2nd message.
0.07s - Average time to parse each of the following messages.
Regards,
Ian