Imported from previous forum
[ original email was from Steven Grossman - steven.grossman@transacttools.net ]
Quick follow-up from this morning’s meeting.
There was a desire to compare FAST compression of XML data with other compression schemes.
For preliminary results, I recommend using the templates that David Rosenborg introduced to support the SCP 1.1 template definition message. This should be fairly straightforward as it does not require any XML Schema to FAST data type mapping – everything is treated as a unicode string.
Steven
The following is taken from SCP 1.1.06:
That is one possible approach. The virtues are that it is simple and straight forward and provides a way of transferring any wellformed XML document over FAST. However it does not provide much in the area of compression.
I’ve been playing with an idea that keeps the simplicity of this model but that opens up for use of the field operators of FAST in the case where you need more compression. Another advantage of this approach is that it can be implemented in a streaming manner (the SCP 1.1 structure requires you to encode the number of children before the children are encoded, so you have to count them first and thereby breaking the streaming possibility).
The idea is to view the XML document as a stream of events, StartElement, EndElement, Data, Comment etc. Each event type gets a corresponding message type in the FAST mapping. In addition to the basic event types that naturally use Unicode strings, a corresponding set of typed messages are added for Data and Attribute events: UInt32Data, UInt32Attribute, …, DecimalData, DecimalAttribute. Additional types could be introcuded to add even more specialized compression such as UtcMillitimeData and BooleanAttribute.
The mapping should provide a set of predefined templates with hard-wired template identifiers. These templates could be used by a simple encoder/decoder.
A more involved encoder would use level 3 SCP 1.1 to define new templates that add field operators to data and attributes.
A predefined template with no optimization could look something like this:
An optimizing encoder (e.g. one primed with data from an XML Schema) could issue a template definition in SCP with the following meaning:
and then use this template whenever a price was to be encoded.
The underlying concept of this idea is to tie as much information as possible to the template identifier. The tid indicates the event type, the data type, a possible operator and in some cases even constant data:
Another advantage with this idea is that it is scalable in terms of complexity and performance (compression) and that it builds upon already defined mechanisms in core FAST 1.1 and SCP 1.1. Scalable because a simple encoder could stick with the predefined templates (and not even use the typed ones if it so see fit). A more involved encoder would leverage the capabilities of SCP 1.1 to optimize the stream further.
/David
Quick follow-up from this morning’s meeting.
There was a desire to compare FAST compression of XML data with other
compression schemes.For preliminary results, I recommend using the templates that David
Rosenborg introduced to support the SCP 1.1 template definition
message. This should be fairly straightforward as it does not require
any XML Schema to FAST data type mapping – everything is treated as a
unicode string.Steven
The following is taken from SCP 1.1.06: