Imported from previous forum
Hi all,
I’m considering to compress market data messages using zlib.
I found some discussions in this forum regarding zlib v.s. FAST. From performance point of view, FAST is a better choice. But FAST seems to be more complex in implementation.
I’d like to know if zlib is considered a valid implemetation for the market data compression? Is it used by anybody in production?
Thanks & Regards,
LiuAn
According to my personal experience, FAST is not necessarily better than zlib in terms of compression ratio. In fact those two are very close. You can even apply FAST/zlib if the time overhead is tolerable for application. However, FAST is more suitable for context-based streaming process because the granularity of which is message-by-message or record-by-record, instead of handling data in a batch process way.
Hi all,
I’m considering to compress market data messages using zlib.
I found some discussions in this forum regarding zlib v.s. FAST. From performance point of view, FAST is a better choice. But FAST seems to be more complex in implementation.
I’d like to know if zlib is considered a valid implemetation for the market data compression? Is it used by anybody in production?
Thanks & Regards,
LiuAn
you make a good point about applying FAST and ZLIB in sequence.
We’ve seen 30% compression from using gzip on FAST encoded data and this is in addition to the ~70-80% that you normally get in the FAST encoding step.
I’ve heard that FAST+gzip (or zip) is used for archival.
Best,
Rolf
According to my personal experience, FAST is not necessarily better
than zlib in terms of compression ratio. In fact those two are very
close. You can even apply FAST/zlib if the time overhead is tolerable
for application. However, FAST is more suitable for context-based
streaming process because the granularity of which is message-
by-message or record-by-record, instead of handling data in a batch
process way.
LiuAn,
ZLIB is a generic text compression library based on the LZ77 (Lempel Ziv 1977) family of compression algorithms. It allows you to compress any message stream without reconfiguration or adaptation so in that sense ZLIB is less complex to deploy. I would argue that this difference is small given that the implementation of a feed is content specific anyway.
The implementation (ie. the code) of FAST and ZLIB respectively are fairly complex but (having implemented both) I believe FAST is easier to implement than ZLIB.
As there are FAST and ZLIB implementations available, you don’t have to go to the trouble of actually doing your own implementation but can instead use an existing implementation.
ZLIB and ZLIB-like implementations have been used by the CME, OMX (SAXESS and CLICK), Nasdaq (ITCH) and others. As far as I know, CME has discontinued its use of ZLIB and OMX has discontinued at least some of their use of a ZLIB-like implementation. Not sure if ITCH compression is still being offered.
The big drawbacks with ZLIB is its relatively high processing overhead and its reliance on a long history of data to compress effectively. As a consequence ZLIB in not well-suited for high performance situations nor when UDP over multicast is used.
It has been shown that the processing overhead of ZLIB is more than 30 times higher than FAST in some use cases. The compression ratios of ZLIB and FAST are comparable for TCP streams and FAST has much better compression efficiency for in the UDP case.
In summary, if you plan to provide a low to medium thruput market data feed which is not latency sensitive and you plan to use TCP as your transport, then ZLIB could used to compress your data.
For other scenarios I would recommend that you review the commercial alternatives available for deploying a FAST-based feed.
Best,
Rolf
Hi all,
I’m considering to compress market data messages using zlib.
I found some discussions in this forum regarding zlib v.s. FAST. From performance point of view, FAST is a better choice. But FAST seems to be more complex in implementation.
I’d like to know if zlib is considered a valid implemetation for the market data compression? Is it used by anybody in production?
Thanks & Regards,
LiuAn