Imported from previous forum
what is the advantage of FAST over GZIP. GZIP compression is better than FAST. We can send very well compressed data ( around 95% compressed) on the network with GZIP. Can somebody throw light on this ?
what is the advantage of FAST over GZIP. GZIP compression is better than FAST. We can send very well compressed data ( around 95% compressed) on the network with GZIP. Can somebody throw light on this ?
GZIP is slow. FAST is … er … fast.
Dale
I compressed a file using FAST and GZIP both. GZIP completes the task in 3 minutes , while FAST takes 15 minutes.
I compressed a file using FAST and GZIP both. GZIP completes the task in 3 minutes , while FAST takes 15 minutes.
hm, whIch data, which FAST impl?
I compressed a file using FAST and GZIP both. GZIP completes the task in 3 minutes , while FAST takes 15 minutes.
Interesting. I recently decoded a file containing 850+ million messages in 4 minutes with QuickFAST (that’s including file I/O time by the way). That means in 15 minutes I could do a little better than three billion messages. Encoding takes slightly longer but one of the characteristics of FAST is it doesn’t have to examine the data to determine how to compress it – it’s guided by the templates – so encoding times are comparable to decoding times.
Let’s estimate that encoding is an order of magnitude slower than decoding. If you don’t have at least 300 million messages in your test file, you might want to check your FAST implementation and make sure the templates you are using are appropriate for the messages being compressed.
Oh, and you might try encoding those messages one at a time with GZIP rather than doing the entire file. That more closely matches the real world where no one is willing to wait until 300 million messages are accumulated before they get to see the first one.
[ original email was from Mark Reece - mark.reece@hsbcib.com ]
… and that per message immediacy is the use case I had when trialling FAST at an exchange.
For reference, the commercial FAST engine we were testing was able to sustain a compression time of less than 1 microsecond about 3 years ago - I should imagine that state of the art encoders are faster again now, given the huge speed up in Intel CPUs, if nothing else.
ie the 300m messages would then have taken us less than 5 minutes, doing them 1 message at a time.
It is also important to distinguish between the encode/decode time and the file or network I/O time. Typically, FAST would be inserted in as an encoding step for publication. In our case, it would have replaced the legacy formater in the tcker plant. Therefore, while the absolute encode time is very interesting, it is also whether it is quicker than the legacy encoder.
And even this may not be a simple answer - one test was a race between legacy and FAST - the FAST packets arrived at the switch ahead of the legacy packet despite, at that stage, the FAST encoding being in addition to the legacty encoding (ie not a substitute). So although the sub-microsecond had to be added, the benefit from the reduction in message serialisation was enough to overcome that. (ie substituted rather than “tacked on” would have been even better)
Mark Reece
I compressed a file using FAST and GZIP both. GZIP completes the task in 3 minutes , while FAST takes 15 minutes.
Interesting. I recently decoded a file containing 850+ million messages in 4 minutes with QuickFAST (that’s including file I/O time by the way). That means in 15 minutes I could do a little better than three billion messages. Encoding takes slightly longer but one of the characteristics of FAST is it doesn’t have to examine the data to determine how to compress it – it’s guided by the templates – so encoding times are comparable to decoding times.
Let’s estimate that encoding is an order of magnitude slower than decoding. If you don’t have at least 300 million messages in your test file, you might want to check your FAST implementation and make sure the templates you are using are appropriate for the messages being compressed.
Oh, and you might try encoding those messages one at a time with GZIP rather than doing the entire file. That more closely matches the real world where no one is willing to wait until 300 million messages are accumulated before they get to see the first one.
Thats gr8 performance. Are 800M messages part of small packets of 10-15 messages or contiguous messages. Secondly if the total messages include packets; are packets PAMP is reset everytime for new packet ? or continued with the old PMAP. In case packets are not reset and packet loss could create the problem.
[ original email was from Mark Reece - mark.reece@hsbcib.com ]
The messages were small - most around 100-150 bytes with a max around 200 before compression.
Mixed message types, each with its own template. Each message was in its own multicast packet. So no dependencies between packets and PMAP reset.
Thats gr8 performance. Are 800M messages part of small packets of 10-15 messages or contiguous messages. Secondly if the total messages include packets; are packets PAMP is reset everytime for new packet ? or continued with the old PMAP. In case packets are not reset and packet loss could create the problem.