Imported from previous forum
Has anyone had any experience decoding the FAST multicast market data from ArcaBook?
I’m attempting to use the decoder they provide in the DevPak (and they insist works on their discussion forums) with sample data I downloaded from them, as well as some sample messages I found posted on their forums but am not having much luck.
If anyone has done this, and has any other sample data or knowledge/experience to pass along I would be very grateful.
Thanks,
Sean
Sean on the ‘FAST Protocol’ forum at fixprotocol.org wrote:
Has anyone had any experience decoding the FAST multicast market data
from ArcaBook?
Hi Sean,
I have built ArcaBook support on top of QuickFAST – the open source implementation of FAST that I maintain.
http://quickfast.googlecode.com
I must qualify this by saying that the ArcaBook-specific code won’t go into the open source repository in it’s current form – it’s too dependent on the system of the particular customer for which I did the work.
I’m attempting to use the decoder they provide in the DevPak (and
they insist works on their discussion forums) with sample data I
downloaded from them, as well as some sample messages I found posted
on their forums but am not having much luck.
Yes, the sample decoder and sample data were pretty much useless. It wasn’t until I started decoding the live data stream that I figured out what they were trying to say in the ArcaBook Multicast spec documentation.
If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.
I can’t provide sample data without a consulting contract, however what I learned about ArcaBook includes:
* The compressed ArcaBook data is best described as FAST-like. It differs from the FAST specification in several ways. To be fair, I believe Arca came first -- they were compressing their feed before the FAST spec existed.
* An ArcaBook packet has an uncompressed header. Within that header is a MsgType. By examining the MsgType you can determine whether the remainder of the packet is compressed -- some are, some are not.
* If it is compressed, the remainder of the packet is a sequence of compressed messages. NumBodyEntries tells you how many. You also have to examine MsgType to determine the size of the uncompressed header so you can find the start of the compressed body entries.
* From the table in Appendix C of the ArcaBook Multicast spec you can determine which fields exist in each message type, however, they do not necessarily appear in the order in which they are given in the table. They appear in the order in which they would have appeared in the uncompressed message (more or less.)
* The presence map bit assigned to each field is hard coded. As a result of this and the previous bullet point, the presence map bits are not consumed in order the way they are in a normal FAST message.
* ArcaBook includes some additional field types which are not in the FAST specification. On the other hand, the ArcaBook data does not contain groups, sequences, decimals, or ByteVector data which greatly simplifies matters.
I hope this helps gets you started.
Dale Wilson
Principal Software Engineer
Object Computing, Inc. (www.ociweb.com)
Thanks for the response Dale. At least you were able to confirm some of the things I suspected, such as the fact that it isn’t exactly FAST and whatnot.
I’m going to (try) and contact them and see about listening to live market data on their multicast channels, and capture that and look at it.
Interesting thing about that message type field in the uncompressed header, anytime I look at it in the sample data I obtained it is set to zero. I’m going to go back and revisit that right now as I am wondering if I was looking in the right place, although you’d think examining an uncompressed header side by side with the spec should be pretty easy.
Sean on the ‘FAST Protocol’ forum at fixprotocol.org wrote:
Has anyone had any experience decoding the FAST multicast market data
from ArcaBook?Hi Sean,
I have built ArcaBook support on top of QuickFAST – the open source
implementation of FAST that I maintain.http://quickfast.googlecode.com
I must qualify this by saying that the ArcaBook-specific code
won’t go into the open source repository in it’s current form –
it’s too dependent on the system of the particular customer for
which I did the work.I’m attempting to use the decoder they provide in the DevPak (and they
insist works on their discussion forums) with sample data I downloaded
from them, as well as some sample messages I found posted on their
forums but am not having much luck.Yes, the sample decoder and sample data were pretty much useless. It
wasn’t until I started decoding the live data stream that I figured out
what they were trying to say in the ArcaBook Multicast spec
documentation.If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.I can’t provide sample data without a consulting contract, however what
I learned about ArcaBook includes:* The compressed ArcaBook data is best described as FAST-like. It differs from the FAST specification in several ways. To be fair, I believe Arca came first -- they were compressing their feed before the FAST spec existed. * An ArcaBook packet has an uncompressed header. Within that header is a MsgType. By examining the MsgType you can determine whether the remainder of the packet is compressed -- some are, some are not. * If it is compressed, the remainder of the packet is a sequence of compressed messages. NumBodyEntries tells you how many. You also have to examine MsgType to determine the size of the uncompressed header so you can find the start of the compressed body entries. * From the table in Appendix C of the ArcaBook Multicast spec you can determine which fields exist in each message type, however, they do not necessarily appear in the order in which they are given in the table. They appear in the order in which they would have appeared in the uncompressed message (more or less.) * The presence map bit assigned to each field is hard coded. As a result of this and the previous bullet point, the presence map bits are not consumed in order the way they are in a normal FAST message. * ArcaBook includes some additional field types which are not in the FAST specification. On the other hand, the ArcaBook data does not contain groups, sequences, decimals, or ByteVector data which greatly simplifies matters.I hope this helps gets you started.
Dale Wilson Principal Software Engineer Object Computing, Inc.
(www.ociweb.com)
Hi Sean,
which of the decoders (exact name of .zip) have you downloaded and which sample data file where you getting problems with?
/Rolf
Has anyone had any experience decoding the FAST multicast market data
from ArcaBook?I’m attempting to use the decoder they provide in the DevPak (and they
insist works on their discussion forums) with sample data I downloaded
from them, as well as some sample messages I found posted on their
forums but am not having much luck.If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.Thanks, Sean
Rolf, I’ve download both ‘DevPakOptions302.zip’ and ‘complex_decode.zip’ from http://www.nyxdata.com/nysedata/default.aspx?tabid=712.
The sample data I have comes from ftp://sampledata@ftp2.nysedata.com/custom/ArcaBookMulticast.
I’ve tried all of the files there, in particular OTC_A-C_compacted_20090309.cap.gz and another file beginning with ‘tcpdump’. Which honestly makes me question what the data is, as the multicast stuff should be UDP (I thought, anyway).
Even looking at the hex bytes in a hex editor doesn’t help me - there is a weird preample on the files ‘0xD4 0xC3 0xB2 0xA1’ and I don’t understand what the data is/how it is laid out in the files.
I’ve done this before with CME messages and those are usually pretty easy to go through by hand - these don’t seem to match up to anything I can find in the spec. I also am hesitant with this sample data as I don’t know exactly when it was captured/posted so I wonder if it could by out of sync with what the spec says.
Hi Sean,
which of the decoders (exact name of .zip) have you downloaded and which
sample data file where you getting problems with?/Rolf
Has anyone had any experience decoding the FAST multicast market data
from ArcaBook?I’m attempting to use the decoder they provide in the DevPak (and they
insist works on their discussion forums) with sample data I downloaded
from them, as well as some sample messages I found posted on their
forums but am not having much luck.If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.Thanks, Sean
Sean, the weird preamble indicates that you are looking at a pcap formatted file (possibly produced by tcpdump -s0 -o …)
If this is indeed the case, then you have ETH, IP and UDP headers in front of the payload.
I’ll try and download one of the files and have a closer look.
HTH,
Rolf
Rolf, I’ve download both ‘DevPakOptions302.zip’ and ‘complex_decode.zip’
from http://www.nyxdata.com/nysedata/default.aspx?tabid=712.The sample data I have comes from
ftp://sampledata@ftp2.nysedata.com/custom/ArcaBookMulticast.I’ve tried all of the files there, in particular OTC_A-
C_compacted_20090309.cap.gz and another file beginning with ‘tcpdump’.
Which honestly makes me question what the data is, as the multicast
stuff should be UDP (I thought, anyway).Even looking at the hex bytes in a hex editor doesn’t help me - there is
a weird preample on the files ‘0xD4 0xC3 0xB2 0xA1’ and I don’t
understand what the data is/how it is laid out in the files.I’ve done this before with CME messages and those are usually pretty
easy to go through by hand - these don’t seem to match up to anything I
can find in the spec. I also am hesitant with this sample data as I
don’t know exactly when it was captured/posted so I wonder if it could
by out of sync with what the spec says.Hi Sean,
which of the decoders (exact name of .zip) have you downloaded and
which sample data file where you getting problems with?/Rolf
Has anyone had any experience decoding the FAST multicast market
data from ArcaBook?I’m attempting to use the decoder they provide in the DevPak (and
they insist works on their discussion forums) with sample data I
downloaded from them, as well as some sample messages I found posted
on their forums but am not having much luck.If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.Thanks, Sean
… and of course the pcap file header (24 bytes IIRC) at the beginning of the file and a pcap packet header (16 bytes) for each packet …
Sean, the weird preamble indicates that you are looking at a pcap
formatted file (possibly produced by tcpdump -s0 -o …)If this is indeed the case, then you have ETH, IP and UDP headers in
front of the payload.I’ll try and download one of the files and have a closer look.
HTH, Rolf
Rolf, I’ve download both ‘DevPakOptions302.zip’ and
‘complex_decode.zip’ from
http://www.nyxdata.com/nysedata/default.aspx?tabid=712.The sample data I have comes from
ftp://sampledata@ftp2.nysedata.com/custom/ArcaBookMulticast.I’ve tried all of the files there, in particular OTC_A-
C_compacted_20090309.cap.gz and another file beginning with ‘tcpdump’.
Which honestly makes me question what the data is, as the multicast
stuff should be UDP (I thought, anyway).Even looking at the hex bytes in a hex editor doesn’t help me - there
is a weird preample on the files ‘0xD4 0xC3 0xB2 0xA1’ and I don’t
understand what the data is/how it is laid out in the files.I’ve done this before with CME messages and those are usually pretty
easy to go through by hand - these don’t seem to match up to anything
I can find in the spec. I also am hesitant with this sample data as I
don’t know exactly when it was captured/posted so I wonder if it could
by out of sync with what the spec says.Hi Sean,
which of the decoders (exact name of .zip) have you downloaded and
which sample data file where you getting problems with?/Rolf
Has anyone had any experience decoding the FAST multicast market
data from ArcaBook?I’m attempting to use the decoder they provide in the DevPak (and
they insist works on their discussion forums) with sample data I
downloaded from them, as well as some sample messages I found
posted on their forums but am not having much luck.If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.Thanks, Sean
Rolf, my response to your e-mail regarding the password bounced. Here is what I sent:
Rolf, thanks for taking an interest in my post and questions, I really
appreciate it!
Anyway, I found that ftp site by stumbling across this post:
http://www.nyxdata.com/nysedata/Home/DiscussionBoard/tabid/108/forumid/4/postid/3744/view/topic/Default.aspx
Of course, that forum is for Openbook, but if you click on the link he
posts in the second post on that forum thread:
ftp://sampledata:datacap@ftp2.nysedata.com/custom
There are several subdirectories, one of which is ArcaBookMulticast.
It’s never asked me for a password, I kind of assumed it must be some
sort of generic ‘anonymous’ access. Was it prompting you for one?
… and of course the pcap file header (24 bytes IIRC) at the beginning
of the file and a pcap packet header (16 bytes) for each packet …Sean, the weird preamble indicates that you are looking at a pcap
formatted file (possibly produced by tcpdump -s0 -o …)If this is indeed the case, then you have ETH, IP and UDP headers in
front of the payload.I’ll try and download one of the files and have a closer look.
HTH, Rolf
Rolf, I’ve download both ‘DevPakOptions302.zip’ and
‘complex_decode.zip’ from
http://www.nyxdata.com/nysedata/default.aspx?tabid=712.The sample data I have comes from
ftp://sampledata@ftp2.nysedata.com/custom/ArcaBookMulticast.I’ve tried all of the files there, in particular OTC_A-
C_compacted_20090309.cap.gz and another file beginning with
‘tcpdump’. Which honestly makes me question what the data is, as the
multicast stuff should be UDP (I thought, anyway).Even looking at the hex bytes in a hex editor doesn’t help me -
there is a weird preample on the files ‘0xD4 0xC3 0xB2 0xA1’ and I
don’t understand what the data is/how it is laid out in the files.I’ve done this before with CME messages and those are usually pretty
easy to go through by hand - these don’t seem to match up to
anything I can find in the spec. I also am hesitant with this sample
data as I don’t know exactly when it was captured/posted so I wonder
if it could by out of sync with what the spec says.Hi Sean,
which of the decoders (exact name of .zip) have you downloaded and
which sample data file where you getting problems with?/Rolf
Has anyone had any experience decoding the FAST multicast market
data from ArcaBook?I’m attempting to use the decoder they provide in the DevPak
(and they insist works on their discussion forums) with sample
data I downloaded from them, as well as some sample messages I
found posted on their forums but am not having much luck.If anyone has done this, and has any other sample data or
knowledge/experience to pass along I would be very grateful.Thanks, Sean
Hi Sean,
Sean, the weird preamble indicates that you are looking at a pcap
formatted file (possibly produced by tcpdump -s0 -o …)If this is indeed the case, then you have ETH, IP and UDP headers in
front of the payload.
It sounds as if may have a better set of sample files than the ones I worked with. Having a pcap formatted file is actually a good thing.
There is an open source library, pcaplib (http://www.tcpdump.org/), that can read these files on *nix systems. There is a windows version as well, winpcap (http://www.winpcap.org/).
When I was working with QuickFAST I wanted to avoid adding dependencies on two external projects, and also I didn’t want to fight the portability problems of using pcaplib on *nix and winpcap on Windows, so I wrote a simple-minded pcap file reader that reads UDP/Multicast files only. For general purpose work you should obviously use one of the pcap libraries, but if all you need to do is read a capture of a multicast stream, you’re welcome to “borrow” the PCapToMulticast program that is one of the example programs in QuickFAST (http://quickfast.googlecode.com)
HTH,
Dale
Dale, I ended up grabbing a copy of that TCPDump and it has been a great help - thanks for letting me know about that.
I’ve got the UDP header identified and what appears to be the ArcaBook header identified, so I’m working on trying to run the actual compacted message data through the decoder they provide. So far, no luck really, it returns ‘0’ which means it was successful but it only processes 3 bytes and only fills in the message type in the destination buffer. But, thats further than I was, so I’ll take what I can get.
Better get back to it.
Thanks again,
Sean
Hi Sean,
Sean, the weird preamble indicates that you are looking at a pcap
formatted file (possibly produced by tcpdump -s0 -o …)If this is indeed the case, then you have ETH, IP and UDP headers in
front of the payload.It sounds as if may have a better set of sample files than the ones I
worked with. Having a pcap formatted file is actually a good thing.There is an open source library, pcaplib (http://www.tcpdump.org/), that
can read these files on *nix systems. There is a windows version as
well, winpcap (http://www.winpcap.org/).When I was working with QuickFAST I wanted to avoid adding dependencies
on two external projects, and also I didn’t want to fight the
portability problems of using pcaplib on *nix and winpcap on Windows, so
I wrote a simple-minded pcap file reader that reads UDP/Multicast files
only. For general purpose work you should obviously use one of the pcap
libraries, but if all you need to do is read a capture of a multicast
stream, you’re welcome to “borrow” the PCapToMulticast program that is
one of the example programs in QuickFAST
(http://quickfast.googlecode.com)HTH,
Dale