How to convert a FIX 5.0 FAST 1.1 dump file to a text (CSV) file?

, ,
1

Imported from previous forum

2

Hi,

for the purposes of academic research, we are trying to open and process a sample .dmp file which is in “FIX 5.0 FAST 1.1” format. Can you advice us on how open or convert the file to a comman file format like *.CSV?

In a later step, for efficiency, we would also like to upload the file content to a MySQL database.

Any comments are appreciated.

Best wishes,
Fred

3

Hi Fred,

how did you get the dump file?

Assuming this dump was done by some network tracing utility (as tcpdump, Solaris snoop etc), you could use wireshark to analyse it:

There should be versions for a couple of operating systems available.

In order to decode the FAST messages, you need a special FAST dissector
found here:

http://code.google.com/p/fast-wireshark/

There are also set up instructions included.

Please reply if you have any issues to get it running.

Cheers, Jörg

Hi,

for the purposes of academic research, we are trying to open and process a sample .dmp file which is in “FIX 5.0 FAST 1.1” format. Can you advice us on how open or convert the file to a comman file format like *.CSV?

In a later step, for efficiency, we would also like to upload the file content to a MySQL database.

Any comments are appreciated.

Best wishes,
Fred

4

Hi Jörg,
Thanks for your prompt reply and help!

The file is from the exchange itself, Deutsche Börse. It’s a sample file of order book data found here: http://deutsche-boerse.com/mda/dispatch/en/kir/gdb_navigation/mda/200_market_data/650_historical_market_data/35_enbs_hist_data_xetra

Best regards,
Fred

Hi Fred,

how did you get the dump file?

Assuming this dump was done by some network tracing utility (as tcpdump, Solaris snoop etc), you could use wireshark to analyse it:

http://www.wireshark.org/

There should be versions for a couple of operating systems available.

In order to decode the FAST messages, you need a special FAST dissector
found here:

http://code.google.com/p/fast-wireshark/

There are also set up instructions included.

Please reply if you have any issues to get it running.

Cheers, Jörg

Hi,

for the purposes of academic research, we are trying to open and process a sample .dmp file which is in “FIX 5.0 FAST 1.1” format. Can you advice us on how open or convert the file to a comman file format like *.CSV?

In a later step, for efficiency, we would also like to upload the file content to a MySQL database.

Any comments are appreciated.

Best wishes,
Fred

5

Fred,

since you are from Goethe University my guess it that you are doing some academic reasearch.

Please tell me a bit more about the things you want to do.
I happen to know some people from DB glass cube and could arrange
some contacts if it seems suitable.

Thanks, Jörg

Hi Jörg,
Thanks for your prompt reply and help!

The file is from the exchange itself, Deutsche Börse. It’s a sample file of order book data found here: http://deutsche-boerse.com/mda/dispatch/en/kir/gdb_navigation/mda/200_market_data/650_historical_market_data/35_enbs_hist_data_xetra

Best regards,
Fred

Hi Fred,

how did you get the dump file?

Assuming this dump was done by some network tracing utility (as tcpdump, Solaris snoop etc), you could use wireshark to analyse it:

http://www.wireshark.org/

There should be versions for a couple of operating systems available.

In order to decode the FAST messages, you need a special FAST dissector
found here:

http://code.google.com/p/fast-wireshark/

There are also set up instructions included.

Please reply if you have any issues to get it running.

Cheers, Jörg

Hi,

for the purposes of academic research, we are trying to open and process a sample .dmp file which is in “FIX 5.0 FAST 1.1” format. Can you advice us on how open or convert the file to a comman file format like *.CSV?

In a later step, for efficiency, we would also like to upload the file content to a MySQL database.

Any comments are appreciated.

Best wishes,
Fred

6

Fred, I forgot to mention that you also need the FAST templates which define this message. It could be tricky to get the correct version of the EnBS templates.

Cheers, Jörg

Fred,

since you are from Goethe University my guess it that you are doing some academic reasearch.

Please tell me a bit more about the things you want to do.
I happen to know some people from DB glass cube and could arrange
some contacts if it seems suitable.

Thanks, Jörg

Hi Jörg,
Thanks for your prompt reply and help!

The file is from the exchange itself, Deutsche Börse. It’s a sample file of order book data found here: http://deutsche-boerse.com/mda/dispatch/en/kir/gdb_navigation/mda/200_market_data/650_historical_market_data/35_enbs_hist_data_xetra

Best regards,
Fred

Hi Fred,

how did you get the dump file?

Assuming this dump was done by some network tracing utility (as tcpdump, Solaris snoop etc), you could use wireshark to analyse it:

http://www.wireshark.org/

There should be versions for a couple of operating systems available.

In order to decode the FAST messages, you need a special FAST dissector
found here:

http://code.google.com/p/fast-wireshark/

There are also set up instructions included.

Please reply if you have any issues to get it running.

Cheers, Jörg

Hi,

for the purposes of academic research, we are trying to open and process a sample .dmp file which is in “FIX 5.0 FAST 1.1” format. Can you advice us on how open or convert the file to a comman file format like *.CSV?

In a later step, for efficiency, we would also like to upload the file content to a MySQL database.

Any comments are appreciated.

Best wishes,
Fred

7

Thanks Jörg, in the meantime DB was so kind to provide me with a Java-based command-line decoder with which I succeeded in converting the dump file to a text file. It included the Xetra Release 12.0 template which I think you were referring to. The output file seems to be in JSON format and is over 1GB.

The next challenge is for me to “understand” the data structure. Ideally, I would like to upload the data to a MySQL database. However, it’s not clear to me how to decompose the file semantically, to see what matters and what not. Currenty, I’m looking for docs and trying to get a “bid-eye” perspective on the data structure.

8

I was mistaken, it’s NOT in JSON format; every line looks like a bit like this:

InsideMarketSnapshotInformation -> {…, …, …, …}

Does anybody know what format this is? Is it common or specific to the Java parser I used?

9

Fred,

this discussion is to far off the standard FIX and FAST topics.

Would you like to contact me via mail?

Cheers, Jörg

I was mistaken, it’s NOT in JSON format; every line looks like a bit like this:

InsideMarketSnapshotInformation → {…, …, …, …}

Does anybody know what format this is? Is it common or specific to the Java parser I used?

10

I think that’s a good idea. Although I don’t want to take too much of your time.

11

I just installed Wireshark and tried to open a “captured file”, but it fails saying “The file … isn’t a capture file in a format Wireshark understands.”

Hi Fred,

how did you get the dump file?

Assuming this dump was done by some network tracing utility (as tcpdump, Solaris snoop etc), you could use wireshark to analyse it:

http://www.wireshark.org/

There should be versions for a couple of operating systems available.

In order to decode the FAST messages, you need a special FAST dissector
found here:

http://code.google.com/p/fast-wireshark/

There are also set up instructions included.

Please reply if you have any issues to get it running.

Cheers, Jörg

Hi,

for the purposes of academic research, we are trying to open and process a sample .dmp file which is in “FIX 5.0 FAST 1.1” format. Can you advice us on how open or convert the file to a comman file format like *.CSV?

In a later step, for efficiency, we would also like to upload the file content to a MySQL database.

Any comments are appreciated.

Best wishes,
Fred