Tablature roundtrip produces empty repository file

1

Hello community,

the Tablature tools on https://github.com/FIXTradingCommunity/tablature convert the FIX Orchestra format into Markdown and vice versa. Many thanks to @donaldmendelson for providing them as open-source at GitHub!

As suggested at the end of the Tablature User Guide for Messages I downloaded the current Orchestra FIX Latest file and called:

orchestra2md OrchestraFIXLatest.xml -o OrchestraFIXLatest.md
md2orchestra OrchestraFIXLatest.md -o OrchestraFIXLatest-1.xml

I expected the OrchestraFIXLatest-1.xml file to have the same contents as the original OrchestraFIXLatest.xml but for some reason it was nearly empty:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fixr:repository xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:fixr="http://fixprotocol.io/2020/orchestra/repository">
    <fixr:metadata/>
    <fixr:datatypes/>
    <fixr:codeSets/>
    <fixr:fields/>
    <fixr:components/>
    <fixr:groups/>
    <fixr:messages/>
</fixr:repository>

What did I wrong? Was my expectation inappropriate?

Thanks, Jörg

2

@jorgthonnes, thanks for your comments. With user feedback we can make Tablature more useable.

First, let me set the expectation. Without adding some command line arguments, md2orchestra and orchestra2md do not output everything from the input, by design. The objective is to make it easy for a user to create rules of engagement for a firm, not to output the entire FIX standard. So a round-trip will not reproduce the original file by default.

Both applications are selective about translation in different ways.

By default orchestra2md does not output datatypes, FIXML attributes, or pedigree of elements (history of add, update, deprecate) – all things in the standard Orchestra file. You can turn these on selectively, or give the argument -a to output all features. (If you just type a command without any arguments, it will print usage with all controlling features.

The md2orchestra application is selective in a different way. It does not output nested components beyond the first level by default. I believe that in the FIX standard there are a few instances of components nested to the 5th or 6th level, but most ROEs do not use them in that way. You can however, turn on deeper nesting with either -d <search-depth> or --fullsearch for all nesting levels.

One last issue, which probably explains your result, and I’m sure we can improve on it. I suspect that md2orchestra is not actually reading your input file. If it does, you should see a line like this in the log:

2022-01-27 09:06:28,912 INFO : Md2Orchestra opening file OrchestraFIXLatest.md

By way of explanation, md2orchestra can input multiple markdown files, so you can divide your work, e.g. a markdown file with a list of fields, another with messages, etc. You can then give an input wildcard like *.md. The program matches the pattern to actual filenames. This may working slightly different on different platforms. Try giving the input file name with a full path or even ./OrchestraFIXLatest.md and let me know if that works for you.

3

@donaldmendelson, you were right, prepending ./ on my Linux system did the job:

md2orchestra ./OrchestraFIXLatest.md -o OrchestraFIXLatest-1.xml

and the tool logged a lot more on stdout. This is much better than on the previous days when I tried and tried and got no result.

And as well, you are right that this should be improved.

Many thanks, Jörg

P.S. Do you have an indication how many people already using these tools? This would be valuable feedback.

4

Good to know that it worked. When I get time, I try to write fuller documentation. There are pages in wiki, but they may be slightly out of date. The only way I know when someone is using the tool is when they speak out, like yourself.

We have plans to make it easier by exposing Tablature with a web interface for FIX members. It is largely already written, but some other tools had higher priority for roll-out.

1 Like
5

I am happy to help with the documentation and will report back to the forum about my progress using the tools. Thank you!

6

@donaldmendelson:

Comparing the output of md2orchestra to the Orchestra file of FIX Latest, I see some notable differences: While the Orchestra FIX Latest only uses the dc: XML name space,
the generated XML code uses dcterms: which causes validation errors in my IDE.

I think that the <fixr:metadata> elements only allows <dc:*>. The Orchestra FIX Latest adheres to this restriction:

<?xml version="1.0" encoding="UTF-8"?>
<fixr:repository xmlns:dc="http://purl.org/dc/elements/1.1/"
                 xmlns:fixr="http://fixprotocol.io/2020/orchestra/repository"
                 xmlns:xs="http://www.w3.org/2001/XMLSchema"
                 name="FIX.Latest"
                 version="FIX.Latest_EP269">
   <fixr:metadata>
      <dc:title>Orchestra</dc:title>
      <dc:creator>unified2orchestra.xslt script</dc:creator>
      <dc:publisher>FIX Trading Community</dc:publisher>
      <dc:date>2021-08-14T22:38:48.950856Z</dc:date>
      <dc:format>Orchestra schema</dc:format>
      <dc:source>FIX Unified Repository</dc:source>
      <dc:rights>Copyright (c) FIX Protocol Ltd. All Rights Reserved.</dc:rights>

I think the problem is the Tablature RepositoryMapper.

Filed https://github.com/FIXTradingCommunity/tablature/issues/62 to discuss this in more detail.

Cheers, Jörg