working with my own FIX Orchestra files in XML and Markdown formats I while like to reach a “stable round trip” with Tablature before I start adding new fields to the Markdown. But for some reasons this does not work.
The round trip XML - Markdown - XML (with reference file only) removes the “added” and “updated” attributes (maybe some pedigree flag is missing) and the category attributes have been removed as well.
It would be helpful to know which command line parameter I should use for every direction.
Currently, I just use “-r” for the reference file (besides “-o” for the output, of course).
@jorgthonnes orchestra2md does not output all attributes by default. This was a deliberate design decision. The assumption was that pedigree (history of elements) and FIXML-specific attributes only matter internally to FIX GTC, not for firm rules of engagement. (Perhaps that is an incorrect assumption.)
In any case, output can be controlled by command line arguments. To output all attributes, use arguments -a or --all. To get help on usage, use -? or --help.
@jorgthonnes Tablature supports roundtrip in the sense that information you stored in one representation (XML or md) makes it back into that same representation after having been converted to the other representation and back. It does not support roundtrip in the sense of identical files on a byte level.
The roundtrip capability is useful for testing the tool itself. The assumption is that there is no use case for a development workflow that contains a regular conversion in both directions once you are in maintenance mode. At some point you have to decide whether the Orchestra XML file or your markdown file is the “golden source” for your RoE. You then make maintenance changes to that representation, either manually or with the help of a tool, and use Tablature (or the tool offers this) to regularly convert it to the other representation.
Please elaborate on the use case you have that requires to do a roundtrip every time you want to make a change.
A reference file is not a valid input parameter when converting from Orchestra XML to Tablature markdown. Only the content of your XML input file is used to create the markdown file. Did you not get the error Unrecognized option: -r when running the command above?
There is no need for an update at all if your Rules of Engagement have not changed. If markdown is your golden source then you should only use md2orchestra to generate the XML file whenever you change the markdown file. For example, if you add new fields or values from the latest extension pack, then you can update your XML file with the corresponding OrchestraFIXLatest.xml. I do not see a case for a roundtrip here.
Maybe I am to test-driven here as a software developer We do like idempotent operations.
How about this scenario:
My RoE uses a specific tag or group in my messages.
A new EP adds new values for this tag or new fields to this group.
In order to update my RoE I re-run the round trip using the updated reference file.
In that case, I would expect no updates besides those related to the specific tag or group.
Now I can be a bit more specific about the “unexpected” changes:
The header of the old XML file generated from the golden Markdown some months ago looks like this:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fixr:repository xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:fixr="http://fixprotocol.io/2020/orchestra/repository"
name="Rules of Engagement of the FIX Trading Adapter"
version="1.0">
<fixr:metadata>
<dc:title>Rules of Engagement of the FIX Trading Adapter</dc:title>
The new XML file generated from the golden Markdown using the latest reference file and the latest Tablature versions looks like this:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<fixr:repository xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:fixr="http://fixprotocol.io/2020/orchestra/repository"
name="Rules of Engagement of the FIX Trading Adapter version 1.0"
version="1.0">
<fixr:metadata>
<dcterms:title>Rules of Engagement of the FIX Trading Adapter</dcterms:title>
Some points to note:
Line 3: The name space declaration for dcterms: has been added.
Line 8: The dc: namespace usage has been replaced by dcterms:.
Line 2: The now obsolete declarations of dc: has been kept (could be removed).
Line 5: The version marker before 1.0 was added to the name.
Regarding the last point I would expect the attribute name to be without the version suffix since this is duplicated in the version attribute.
There are also a lot of white-space changes I did not report here. I had to reformat the XML file to get some useful differences.
The change of namespace for Dublin Core Terms was deliberate to be consistent with other tools. It’s quite complicated, but their namespaces overlap. Dublin Core Terms is an industry standard for metadata of artifacts, the who, what, when. The XML schema with alias “dc” only contains the original 15 terms while the “dcterms” contains all of the terms, including the ones in “dc”. Full explanation here.
Updating the reference file won’t necessarily apply all changes from an EP. One approach in XML is to use the XML Diff/Merge Utilities to selectively apply an EP to your own rules of engagement.
Thanks for pointing to the XML Diff/Merge Utilities.
Actually I found one issue with the changed formatting of fixr:documentation elements:
Old XML:
<fixr:documentation purpose="SYNOPSIS" contentType="text/markdown">Defines message type ALWAYS
THIRD FIELD IN MESSAGE. (Always unencrypted) Note: A "U" as the first character in the
MsgType field (i.e. U, U2, etc) indicates that the message format is privately defined
between the sender and receiver. *** Note the use of lower case letters ***
</fixr:documentation>
New XML:
<fixr:documentation purpose="SYNOPSIS" contentType="text/markdown">Defines message type ALWAYS THIRD FIELD IN MESSAGE. (Always unencrypted) Note: A "U" as the first character in the MsgType field (i.e. U, U2, etc) indicates that the message format
<fixr:documentation purpose="SYNOPSIS">
Note: A "U" as the first character in the MsgType field (i.e. U, U2, etc) indicates that the message format is privately defined between the sender and receiver.</fixr:documentation>
<fixr:documentation purpose="SYNOPSIS">
*** Note the use of lower case letters ***</fixr:documentation>
The text starting Note: A "U" as the first character is duplicated and appears both in the first and the second documentation element.
The duplication is forwarded to the Markdown file.
I am still confused by your use of the term “roundtrip”. Roundtrip means to either go from XML to md and back to XML or to go from md to XML and back to md. You want to go from a previous XML to a new XML and convert that to markdown. If that is the case then your golden source is XML and not markdown. As @donaldmendelson pointed out, you can use XML Diff/Merge to do the first step. Use XML Diff to obtain the difference between two versions of OrchestraFIXLatest.xml. Strip that down to the new “stuff” that you need (GTC may have added 20 new values to a field but you only want 5 of them) and use XML Merge to update your golden source, which is a third XML file with the subset of what you actually support.
You point to various differences in the meta-data section between what Tablature generates and what is in OrchestraFIXLatest.xml. They do not have to be identical (and should not because the former is YOUR meta-data and the latter is FIX meta-data). They only have to follow the Orchestra v1.0 schema. We have probably made changes to Tablature versions regarding the meta-data and GitHub has the public audit trail. It is open source and anybody can submit pull requests to make changes. FIX is the maintainer of this project and approves suggested changes whilst making its own as well.
Thanks for the detailed explanation. It seems that your workflow is quite different from mine.
If I see the Markdown as the golden source the only way to incorporate changes to the official FIX repository is using the reference file option “-r”. The resulting XML file will also contain the official updates.
Maybe the golden source must be XML to use XML Diff/Merge. I will consider this.
I agree we have simply been talking about two different workflows. Your workflow is about the question “How should I maintain an Orchestra repository for my RoE?”. My understanding of your workflow had initially been that your question was “How can I convert an existing Orchestra repository back and forth between the XML and markdown representation?”. I believe the difference in interpretation of the term “roundtrip” caused the confusion on my side.
The choice of the golden source has implications so that it is very important to put down the business requirements first before taking a decision. Both formats have their pros and cons and it really depends on your specific situation what is best. This forum can only explain the tools and how they work. The choice of tools and workflows is a design decision and much more complex.
The assumption is that users will rather be faced with the addition of new fields and values that are already in FIX Latest (but not in one’s own subset) than with the need to add fields and values that have just been released with the latest EP. In any case you need a process to add elements to your own subset, including user-defined fields and/or values that are not available in FIX Latest.
We do like idempotent operations.
Welcome to the world of semantics guys 
