Imported from previous forum
We are defining a Market Data Protocol using FIX over FAST, over TCP.
The MsgSeqNum TAG(34) is usually defined as “increment” field in FAST templates, so the MsgSeqNum TAG will never sent across the network.
If the receiving application lost an entire FIX message over the FAST layer or over the network, it cannot realy check the FIX messages sequence.
Could be a valid solution to define the MsgSeqNum TAG as “copy” field on the FAST template (even if this will enlarge the bandwidth)?
Loss of a FIX message over TCP will show in a gap in the MsgSeqNum values. However, MsgSeqNum only makes sense in the context of the FIX session layer which you do not seem to be using. What concept do you intend to offer for recovery? MarketDataRequest messages do not work as they are functional (you cannot ask for messages from x to y).
Have a look at FIX Application Sequencing where you move the sequencing to the application level. ApplSeqNum (1181) acts similar to MsgSeqNum but in the context of an ApplID (1180) that you assign yourself. ApplicationMessageRequest messages can then recover from gaps, i.e. you take in such a request and re-issue the market data messages with the ApplSeqNum values inside the gap. ApplicationMessageRequest can also be used to inquire the last valid sequence number to make sure that one has not lost the last n messages (gap at the end of a transmission).
Please refer to FIX 5.0 SP2 Specification Volume 1, pages 117-127.
Regards,
Hanno.
We are defining a Market Data Protocol using FIX over FAST, over TCP.
The MsgSeqNum TAG(34) is usually defined as “increment” field in FAST templates, so the MsgSeqNum TAG will never sent across the network.
If the receiving application lost an entire FIX message over the FAST layer or over the network, it cannot realy check the FIX messages sequence.
Could be a valid solution to define the MsgSeqNum TAG as “copy” field on the FAST template (even if this will enlarge the bandwidth)?
Correct me if I’m wrong. But if you lose a FAST message. Won’t the dictionaries be incorrect? Or do you plan to send a fast reset as part of each message?
Again, if you would lose parts of a FAST message, can you actually recover and find out where the next message begins? Don’t you need to reply on that TCP/IP don’t drop packages to be able to utilize FAST over TCP/IP?
Loss of a FIX message over TCP will show in a gap in the MsgSeqNum values. However, MsgSeqNum only makes sense in the context of the FIX session layer which you do not seem to be using. What concept do you intend to offer for recovery? MarketDataRequest messages do not work as they are functional (you cannot ask for messages from x to y).
Have a look at FIX Application Sequencing where you move the sequencing to the application level. ApplSeqNum (1181) acts similar to MsgSeqNum but in the context of an ApplID (1180) that you assign yourself. ApplicationMessageRequest messages can then recover from gaps, i.e. you take in such a request and re-issue the market data messages with the ApplSeqNum values inside the gap. ApplicationMessageRequest can also be used to inquire the last valid sequence number to make sure that one has not lost the last n messages (gap at the end of a transmission).
Please refer to FIX 5.0 SP2 Specification Volume 1, pages 117-127.
Regards,
Hanno.We are defining a Market Data Protocol using FIX over FAST, over TCP.
The MsgSeqNum TAG(34) is usually defined as “increment” field in FAST templates, so the MsgSeqNum TAG will never sent across the network.
If the receiving application lost an entire FIX message over the FAST layer or over the network, it cannot realy check the FIX messages sequence.
Could be a valid solution to define the MsgSeqNum TAG as “copy” field on the FAST template (even if this will enlarge the bandwidth)?
We plan to implement a market data protocol:
- FIX 5.0 Market Data, over FIX session layer FIXT1.1
- the FIXT 1.1 will be implemented over FAST 1.2 and FAST over TCP
In my previous mail I simplified the scenario.
The main use of market data protocol is: - initiator sends a market data request
- acceptor send a first message of market data snapshot
- further updates are sent via Incremental refresh
The application protocol does not really need of an Application Sequence number to recover data (the market data snapshot will send all needed information).
Our intention was to check message sequencing at FIX session level (to detect if message are lost by FAST or TCP layers)
The receiving FAST layer will automatically increment the MsgSeqNum so the check done by the FIX session layer refers to s Sequence number generated locally to the application and migh not be the real sequence number.
Configuring the MsgSeqNum as copy operator could help to detect loss of messages across protocol layers.
We know that TCP is a connection oriented reliable protocol, howevever systems loose messages sometimes doing to software bugs.
We think that discovering GAP at reconnection is too late if the information we loose was important.
So we are looking for a standard way to (online) dectect gaps in fix protocol over FAST on TCP Transport.
Our intention was to check message sequencing at FIX session level (to detect if message are lost by FAST or TCP layers)
As Lars and I both said in different ways: This cannot and will not happen unless the connection is lost. EVER.
The receiving FAST layer will automatically increment the MsgSeqNum so the check done by the FIX session layer refers to s Sequence number generated locally to the application and migh not be the real sequence number.
Configuring the MsgSeqNum as copy operator could help to detect loss of messages across protocol layers.
Just as an aside, COPY does not make sense. If you want to force the sequence number onto the wire, use no operator.
We know that TCP is a connection oriented reliable protocol, howevever systems loose messages sometimes doing to software bugs.
If there were a bug in the TCP layer, you would not lose messages. You would lose chucks of data out of the middle of the messages. The FAST decoder would not be able to decode the resulting stream of data and it would never recover from the problem. If it is well written, the FAST decoder will report the error to your application. The connection is now useless. You will never see the message “after the gap.”
If there is a bug in the FAST decoder, then it might be possible to lose a message. However that would mean that all data following the lost message would be invalid and untrustworthy. In fact if the FAST decoder has this type of bug all data delivered to you application will be unreliable whether or not there is a a message sequence number gap.
If that were to occur, the solution would be to find and fix the bug in the FAST decoder. Assuming as you indicate the data is important, the worst thing you could do would be to “recover” enough to let you continue to operate with bogus data.
We think that discovering GAP at reconnection is too late if the information we loose was important.
That’s unfortunate, because it is the only option you have. If the data coming in cannot be decoded, you could certainly report the problem to the application BEFORE dropping and reconnecting the line, but you wouldn’t know, yet, if any data had been lost.
So we are looking for a standard way to (online) dectect gaps in fix protocol over FAST on TCP Transport.
The standard way to handle the problems that might occur on a TCP connection but that aren’t automatically fixed for you by TCP is to drop the connection and open a new one. Any other attempt to recover from missing data in a FAST data stream will give you invalid results if it “works” at all.
Dale
We are defining a Market Data Protocol using FIX over FAST, over TCP.
…in which case you don’t need to worry about losing a message unless you lose the connection.
TCP is a reliable, stream oriented protocol. If you were to lose a packet without losing the connection (which won’t happen because the protocol is reliable) you would have far more to worry about than the message sequence number. Because TCP is sending a stream, the packet boundaries do not correspond to message boundaries. If you lose any data in a stream oriented protocol such as TCP when sending FAST data, there is no way to recover other than dropping the connection and starting over with a new connection.
When you start the new connection there will be no previous value to be incremented so FAST will send the MsgSeqNum for the first message on the wire (you are wrong about it “never” being sent across the network.) This will let you know if any messages were lost during the disconnect/reconnect process and you may take appropriate recovery action.
Dale