Imported from previous forum
Hi,
is there are a standard way to fragment a FIX message?
I want to split up a SecurityList message and looked through the FIX 4.3 standard, but I didn’t found anything how this should be done.
Should all message fragments have the same sequence number?
Should the field TotalNumSecurities (393) in the SecurityList message be decremented in each fragment?
Thanks
Oliver
[ original email was from Aaron Pryce - aaron.pryce@citigroup.com ]
Oliver
Each message fragment will be a new message from the FIX session perspective
and will have a unique sequence number.
TotalNumSecurities (393) would not be decremented with each message as it refers to the total
number of securites in the fragmented block.
rgds
Aaron
> Hi,
>
> is there are a standard way to fragment a FIX message?
> I want to split up a SecurityList message and looked through the FIX 4.3 standard, but I didn’t found anything how this should be done.
>
> Should all message fragments have the same sequence number?
>
> Should the field TotalNumSecurities (393) in the SecurityList message be decremented in each fragment?
>
> Thanks
>
> Oliver
>
[ original email was from Jim Northey - jnorthey@lasalletech.com ]
Oliver, Aaron:
We didn’t provide a recommendation as to how to fragment messages when we developed the FIX 4.2 security definition message, however we did work through some examples. Aaron has pointed out one example that is used - each security is sent as a separate message - the SecurityReponseID(tag 322) can be used as the sequence number (not to be confused with MsgSeqNum please) for each response, although this is not needed - as the session layer provides sequencing. As Aaron indicates, the TotNumSecurities(tag 393)should be set to the total number of securities being reported across the entire response and should be unchanging across the messages that make up the response.
In this example if 10 securities were being returned as a result of a request:
Msg N:TotNumSecurities=10, SecurityResponseID=1
Msg N+1:TotNumSecurities=10, SecurityResponseID=2
Msg N+2:TotNumSecurities=10, SecurityResponseID=3
…
Msg N+9:TotNumSecurities=10, SecurityResponseID=10
Again, I am hesitant to say that SecurityResponseID should be set to a serial number - as this is not necessary - if it helps to provide uniqueness or some other purpose great, knowing that messages will be delivered in sequence by the transport and knowing with each record that you will be receiving 10 total should be enough information. The SecurityResponseID does not have to change or be unique across multiple responses to a request.
Sending one security per security list message (or security definition message in the case of FIX 4.2) can be a bit inefficient in terms of the ratio of information content vs. message overhead. At the CBOE we chose to batch the response into groups in each message. Currently we return 100 securities per message. This 100 is a system configurable value that can be tuned to improve overall throughput.
In the case of batching - the SecurityResponseID does not indicate the security number, but instead a batch number (again the SecurityResponseID just provides a unique identifier for the message, it does not a great deal of value beyond that).
In this case, you have to take the NoRelatedSym(tag 146) into account to determine how many securities from the overall response are contained in the message.
Assume that there are 10 securities per security list message (batch/fragment size) and that there are a total of 99 securities being returned - the messages would be as follows (switching to tag numbers for message size efficiency :)):
Msg N: TotNumSecurities(tag 393)=99, SecurityResponseID(tag 322)=“Today’s Batch#1”,
NoRelatedSym(tag 146)=10
Msg N+1: 393=99,322=“Here is your second batch”,146=10
Msg N+2: 393=99,322=“3rd batch”,146=10
Msg N+9: 393=99,322=“tenth”,146=9
During message design and review for FIX 4.2 the ECN & Exchanges working group was hesitant to put any qualifications or just one way of fragmenting messages. It seemed like an artificial restriction on implementations. Instead a more generic approach was developed wherein each message should be self standing - i.e. you could take that message and run with it - not needing previous or subsequent responses to be queued before processing. This is accomplished by including the TotNumSecurities on each message.
We also wanted the number of securities to be returned per message to be variable based upon application specific criteria. Implementors can adapt to this by writing their applications so that they use TotNumSecurities on the first response to determine the number of subsequent elements to be returned to allocate a hold space (if needed). Each message can be examined for the NoRelatedSym(tag 146) to determine how many of the total number are being returned on that message. If a non-array data structure is used - this is enough information to process the requests - no intermediate information is required. If an array is used - the number received so far has to be maintained (next element).
So to conclude, now that we have discussed one security per message and a batch of securities per message, please forget both of these delineations and focus on receiving TotNumSecurities back and each message received will have between [ 1 and TotNumSecurites ] securities contained within it. Using this approach - your application should be able to be written genericly to work with multiple implementations.
Hope this helps. If you think we should include something about this in the specification - let us know - we can add it to the FIX 4.4 Changes and track it for the next version of FIX.
Jim
> Oliver
>
> Each message fragment will be a new message from the FIX session perspective
> and will have a unique sequence number.
>
> TotalNumSecurities (393) would not be decremented with each message as it refers to the total
> number of securites in the fragmented block.
>
> rgds
> Aaron
>
>
> > Hi,
> >
> > is there are a standard way to fragment a FIX message?
> > I want to split up a SecurityList message and looked through the FIX 4.3 standard, but I didn’t found anything how this should be done.
> >
> > Should all message fragments have the same sequence number?
> >
> > Should the field TotalNumSecurities (393) in the SecurityList message be decremented in each fragment?
> >
> > Thanks
> >
> > Oliver
> >
>
Hi Jim,
I think it can be helpful when somthing like your example can be added to FIX 4.4.
Regards
Oliver
> Oliver, Aaron:
>
> We didn’t provide a recommendation as to how to fragment messages when we developed the FIX 4.2 security definition message, however we did work through some examples. Aaron has pointed out one example that is used - each security is sent as a separate message - the SecurityReponseID(tag 322) can be used as the sequence number (not to be confused with MsgSeqNum please) for each response, although this is not needed - as the session layer provides sequencing. As Aaron indicates, the TotNumSecurities(tag 393)should be set to the total number of securities being reported across the entire response and should be unchanging across the messages that make up the response.
>
> In this example if 10 securities were being returned as a result of a request:
>
> Msg N:TotNumSecurities=10, SecurityResponseID=1
> Msg N+1:TotNumSecurities=10, SecurityResponseID=2
> Msg N+2:TotNumSecurities=10, SecurityResponseID=3
> …
> Msg N+9:TotNumSecurities=10, SecurityResponseID=10
>
> Again, I am hesitant to say that SecurityResponseID should be set to a serial number - as this is not necessary - if it helps to provide uniqueness or some other purpose great, knowing that messages will be delivered in sequence by the transport and knowing with each record that you will be receiving 10 total should be enough information. The SecurityResponseID does not have to change or be unique across multiple responses to a request.
>
> Sending one security per security list message (or security definition message in the case of FIX 4.2) can be a bit inefficient in terms of the ratio of information content vs. message overhead. At the CBOE we chose to batch the response into groups in each message. Currently we return 100 securities per message. This 100 is a system configurable value that can be tuned to improve overall throughput.
>
> In the case of batching - the SecurityResponseID does not indicate the security number, but instead a batch number (again the SecurityResponseID just provides a unique identifier for the message, it does not a great deal of value beyond that).
>
> In this case, you have to take the NoRelatedSym(tag 146) into account to determine how many securities from the overall response are contained in the message.
>
> Assume that there are 10 securities per security list message (batch/fragment size) and that there are a total of 99 securities being returned - the messages would be as follows (switching to tag numbers for message size efficiency :)):
>
> Msg N: TotNumSecurities(tag 393)=99, SecurityResponseID(tag 322)=“Today’s Batch#1”,
> NoRelatedSym(tag 146)=10
> Msg N+1: 393=99,322=“Here is your second batch”,146=10
> Msg N+2: 393=99,322=“3rd batch”,146=10
> Msg N+9: 393=99,322=“tenth”,146=9
>
> During message design and review for FIX 4.2 the ECN & Exchanges working group was hesitant to put any qualifications or just one way of fragmenting messages. It seemed like an artificial restriction on implementations. Instead a more generic approach was developed wherein each message should be self standing - i.e. you could take that message and run with it - not needing previous or subsequent responses to be queued before processing. This is accomplished by including the TotNumSecurities on each message.
>
> We also wanted the number of securities to be returned per message to be variable based upon application specific criteria. Implementors can adapt to this by writing their applications so that they use TotNumSecurities on the first response to determine the number of subsequent elements to be returned to allocate a hold space (if needed). Each message can be examined for the NoRelatedSym(tag 146) to determine how many of the total number are being returned on that message. If a non-array data structure is used - this is enough information to process the requests - no intermediate information is required. If an array is used - the number received so far has to be maintained (next element).
>
> So to conclude, now that we have discussed one security per message and a batch of securities per message, please forget both of these delineations and focus on receiving TotNumSecurities back and each message received will have between [ 1 and TotNumSecurites ] securities contained within it. Using this approach - your application should be able to be written genericly to work with multiple implementations.
>
> Hope this helps. If you think we should include something about this in the specification - let us know - we can add it to the FIX 4.4 Changes and track it for the next version of FIX.
>
> Jim
>
> > Oliver
> >
> > Each message fragment will be a new message from the FIX session perspective
> > and will have a unique sequence number.
> >
> > TotalNumSecurities (393) would not be decremented with each message as it refers to the total
> > number of securites in the fragmented block.
> >
> > rgds
> > Aaron
> >
> >
> > > Hi,
> > >
> > > is there are a standard way to fragment a FIX message?
> > > I want to split up a SecurityList message and looked through the FIX 4.3 standard, but I didn’t found anything how this should be done.
> > >
> > > Should all message fragments have the same sequence number?
> > >
> > > Should the field TotalNumSecurities (393) in the SecurityList message be decremented in each fragment?
> > >
> > > Thanks
> > >
> > > Oliver
> > >
> >
>