FIX in Disaster Recovery Site Scenario

, ,
1

Imported from previous forum

2

Hi,
We are building a trading system that uses FIX 4.2. This system has been successfully deployed at multiple client places who had a cold DR setup.
However we now have a client who is interested in achieving failover in either hot or warm modes. This means that the failover should happen within a maximum of few minutes in case of a disaster occuring at the primary site. The proposed solution ensures that the data between the two sites can have a lag of few seconds. But this could potentially result in sequence number mismatch to few clients when failover happens. How should FIX session behave in this case?

I would appreciate greatly if anyone could provide response to this query or provide pointers to articles/whitepapers/anything else available on the internet.

Thanks,
Manjunath M V
Headstrong Corporation
India

3

[ original email was from Greg Orsini - go@cameronsystems.com ]
> The proposed solution ensures that the

data between the two sites can have a lag of few seconds. But this could
potentially result in sequence number mismatch to few clients when
failover happens. How should FIX session behave in this case?

From your counter-party’s perspective, it will appear that your sequence numbers are being repeated.

For example, say your system has sent seq# 1000 to your counter-party, but your DR site is behind and is only up to 997. At this instant, your primary system fails. You initiate a failover to your DR site. It will initiate a logon at seq# 998 which the counter-party will recognize as duplicate and disconnect your session. Typically, you will retry automatically at #999, and again at #1000 with the same results. When you at last try #1001, the login will succeed.

While this may seem as though you have successfully recovered, you must keep in mind that your counter-party has received messages from you of which you are no longer aware. Thus, perhaps you sent orders messages from #998 to #1000 and you don’t have record of having sent them.

This is the trade-off between risk and performance that you need to make when designing a DR solution.

Greg.

4

[ original email was from KEVIN YOUNG - KYOUNG@YOUNGFE.COM ]
> Hi, We are building a trading system that uses FIX 4.2. This system has

been successfully deployed at multiple client places who had a cold DR
setup. However we now have a client who is interested in achieving
failover in either hot or warm modes. This means that the failover
should happen within a maximum of few minutes in case of a disaster
occuring at the primary site. The proposed solution ensures that the
data between the two sites can have a lag of few seconds. But this could
potentially result in sequence number mismatch to few clients when
failover happens. How should FIX session behave in this case?

Greg describes the situation where your FIX engine is the initiator. If your FIX engine is the acceptor and a failover were to happen, most likely your counterparty will connect with a logon with the inbound MsgSeqNum higher than expected. If you receive a higher than expected sequence number you should reply with a Resend Request (MsgType 2) requesting a gap fill from the initiator. A document called “FIX Session-level Test Cases and Expected Behaviors” is located on the FIX Protocol website at http://www.fixprotocol.org/documents/614/FIX_TestCase_SessionLevel20010710.doc as well as a FIX session layer presentation for those new to FIX at http://www.fixprotocol.org/documents/742/FIX_Session_Layer_rev1.ppt

5

Thanks for the replies, Greg/Kevin.
As kevin stated our trading system is the acceptor. Therefore the real problem is not much with respect to client’s sequence no but with respect to acceptor’s response sequence no.
Imagine that the acceptor has sent responses (possibly order+trade msgs) with RespSeqNo #199 & #200. Now the primary site goes down before the DR site can get these responses. Therefore in addition to the situation Greg has described in his mail, we have an issue on the acceptor side as well. What should the client do when on successful login, the acceptor at DR site starts sending responses with RespSeqNo #199? This is the real problem that we are contemplating.

Possible choices that we have explored so far are:

  1. When the client(initiator, that is) receives a response msg with lower sequence no, it ‘trusts’ the acceptor and carries out ‘undo’ on msgs with higher sequence nos. In the example above, one trade needs to be ‘cancelled’ and and order confirm reversed.
  2. In login response add a custom tag that indicates whether the acceptor is on primary or DR site. On client side, when the login response msg received is from the DR site acceptor, through ‘out-of-band’ mechanism (that is through a different channel) the client application obtains the ReqSeqNo and RespSeqNo as expected by the acceptor, marks all the requests/responses that are outside the range as ‘suspect’.

The above 2 choices can be mixed to obtain variations but the real question is: Is there any recommended practice by FIX group for this situation? Or is there a better solution adopted by any one else?

Thanks,
Manju

Hi, We are building a trading system that uses FIX 4.2. This system
has been successfully deployed at multiple client places who had a
cold DR setup. However we now have a client who is interested in
achieving failover in either hot or warm modes. This means that the
failover should happen within a maximum of few minutes in case of a
disaster occuring at the primary site. The proposed solution
ensures that the data between the two sites can have a lag of few
seconds. But this could potentially result in sequence number
mismatch to few clients when failover happens. How should FIX
session behave in this case?

Greg describes the situation where your FIX engine is the initiator. If
your FIX engine is the acceptor and a failover were to happen, most
likely your counterparty will connect with a logon with the inbound
MsgSeqNum higher than expected. If you receive a higher than expected
sequence number you should reply with a Resend Request (MsgType 2)
requesting a gap fill from the initiator. A document called “FIX Session-
level Test Cases and Expected Behaviors” is located on the FIX Protocol
website at http://www.fixprotocol.org/documents/614/FIX_TestCase_Sessio-
nLevel20010710.doc as well as a FIX session layer presentation for those
new to FIX at
http://www.fixprotocol.org/documents/742/FIX_Session_Layer_rev1.ppt

6

[ original email was from Dennis Wiatzka - dennis.wiatzka@computershare.com ]
Hi Manju,

Our system handles the hot failover by:

  • the “active” always copying outbound messages to the “shadow” before sending it to the remote client
  • the “active” always copying inbound messages to the “shadow” before sending it to the server side system

This way the remote client application should never have a message that the “shadow” does not already have.

After a failure on the “active” side, when the remote client logs in to the “shadow” the remote client’s inbound log either has the same message sequence number or a lower one (retransmission occurs).

The same is true for the “shadow” side’s inbound log. The “shadow” will either have the same inbound sequence number or a lower one (retransmission occurs).

If, after a failover, the remote client has not received a response to a transmitted message (e.g.- new order) from the server side components then it must:

  • send in an order status request
  • if the order status is rejected (i.e.- unknown client order id)
  • send the new order again (with appropriate sequence number NOT original sequence number) with PossDup set to ‘Y’

Hope that gives you some ideas.

Dennis

Thanks for the replies, Greg/Kevin. As kevin stated our trading system
is the acceptor. Therefore the real problem is not much with respect to
client’s sequence no but with respect to acceptor’s response sequence
no. Imagine that the acceptor has sent responses (possibly order+trade
msgs) with RespSeqNo #199 & #200. Now the primary site goes down before
the DR site can get these responses. Therefore in addition to the
situation Greg has described in his mail, we have an issue on the
acceptor side as well. What should the client do when on successful
login, the acceptor at DR site starts sending responses with RespSeqNo
#199? This is the real problem that we are contemplating.

Possible choices that we have explored so far are:

  1. When the client(initiator, that is) receives a response msg with
    lower sequence no, it ‘trusts’ the acceptor and carries out ‘undo’ on
    msgs with higher sequence nos. In the example above, one trade needs
    to be ‘cancelled’ and and order confirm reversed.
  2. In login response add a custom tag that indicates whether the
    acceptor is on primary or DR site. On client side, when the login
    response msg received is from the DR site acceptor, through ‘out-of-
    band’ mechanism (that is through a different channel) the client
    application obtains the ReqSeqNo and RespSeqNo as expected by the
    acceptor, marks all the requests/responses that are outside the range
    as ‘suspect’.

The above 2 choices can be mixed to obtain variations but the real
question is: Is there any recommended practice by FIX group for this
situation? Or is there a better solution adopted by any one else?

Thanks, Manju

Hi, We are building a trading system that uses FIX 4.2. This system
has been successfully deployed at multiple client places who had a
cold DR setup. However we now have a client who is interested in
achieving failover in either hot or warm modes. This means that the
failover should happen within a maximum of few minutes in case of a
disaster occuring at the primary site. The proposed solution
ensures that the data between the two sites can have a lag of few
seconds. But this could potentially result in sequence number
mismatch to few clients when failover happens. How should FIX
session behave in this case?

Greg describes the situation where your FIX engine is the initiator.
If your FIX engine is the acceptor and a failover were to happen, most
likely your counterparty will connect with a logon with the inbound
MsgSeqNum higher than expected. If you receive a higher than expected
sequence number you should reply with a Resend Request (MsgType 2)
requesting a gap fill from the initiator. A document called “FIX Session-
level Test Cases and Expected Behaviors” is located on the FIX
Protocol website at http://www.fixprotocol.org/documents/614/FIX_TestCase_Sessio-
nLevel20010710.doc as well as a FIX session layer presentation for
those new to FIX at
http://www.fixprotocol.org/documents/742/FIX_Session_Layer_rev1.ppt

7

Our system handles the hot failover by:

  • the “active” always copying outbound messages to the “shadow” before
    sending it to the remote client
  • the “active” always copying inbound messages to the “shadow” before
    sending it to the server side system

According to your experience, what are the performance implications of this copying?

If, after a failover, the remote client has not received a response to a
transmitted message (e.g.- new order) from the server side components
then it must:

  • send in an order status request
  • if the order status is rejected (i.e.- unknown client order id)
  • send the new order again (with appropriate sequence number NOT
    original sequence number) with PossDup set to ‘Y’

Most probably, you mean PossResend=Y.

Cheers, Jörg

8

[ original email was from Dennis Wiatzka - dennis.wiatzka@computershare.com ]
>

According to your experience, what are the performance implications of
this copying?

We use a gigabit fiber optic connection between sites. Performance impact is neglible. The latencies of the server side components and the remote client’s order management systems have been the limiting factors; not the FIX gateways.

Most probably, you mean PossResend=Y.

Cheers, Jörg

Whoops! Yes. Thanks for catching that.

9

Thanks for the replies, Dennis/Jorg.

The active/shadow synching mechanism is interesting. This avoids any additional complexity on the client side completely.

Dennis,
You have mentioned that you are using gigabit fiber optic connection btw sites. Doesn’t it have distance limitation? Will this connection work even if the distance btw the two sites is >20-25 KMs?

Regards,
Manju

According to your experience, what are the performance implications of
this copying?

We use a gigabit fiber optic connection between sites. Performance
impact is neglible. The latencies of the server side components and the
remote client’s order management systems have been the limiting factors;
not the FIX gateways.

Most probably, you mean PossResend=Y.

Cheers, Jörg

Whoops! Yes. Thanks for catching that.

10

[ original email was from Dennis Wiatzka - dennis.wiatzka@computershare.com ]
> Thanks for the replies, Dennis/Jorg.

The active/shadow synching mechanism is interesting. This avoids any
additional complexity on the client side completely.

Dennis, You have mentioned that you are using gigabit fiber optic
connection btw sites. Doesn’t it have distance limitation? Will this
connection work even if the distance btw the two sites is >20-25 KMs?

Regards, Manju

Hi Manju,

The connection include additional hops with no impact to the source and very little additional latency when using dedicated networks (as opposed to public domain). Messages will still be delivered once they leave the local subnet.

Finally, it is rare to have the backup sites at extreme remote distances. If the backup is that far from the active, then it will be very far from the client, too. They are usually within the same metropolitan area and, for the very reason you cite above, if hot failover is desired then the hot backup is usually within such geographic limits.

Best regards,

Dennis

11

Hi Manju / Dennis,

I thought to add in some of my ideas as well in this regard. Pre-synchronization of the mirror(backup site) before transmission is the easiest and light-weight for the client. But it is at the extra cost of performance trad-off cos it involves a mirror handshaking + additional complexity on the server side implementer.

Manju, my recommendation would be based on our order rates / latency and the hardware configuration you are using. If you are in the range of few hundreds of orders per sec at latencies in the range of seconds, its fine to go with the mirror handshaking mechanism. On the contrary, if you are in the range of few thousands of orders at latencies in the range of 10s of milli-seconds, this wouldn;t work out efficiently.

The recommended practice as per FIX, is to use “POSSIBLE DUPLICATES”. At the session, you have the facility to tag a message as either of the following two.

  1. POSSIBLE DUPLICATE (Means, we have no track of last sequence, hence re-transmitting from a lower sequence) :-

When tagged as POS-DUP, the receiver is “not” supposed to reject saying its low in sequence. Receiver should use, higher layer logic to determine duplicates. For Ex, can use “Client Order ID” for orders and “Exec ID” for trades.

  1. POSSIBLE RESEND (Means, we completely lost track of sequence numbers, and are transmitting duplicates / new from a determined higher sequence number than that was before the disaster)

This is also same as DUP, but its actually, fresh re-transmission with new sequence numbers.

I recommend using POS-DUP. In your example, there is an elegant way to handle this.

Say the primary sent upto #200 and mirror is only aware of #199. Simple implementation would be for the mirror to transmit the first N number of messages as POS-DUP. That means upto # 199 + N, its POS_DUP.

Again to be more clean, stamp the failover time in the mirror, at the time of failover, and transmit messages out as POS-DUP, if the transact-time is less than the failover time. At the reception of the very first message that bears a transact time greater than or equals failover time, the POS-DUP window ends.

Hope this helps…!! Let me know! :slight_smile:

Rgds,
Thaya.

Thanks for the replies, Dennis/Jorg.

The active/shadow synching mechanism is interesting. This avoids any
additional complexity on the client side completely.

Dennis, You have mentioned that you are using gigabit fiber optic
connection btw sites. Doesn’t it have distance limitation? Will this
connection work even if the distance btw the two sites is >20-25 KMs?

Regards, Manju

Hi Manju,

The connection include additional hops with no impact to the source and
very little additional latency when using dedicated networks (as opposed
to public domain). Messages will still be delivered once they leave the
local subnet.

Finally, it is rare to have the backup sites at extreme remote
distances. If the backup is that far from the active, then it will be
very far from the client, too. They are usually within the same
metropolitan area and, for the very reason you cite above, if hot
failover is desired then the hot backup is usually within such
geographic limits.

Best regards,

Dennis