Persistence

[kaymahes_x]

Imported from previous forum

[kaymahes_x]

Note on Persistence

For the purpose of this whitepaper, persistence should be be defined as using non-volatile memory - either using a DataBase management system or a native file system. “Inmemory persistence” would not be considered persistence because the FIX engine cannot recover state after a crash / unexpected shutdown and restart. All benchmarking should be made with FIX engines using one of these two approaches.

One benchmark would be to compare performance difference between these two approaches keeping all other design factors constant.

[andersfuruhed_x]

What’s your view on tolerance to single faults including site failure that would require the concept of persistence to cover multiple sites?

When looking at persistence requirements and running benchmarks without persistence, with persistence and finally with persistence covering any single failure, it could be worthwhile to also defines availability modes (service interruption, service recovery) when benchmarking.

Anders

Note on Persistence

For the purpose of this whitepaper, persistence should be be defined as
using non-volatile memory - either using a DataBase management system or
a native file system. “Inmemory persistence” would not be considered
persistence because the FIX engine cannot recover state after a crash /
unexpected shutdown and restart. All benchmarking should be made with
FIX engines using one of these two approaches.

One benchmark would be to compare performance difference between these
two approaches keeping all other design factors constant.

[fplpo_x]

[ original email was from Clive Browning - clive.browning@rapidaddition.co.uk ]

You do mention non-volatile memory; to add to you list of database and native file system persistence options we can also consider the use of battery-backed RAM and the adavantages and disadvantages of such an approach.

Note on Persistence

For the purpose of this whitepaper, persistence should be be defined as
using non-volatile memory - either using a DataBase management system or
a native file system. “Inmemory persistence” would not be considered
persistence because the FIX engine cannot recover state after a crash /
unexpected shutdown and restart. All benchmarking should be made with
FIX engines using one of these two approaches.

One benchmark would be to compare performance difference between these
two approaches keeping all other design factors constant.

[fplpo_x]

[ original email was from John Prewett - jprewett@lavatrading.com ]
In my (humble) opinion, the first point in the “persistence” topic is whether the application (including both sides of the FIX session in this) actually require any persistence at all. We certainly have applications communicating using FIX that don’t persist any messages and automatically reset sequences to 1,1 upon disconnect.

For persistence, I was hoping that the following topics could be covered:

1. SQL database
2. Local file system
3. Remote (shared) file system
4. Duplicated file system.
5. None
6. Anything else I haven’t though of such as battery-backed up memory as mentioned earlier.

Certainly not everyone is using applications that are fully fault-tolerant and as such some of those just store the sent and received FIX messages on their local drive. If the machine (or the disk) dies, the FIX session is compromised. Some applications are willing to endure that risk.

Many applications use some sort of shared storage to offer a fault-tolerant FIX session where another machine in the same datacenter can take over in the case of a hardware failure with no loss of messages. Many of these do not handle more drastic fault-tolerance where a disaster strikes the entire datacenter and the backup datacenter has to restart the FIX sessions with 1,1 and suffer the potential loss of messages.

Lastly, the bulletproof ones that offer fault-tolerance inside the datacenter and also include the backup datacenter. If the primary datacenter dies, the backup datacenter can take over with no loss of messages. There appear to be not many corporations that offer this type of service due to potential latencies involved.

So the level of persistence has a profound relationship to the latency of the system. It is our job to give some advice as to how to get the best performance out of the level of persistence that is required by the application.

I hope some of this rambling helps.

[qingjunwei76_x]

In my (humble) opinion, the first point in the “persistence” topic is
whether the application (including both sides of the FIX session in
this) actually require any persistence at all. We certainly have
applications communicating using FIX that don’t persist any messages and
automatically reset sequences to 1,1 upon disconnect.

Fast should better suit those types of applications. If you don’t need sequence number you don’t have to use FIX transport.

For persistence, I was hoping that the following topics could be
covered:

1. SQL database
2. Local file system
3. Remote (shared) file system
4. Duplicated file system.
5. None
6. Anything else I haven’t though of such as battery-backed up memory
   as mentioned earlier.

Certainly not everyone is using applications that are fully fault-
tolerant and as such some of those just store the sent and received FIX
messages on their local drive. If the machine (or the disk) dies, the
FIX session is compromised. Some applications are willing to endure
that risk.

Many applications use some sort of shared storage to offer a fault-
tolerant FIX session where another machine in the same datacenter
can take over in the case of a hardware failure with no loss of
messages. Many of these do not handle more drastic fault-tolerance
where a disaster strikes the entire datacenter and the backup
datacenter has to restart the FIX sessions with 1,1 and suffer the
potential loss of messages.

Lastly, the bulletproof ones that offer fault-tolerance inside the
datacenter and also include the backup datacenter. If the primary
datacenter dies, the backup datacenter can take over with no loss of
messages. There appear to be not many corporations that offer this type
of service due to potential latencies involved.

So the level of persistence has a profound relationship to the latency
of the system. It is our job to give some advice as to how to get the
best performance out of the level of persistence that is required by the
application.

I hope some of this rambling helps.

In my opinion, we have to be practical. I believe a dual server set up with high speed inter-connect network is the best idea. Of course we have to make sure the network device is reliable enough.

As for the “bullet-proof” inter-datacenter fault tolerance, I would appreciate if anybody could enlighten me with any system that ever worked in that way. First of all, the huge latency will just push people away. Secondly, your trading system can not just work alone. We are talking about the entire systems that keeps the business running, including a lot of database server, the remote file system, servers used for other purposes such as heavy calculation of models. If the data-center got hit, even if your FIX engine survives, the state of some other components may still be undetermined. In that case, you should be better off not doing anything at all.

[andersfuruhed_x]

I hope some of this rambling helps.

In my opinion, we have to be practical. I believe a dual server set up
with high speed inter-connect network is the best idea. Of course we
have to make sure the network device is reliable enough.

To support automatic failover and repair, you need an odd number of entities to handle a network split situation to make sure there is no split brain (two primaries) or no primary at all (the system does not have sufficient information to decide that there is not a split brain situation). Traditional clusters use two nodes and a token, other systems use active-active replication and a third voting node, and then you can have triple active systems providing the lowest latency when there is a node or network brownout and the lowest automated time to repair. Persistence (protection from loss of data) is different to fault masking (maintaining the service in the face of a failure). Persistence come in different flavors, from simply making sure that loss of power does not cause loss of data to making sure that a transaction is not completed until it has been stored safely to multiple places (in the same or multiple locations). The same goes for fault masking, where you can have a number of different solutions and configurations of a solution.

As for the “bullet-proof” inter-datacenter fault tolerance, I would
appreciate if anybody could enlighten me with any system that ever
worked in that way. First of all, the huge latency will just push people
away. Secondly, your trading system can not just work alone. We are
talking about the entire systems that keeps the business running,
including a lot of database server, the remote file system, servers used
for other purposes such as heavy calculation of models. If the data-
center got hit, even if your FIX engine survives, the state of some
other components may still be undetermined. In that case, you should be
better off not doing anything at all.

There are trading solutions and FIX engines that do that and I agree that you need to take care of all components in the transaction processing chain or fall flat. The latency issue (speed of light) is valid and as a consequence, there is a compromise to be had. Even within a data center, it can be an option to setup multiple virtual locations that don’t have the same failure mode, though a catastrophic event is more likely to strike all of them than a more latency-inducing alternative.

[qingjunwei76_x]

I hope some of this rambling helps.

In my opinion, we have to be practical. I believe a dual server set up
with high speed inter-connect network is the best idea. Of course we
have to make sure the network device is reliable enough.

To support automatic failover and repair, you need an odd number of
entities to handle a network split situation to make sure there is no
split brain (two primaries) or no primary at all (the system does not
have sufficient information to decide that there is not a split brain
situation). Traditional clusters use two nodes and a token, other
systems use active-active replication and a third voting node, and then
you can have triple active systems providing the lowest latency when
there is a node or network brownout and the lowest automated time to
repair. Persistence (protection from loss of data) is different to
fault masking (maintaining the service in the face of a failure).
Persistence come in different flavors, from simply making sure that
loss of power does not cause loss of data to making sure that a
transaction is not completed until it has been stored safely to
multiple places (in the same or multiple locations). The same goes for
fault masking, where you can have a number of different solutions and
configurations of a solution.

The two nodes design can effectively protect from the loss of data (persistence). Low latency can ensure persistence is performed within transactional context. It can also protect from the system crash caused by hardware failure.

However, system malfunction from hardware failure is even almost impossible to detect. Failover is much easier than disaster recovery. There is no such thing as guaranteed disaster recovery. That’s why we can it a disaster.

As for the “bullet-proof” inter-datacenter fault tolerance, I would
appreciate if anybody could enlighten me with any system that ever
worked in that way. First of all, the huge latency will just push
people away. Secondly, your trading system can not just work alone. We
are talking about the entire systems that keeps the business running,
including a lot of database server, the remote file system, servers
used for other purposes such as heavy calculation of models. If the
data- center got hit, even if your FIX engine survives, the state of
some other components may still be undetermined. In that case, you
should be better off not doing anything at all.

There are trading solutions and FIX engines that do that and I agree
that you need to take care of all components in the transaction
processing chain or fall flat. The latency issue (speed of light) is
valid and as a consequence, there is a compromise to be had. Even within
a data center, it can be an option to setup multiple virtual locations
that don’t have the same failure mode, though a catastrophic event is
more likely to strike all of them than a more latency-inducing
alternative.

For an exchange, that solution might be doable, only theoretically. Because the business logic of an exchange is relatively well defined (which makes it “simpler”). If you want to combat latency, you have to make a comprise to accept a certain level of loss of information from asynchronous inter-datacenter failover. I don’t know which exchange is willing to accept that.

For institutional trading companies, it’s even more difficult to be justified.

[andersfuruhed_x]

I hope some of this rambling helps.

In my opinion, we have to be practical. I believe a dual server set
up with high speed inter-connect network is the best idea. Of course
we have to make sure the network device is reliable enough.

To support automatic failover and repair, you need an odd number of
entities to handle a network split situation to make sure there is no
split brain (two primaries) or no primary at all (the system does not
have sufficient information to decide that there is not a split brain
situation). Traditional clusters use two nodes and a token, other
systems use active-active replication and a third voting node, and
then you can have triple active systems providing the lowest latency
when there is a node or network brownout and the lowest automated time
to repair. Persistence (protection from loss of data) is different to
fault masking (maintaining the service in the face of a failure).
Persistence come in different flavors, from simply making sure that
loss of power does not cause loss of data to making sure that a
transaction is not completed until it has been stored safely to
multiple places (in the same or multiple locations). The same goes for
fault masking, where you can have a number of different solutions and
configurations of a solution.

The two nodes design can effectively protect from the loss of data
(persistence). Low latency can ensure persistence is performed within
transactional context. It can also protect from the system crash caused
by hardware failure.

However, system malfunction from hardware failure is even almost
impossible to detect. Failover is much easier than disaster recovery.
There is no such thing as guaranteed disaster recovery. That’s why we
can it a disaster.

Almost impossible to detext (and fix) - has been done numerous times in different industries such as triple mode redundant systems used in the flight industry.
We do such detection and majority voting in software and it solves a range of problems that we and clients of ours think are relevant to fix. This means that we provide triple active FIX session management state.
Without going into what level of fault tolerance is relevant in what use cases, I think it is relevant to categorize persistence and fault masking modes.

As for the “bullet-proof” inter-datacenter fault tolerance, I would
appreciate if anybody could enlighten me with any system that ever
worked in that way. First of all, the huge latency will just push
people away. Secondly, your trading system can not just work alone.
We are talking about the entire systems that keeps the business
running, including a lot of database server, the remote file system,
servers used for other purposes such as heavy calculation of models.
If the data- center got hit, even if your FIX engine survives, the
state of some other components may still be undetermined. In that
case, you should be better off not doing anything at all.

There are trading solutions and FIX engines that do that and I agree
that you need to take care of all components in the transaction
processing chain or fall flat. The latency issue (speed of light) is
valid and as a consequence, there is a compromise to be had. Even
within a data center, it can be an option to setup multiple virtual
locations that don’t have the same failure mode, though a catastrophic
event is more likely to strike all of them than a more latency-
inducing alternative.

For an exchange, that solution might be doable, only theoretically.
Because the business logic of an exchange is relatively well defined
(which makes it “simpler”). If you want to combat latency, you have to
make a comprise to accept a certain level of loss of information from
asynchronous inter-datacenter failover. I don’t know which exchange is
willing to accept that.

For institutional trading companies, it’s even more difficult to be
justified.

[qingjunwei76_x]

I hope some of this rambling helps.

In my opinion, we have to be practical. I believe a dual server
set up with high speed inter-connect network is the best idea. Of
course we have to make sure the network device is reliable enough.

To support automatic failover and repair, you need an odd number of
entities to handle a network split situation to make sure there is
no split brain (two primaries) or no primary at all (the system does
not have sufficient information to decide that there is not a split
brain situation). Traditional clusters use two nodes and a token,
other systems use active-active replication and a third voting node,
and then you can have triple active systems providing the lowest
latency when there is a node or network brownout and the lowest
automated time to repair. Persistence (protection from loss of data)
is different to fault masking (maintaining the service in the face
of a failure). Persistence come in different flavors, from simply
making sure that loss of power does not cause loss of data to making
sure that a transaction is not completed until it has been stored
safely to multiple places (in the same or multiple locations). The
same goes for fault masking, where you can have a number of
different solutions and configurations of a solution.

The two nodes design can effectively protect from the loss of data
(persistence). Low latency can ensure persistence is performed within
transactional context. It can also protect from the system crash
caused by hardware failure.

However, system malfunction from hardware failure is even almost
impossible to detect. Failover is much easier than disaster recovery.
There is no such thing as guaranteed disaster recovery. That’s why we
can it a disaster.

Almost impossible to detext (and fix) - has been done numerous times in
different industries such as triple mode redundant systems used in the
flight industry. We do such detection and majority voting in software
and it solves a range of problems that we and clients of ours think are
relevant to fix. This means that we provide triple active FIX session
management state. Without going into what level of fault tolerance is
relevant in what use cases, I think it is relevant to categorize
persistence and fault masking modes.

So exactly what kind of vote your system is doing? If two nodes tell you to buy and one node tells to sell then you buy? I’m just kidding.

Triple (sometimes even quadro) redundant is usually used in fighter jets. I agree your system must be extremely robust if one node is shot by a machine gun.

To me fault tolerance is just remote replication of persistence. There must always exist a primary and a (or two, three) backups. Only primary is allowed to make decision and backups are not allowed to vote. I don’t care about how the switching is done because according to you it depends on the use cases.

Software can’t solve all the problem. We can only try our best. Just consider how your system is going to work in case of a single bit error occurs in you system memory? Do you want a “triple voting system” or machines with ECC memory modules? As software designer we are supposed to simplify or generalize the problem. If we can generalize the problem to a binary scenario we don’t have to introduce more levels of fault tolerance. Just like the single bit memory fault example I just mentioned. Let the other people (in that case the hardware engineer) to worry about the problem they are supposed to be responsible for and take it for granted.

As for the “bullet-proof” inter-datacenter fault tolerance, I
would appreciate if anybody could enlighten me with any system
that ever worked in that way. First of all, the huge latency will
just push people away. Secondly, your trading system can not just
work alone. We are talking about the entire systems that keeps the
business running, including a lot of database server, the remote
file system, servers used for other purposes such as heavy
calculation of models. If the data- center got hit, even if your
FIX engine survives, the state of some other components may still
be undetermined. In that case, you should be better off not doing
anything at all.

There are trading solutions and FIX engines that do that and I agree
that you need to take care of all components in the transaction
processing chain or fall flat. The latency issue (speed of light) is
valid and as a consequence, there is a compromise to be had. Even
within a data center, it can be an option to setup multiple virtual
locations that don’t have the same failure mode, though a
catastrophic event is more likely to strike all of them than a more
latency- inducing alternative.

For an exchange, that solution might be doable, only theoretically.
Because the business logic of an exchange is relatively well defined
(which makes it “simpler”). If you want to combat latency, you have to
make a comprise to accept a certain level of loss of information from
asynchronous inter-datacenter failover. I don’t know which exchange is
willing to accept that.

For institutional trading companies, it’s even more difficult to be
justified.

[andersfuruhed_x]

I hope some of this rambling helps.

In my opinion, we have to be practical. I believe a dual server
set up with high speed inter-connect network is the best idea.
Of course we have to make sure the network device is reliable
enough.

To support automatic failover and repair, you need an odd number
of entities to handle a network split situation to make sure there
is no split brain (two primaries) or no primary at all (the system
does not have sufficient information to decide that there is not a
split brain situation). Traditional clusters use two nodes and a
token, other systems use active-active replication and a third
voting node, and then you can have triple active systems providing
the lowest latency when there is a node or network brownout and
the lowest automated time to repair. Persistence (protection from
loss of data) is different to fault masking (maintaining the
service in the face of a failure). Persistence come in different
flavors, from simply making sure that loss of power does not cause
loss of data to making sure that a transaction is not completed
until it has been stored safely to multiple places (in the same or
multiple locations). The same goes for fault masking, where you
can have a number of different solutions and configurations of a
solution.

The two nodes design can effectively protect from the loss of data
(persistence). Low latency can ensure persistence is performed
within transactional context. It can also protect from the system
crash caused by hardware failure.

However, system malfunction from hardware failure is even almost
impossible to detect. Failover is much easier than disaster
recovery. There is no such thing as guaranteed disaster recovery.
That’s why we can it a disaster.

Almost impossible to detext (and fix) - has been done numerous times
in different industries such as triple mode redundant systems used in
the flight industry. We do such detection and majority voting in
software and it solves a range of problems that we and clients of ours
think are relevant to fix. This means that we provide triple active
FIX session management state. Without going into what level of fault
tolerance is relevant in what use cases, I think it is relevant to
categorize persistence and fault masking modes.

So exactly what kind of vote your system is doing? If two nodes tell you
to buy and one node tells to sell then you buy? I’m just kidding.

Yes. I am not kidding.

Triple (sometimes even quadro) redundant is usually used in fighter
jets. I agree your system must be extremely robust if one node is shot
by a machine gun.

To me fault tolerance is just remote replication of persistence. There
must always exist a primary and a (or two, three) backups. Only primary
is allowed to make decision and backups are not allowed to vote. I don’t
care about how the switching is done because according to you it depends
on the use cases.

Software can’t solve all the problem. We can only try our best. Just
consider how your system is going to work in case of a single bit error
occurs in you system memory? Do you want a “triple voting system” or
machines with ECC memory modules? As software designer we are supposed
to simplify or generalize the problem. If we can generalize the problem
to a binary scenario we don’t have to introduce more levels of fault
tolerance. Just like the single bit memory fault example I just
mentioned. Let the other people (in that case the hardware engineer) to
worry about the problem they are supposed to be responsible for and take
it for granted.

If you detect discrepancies, you are also going to detect and handle a number of single bit error problems as well. Please don’t take this discussion to a point where the other members of the group will just stop listening. The categorization of persistence and fault masking models is important to have and it is getting lost in noise.

As for the “bullet-proof” inter-datacenter fault tolerance, I
would appreciate if anybody could enlighten me with any system
that ever worked in that way. First of all, the huge latency
will just push people away. Secondly, your trading system can
not just work alone. We are talking about the entire systems
that keeps the business running, including a lot of database
server, the remote file system, servers used for other purposes
such as heavy calculation of models. If the data- center got
hit, even if your FIX engine survives, the state of some other
components may still be undetermined. In that case, you should
be better off not doing anything at all.

There are trading solutions and FIX engines that do that and I
agree that you need to take care of all components in the
transaction processing chain or fall flat. The latency issue
(speed of light) is valid and as a consequence, there is a
compromise to be had. Even within a data center, it can be an
option to setup multiple virtual locations that don’t have the
same failure mode, though a catastrophic event is more likely to
strike all of them than a more latency- inducing alternative.

For an exchange, that solution might be doable, only theoretically.
Because the business logic of an exchange is relatively well defined
(which makes it “simpler”). If you want to combat latency, you have
to make a comprise to accept a certain level of loss of information
from asynchronous inter-datacenter failover. I don’t know which
exchange is willing to accept that.

For institutional trading companies, it’s even more difficult to be
justified.

[qingjunwei76_x]

I agree we should stop. It’s good to know that you are not kidding.