Imported from previous forum
Hi there,
I have a question regarding proper use of FIX user-defined message types. As I understand it, if certain version of FIX protocol doesn’t contain necessary message type, it can be added as user-defined message with MsgType field starting with “U”.
We are looking at some specification from Citi that claims to support FIX 4.4 and it has two messages defined by Citi, which use “PO” and “PI” as MsgType.
Is it a normal practice? If not, how come one of the big names in this industry does not follow the rules?
Thanks,
Serge
Your understanding is correct, all user-defined messages are to be pre-fixed with “U”, i.e. “UPO” and “UPI” would be fine. A standard FIX engine would probably respond with a Reject message with the field SessionRejectReason set to 11=Invalid MsgType. FIX 4.4 does not support an alternate transport so that the FIX engine must have been tweaked to get such a message through to the application layer.
What is the content of these messages and are they sent to you or expected from you? Are they similar to one of the standard FIX messages or outright custom?
Regards,
Hanno.
Hi there,
I have a question regarding proper use of FIX user-defined message
types. As I understand it, if certain version of FIX protocol doesn’t
contain necessary message type, it can be added as user-defined message
with MsgType field starting with “U”.We are looking at some specification from Citi that claims to support
FIX 4.4 and it has two messages defined by Citi, which use “PO” and “PI”
as MsgType.Is it a normal practice? If not, how come one of the big names in this
industry does not follow the rules?Thanks, Serge
Hello Hanno,
Thank you very much for your explanation. According to the specification for Citi’s FX e-Commerce, the offending messages (“PO”, “PI”) are called “Application Ping” and used to measure round trip, end-to-end latency time between Citi and its counterparties. “PO” is sent by Citi’s servers and we are expected to reply with “PI” immediately.
Regards,
Serge
Your understanding is correct, all user-defined messages are to be pre-
fixed with “U”, i.e. “UPO” and “UPI” would be fine. A standard FIX
engine would probably respond with a Reject message with the field
SessionRejectReason set to 11=Invalid MsgType. FIX 4.4 does not support
an alternate transport so that the FIX engine must have been tweaked to
get such a message through to the application layer.What is the content of these messages and are they sent to you or
expected from you? Are they similar to one of the standard FIX messages
or outright custom?Regards, Hanno.
Hi there,
I have a question regarding proper use of FIX user-defined message
types. As I understand it, if certain version of FIX protocol doesn’t
contain necessary message type, it can be added as user-defined
message with MsgType field starting with “U”.We are looking at some specification from Citi that claims to support
FIX 4.4 and it has two messages defined by Citi, which use “PO” and
“PI” as MsgType.Is it a normal practice? If not, how come one of the big names in this
industry does not follow the rules?Thanks, Serge
Ok, sounds similar to what the test request message followed by a heartbeat message can accomplish. The field TestReqID lets you tie the two together. However, there are no additional application level fields in these messages, that could be a reason for using user-defined messages. Message types sound like “ping” and “pong”. I would prefer people to come forward with extension requests for the standard messages but you can’t force them.
Regards,
Hanno.
Hello Hanno,
Thank you very much for your explanation. According to the specification
for Citi’s FX e-Commerce, the offending messages (“PO”, “PI”) are called
“Application Ping” and used to measure round trip, end-to-end latency
time between Citi and its counterparties. “PO” is sent by Citi’s servers
and we are expected to reply with “PI” immediately.Regards, Serge
Your understanding is correct, all user-defined messages are to be pre-
fixed with “U”, i.e. “UPO” and “UPI” would be fine. A standard FIX
engine would probably respond with a Reject message with the field
SessionRejectReason set to 11=Invalid MsgType. FIX 4.4 does not
support an alternate transport so that the FIX engine must have been
tweaked to get such a message through to the application layer.What is the content of these messages and are they sent to you or
expected from you? Are they similar to one of the standard FIX
messages or outright custom?Regards, Hanno.
Hi there,
I have a question regarding proper use of FIX user-defined message
types. As I understand it, if certain version of FIX protocol
doesn’t contain necessary message type, it can be added as user-
defined message with MsgType field starting with “U”.We are looking at some specification from Citi that claims to
support FIX 4.4 and it has two messages defined by Citi, which use
“PO” and “PI” as MsgType.Is it a normal practice? If not, how come one of the big names in
this industry does not follow the rules?Thanks, Serge
Hanno,
I do not think Test Request - Heartbeat would achieve what Citi is trying to do with Ping In (PI) and Ping Out messages (PO). Test request and Heartbeat is from FIX engine to FIX engine (sent to keep FIX Session alive when there is no other traffic and Heartbeating is not in proper intervals), but from the description in Serge’s post states “used to measure round trip, end-to-end latency time between Citi and its counterparties” by which I understand they are trying to measure the round trip time from Application behind Citi’s FIX Engine to the Application behind counterparty’s FIX engine.
Serge,
The only option I see is to modify the your FIX engine to accept ^35=PI^, forward it to application and the application replies with ^35=PO^ and Citi uses it to measure end-to-end latency.
Regards,
K. Mahesh
Ok, sounds similar to what the test request message followed by a
heartbeat message can accomplish. The field TestReqID lets you tie the
two together. However, there are no additional application level fields
in these messages, that could be a reason for using user-defined
messages. Message types sound like “ping” and “pong”. I would prefer
people to come forward with extension requests for the standard messages
but you can’t force them.Regards, Hanno.
Hello Hanno,
Thank you very much for your explanation. According to the
specification for Citi’s FX e-Commerce, the offending messages (“PO”,
“PI”) are called “Application Ping” and used to measure round trip,
end-to-end latency time between Citi and its counterparties. “PO” is
sent by Citi’s servers and we are expected to reply with “PI”
immediately.Regards, Serge
Your understanding is correct, all user-defined messages are to be
pre- fixed with “U”, i.e. “UPO” and “UPI” would be fine. A standard
FIX engine would probably respond with a Reject message with the
field SessionRejectReason set to 11=Invalid MsgType. FIX 4.4 does
not support an alternate transport so that the FIX engine must have
been tweaked to get such a message through to the application layer.What is the content of these messages and are they sent to you or
expected from you? Are they similar to one of the standard FIX
messages or outright custom?Regards, Hanno.
Hi there,
I have a question regarding proper use of FIX user-defined message
types. As I understand it, if certain version of FIX protocol
doesn’t contain necessary message type, it can be added as user-
defined message with MsgType field starting with “U”.We are looking at some specification from Citi that claims to
support FIX 4.4 and it has two messages defined by Citi, which use
“PO” and “PI” as MsgType.Is it a normal practice? If not, how come one of the big names in
this industry does not follow the rules?Thanks, Serge
I would argue that Citi is not interested in additional time required within the infrastructure of the client behind the FIX engine but wants to establish the time it takes between Citi and the first point of FIX entry at the client that is also comparable across clients. However, this is also speculation on my side. I am always a little concerned when tweaking the FIX engine is the solution to solve a problem with the standard. FIX deserves better…
Hanno,
I do not think Test Request - Heartbeat would achieve what Citi is
trying to do with Ping In (PI) and Ping Out messages (PO). Test request
and Heartbeat is from FIX engine to FIX engine (sent to keep FIX Session
alive when there is no other traffic and Heartbeating is not in proper
intervals), but from the description in Serge’s post states “used to
measure round trip, end-to-end latency time between Citi and its
counterparties” by which I understand they are trying to measure the
round trip time from Application behind Citi’s FIX Engine to the
Application behind counterparty’s FIX engine.Serge,
The only option I see is to modify the your FIX engine to accept
^35=PI^, forward it to application and the application replies with
^35=PO^ and Citi uses it to measure end-to-end latency.Regards,
K. Mahesh
If Citi were only interested in the time it takes between FIX engines, then TestRequest-Heartbeat is the best option.
I agree with that tweaking FIX engine to accept a User defined message which does not start with U is really bad and a User defined message type which do not start with U is clearly a violation of FIX Protocol.
Definetly FIX deserves better…
Is there any other way to measure the “end-to-end” time including the time within both the infrastructures - Buy side and Sell side, I mean the time for a complete round trip
BuySide application - BuySide FIX Engine - FIX Network - SellSide FIX Engine - SellSide application
Since these message types (PI & PO) are not present in FIX.5.0SP1, maybe FPL GTC could consider adding these Message Types to newer versions to provide this functionality since there is no other known method of measuring end-to-end time, what I would call round trip time. And the PI message can be sent by either side - Buy or Sell.
Regarding comparability across different connections / sessions, I think comparing end-to-end times would be useful. Traders would want to know where they get faster Fills for their IOC orders. Brokers would want to know which Buy side would give faster Orders in response to IOIs.
I would argue that Citi is not interested in additional time required
within the infrastructure of the client behind the FIX engine but wants
to establish the time it takes between Citi and the first point of FIX
entry at the client that is also comparable across clients. However,
this is also speculation on my side. I am always a little concerned when
tweaking the FIX engine is the solution to solve a problem with the
standard. FIX deserves better…Hanno,
I do not think Test Request - Heartbeat would achieve what Citi is
trying to do with Ping In (PI) and Ping Out messages (PO). Test
request and Heartbeat is from FIX engine to FIX engine (sent to keep
FIX Session alive when there is no other traffic and Heartbeating is
not in proper intervals), but from the description in Serge’s post
states “used to measure round trip, end-to-end latency time between
Citi and its counterparties” by which I understand they are trying to
measure the round trip time from Application behind Citi’s FIX Engine
to the Application behind counterparty’s FIX engine.Serge,
The only option I see is to modify the your FIX engine to accept
^35=PI^, forward it to application and the application replies with
^35=PO^ and Citi uses it to measure end-to-end latency.Regards,
K. Mahesh
I have to say I am enamored with this concept and see its value.
It is not clear to me if the Network Status Request / Report (MsgType=‘BC’ and MsgType=‘BD’) can’t be used for this purpose within the FIX Protocol now. Or if not maybe they could be extended. It looks like you could use Network Status Request (with a RequestType=1 (Snapshot)) to perform a ping through a network.
It may be better to extend the Application* suite of messages providing an Application Status Request / Application Status Report (using PI and PO for message types is interesting - even though it does violate the current sequential assignment of message identifiers).
This seems like a good topic for continued exploration and development.
If Citi were only interested in the time it takes between FIX engines,
then TestRequest-Heartbeat is the best option.I agree with that tweaking FIX engine to accept a User defined message
which does not start with U is really bad and a User defined message
type which do not start with U is clearly a violation of FIX Protocol.Definetly FIX deserves better…
Is there any other way to measure the “end-to-end” time including the
time within both the infrastructures - Buy side and Sell side, I mean
the time for a complete round tripBuySide application - BuySide FIX Engine - FIX Network - SellSide FIX
Engine - SellSide applicationSince these message types (PI & PO) are not present in FIX.5.0SP1, maybe
FPL GTC could consider adding these Message Types to newer versions to
provide this functionality since there is no other known method of
measuring end-to-end time, what I would call round trip time. And the PI
message can be sent by either side - Buy or Sell.Regarding comparability across different connections / sessions, I think
comparing end-to-end times would be useful. Traders would want to know
where they get faster Fills for their IOC orders. Brokers would want to
know which Buy side would give faster Orders in response to IOIs.I would argue that Citi is not interested in additional time required
within the infrastructure of the client behind the FIX engine but
wants to establish the time it takes between Citi and the first point
of FIX entry at the client that is also comparable across clients.
However, this is also speculation on my side. I am always a little
concerned when tweaking the FIX engine is the solution to solve a
problem with the standard. FIX deserves better…Hanno,
I do not think Test Request - Heartbeat would achieve what Citi is
trying to do with Ping In (PI) and Ping Out messages (PO). Test
request and Heartbeat is from FIX engine to FIX engine (sent to keep
FIX Session alive when there is no other traffic and Heartbeating is
not in proper intervals), but from the description in Serge’s post
states “used to measure round trip, end-to-end latency time between
Citi and its counterparties” by which I understand they are trying
to measure the round trip time from Application behind Citi’s FIX
Engine to the Application behind counterparty’s FIX engine.Serge,
The only option I see is to modify the your FIX engine to accept
^35=PI^, forward it to application and the application replies with
^35=PO^ and Citi uses it to measure end-to-end latency.Regards,
K. Mahesh
If I am rambling or getting off topic here, I apologize. You have hit on a topic that I am passionate about: performance. I have spent the last few years working to perfect the measurement and analysis of trading latency. I can offer the following thoughts:
-
You don’t want to measure timings in your application. System timings can drift significantly under load, so timestamping should be done in hardware synchronized with a GPS or CDMA time source. For example, you can non-invasively watch network traffic at point A and know with a high-resolution of accuracy (sub-microsecond) when messages were transmitted and received at that point. In a ping/pong FIX message exchange, you would have to ensure that the FIX servers at both ends are not under heavy load and both had high quality Stratum 1 clock synchronization.
-
You want to measure the actual application messages you wish to benchmark, not a special set of messages that may ultimately be treated differently than those you care about. In other words, adding new messages may give you not get you the metric you want, just add more traffic.
-
Things can get even more interesting when you consider the network layer. If a router somewhere along the path is congested, there may be dropped packets followed by TCP retries, to which application layer is unaware. So the buy side may see a FIX response-time of 300ms, where the sell-side says it only shows 50ms on their end. By looking at the network and application layers, you can see both the network and application delays. Most of the ink given to low-latency is talking about microsecond level jitter in market-data. The truth is, that latency on the execution side can be much more significant. The TCP retransmit timeout is 240ms in most cases! Because TCP guarantees in-order delivery, subsequent packets are not passed up to the application until the retransmitted packet is received. So a little hiccup that affects a single packet containing an order or exec-report message can affect many other orders in the same time frame.
-
Having rich response-time data can be very valuable. Data mining will reveal a wide range of interesting things. We have spent a lot of time and effort figuring out how to best interpret and visualize the data. For example, we went from peak/avg response-times to percentiles, baselines, and thresholds. A solution should be able to determine what is normal, and when performance degrades. Sophisticated firms can have a smart order router use the information to determine the fastest route possible. With DMA connections to a large number of exchanges/banks/venues, your solution needs to be flexible enough to understand that normal performance for broker A is different than broker B.
Managed service providers are increasingly providing reporting of network and application response-time data to provide their clients transparency and proof of performance SLAs.
Best Regards,
Alex
I have to say I am enamored with this concept and see its value.
It is not clear to me if the Network Status Request / Report
(MsgType=‘BC’ and MsgType=‘BD’) can’t be used for this purpose within
the FIX Protocol now. Or if not maybe they could be extended. It looks
like you could use Network Status Request (with a RequestType=1
(Snapshot)) to perform a ping through a network.It may be better to extend the Application* suite of messages providing
an Application Status Request / Application Status Report (using PI and
PO for message types is interesting - even though it does violate the
current sequential assignment of message identifiers).This seems like a good topic for continued exploration and development.
If Citi were only interested in the time it takes between FIX engines,
then TestRequest-Heartbeat is the best option.I agree with that tweaking FIX engine to accept a User defined message
which does not start with U is really bad and a User defined message
type which do not start with U is clearly a violation of FIX Protocol.Definetly FIX deserves better…
Is there any other way to measure the “end-to-end” time including the
time within both the infrastructures - Buy side and Sell side, I mean
the time for a complete round tripBuySide application - BuySide FIX Engine - FIX Network - SellSide FIX
Engine - SellSide applicationSince these message types (PI & PO) are not present in FIX.5.0SP1,
maybe FPL GTC could consider adding these Message Types to newer
versions to provide this functionality since there is no other known
method of measuring end-to-end time, what I would call round trip
time. And the PI message can be sent by either side - Buy or Sell.Regarding comparability across different connections / sessions, I
think comparing end-to-end times would be useful. Traders would want
to know where they get faster Fills for their IOC orders. Brokers
would want to know which Buy side would give faster Orders in response
to IOIs.I would argue that Citi is not interested in additional time
required within the infrastructure of the client behind the FIX
engine but wants to establish the time it takes between Citi and the
first point of FIX entry at the client that is also comparable
across clients. However, this is also speculation on my side. I am
always a little concerned when tweaking the FIX engine is the
solution to solve a problem with the standard. FIX deserves
better…Hanno,
I do not think Test Request - Heartbeat would achieve what Citi is
trying to do with Ping In (PI) and Ping Out messages (PO). Test
request and Heartbeat is from FIX engine to FIX engine (sent to
keep FIX Session alive when there is no other traffic and
Heartbeating is not in proper intervals), but from the description
in Serge’s post states “used to measure round trip, end-to-end
latency time between Citi and its counterparties” by which I
understand they are trying to measure the round trip time from
Application behind Citi’s FIX Engine to the Application behind
counterparty’s FIX engine.Serge,
The only option I see is to modify the your FIX engine to accept
^35=PI^, forward it to application and the application replies
with ^35=PO^ and Citi uses it to measure end-to-end latency.Regards,
K. Mahesh
Jim,
I agree that PI and PO violates current sequential assignment of message identifiers, but these values appeared to be intuitively easy to remember
PI = Ping Into Counterparty
PO = Ping Out from Counterparty
Alex,
-
There should be a mechanism to measure total round trip time including applications at both ends. I agree system timings would drift significantly under load, but thats the idea - benchmark the round trip time under different conditions (no load, average load & peak load). You are right regarding clock synchronization.
-
Actual messages are generated for business reasons, not for performance benchmarking. The idea of adding PI and PO message types is only for benchmarking performance. Obviously these messages would be treated differently than messages with business value like New Order Single, Execution Report, IOI etc. But these business messages cannot be generated solely for the purpose of measuring performance because if a business message is sent out and trade happens in production, it costs money. These PI and PO messages do not cost money, they only measure round trip time.
-
If the Buy side sees 300 ms and the Sell side sees 50 ms, then its a discrepancy. So both sides can Ping each other and see how the round trip times tally.
-
This is precisely what I am trying to measure. If Broker A is faster than Broker B most of the times, if all other factors like commission are equal, then my Traders would prefer to FIX most orders to the faster Broker A. Smart Order routers can use messages PI / PO to dynamically decide which Broker / DMA to send the Orders over FIX.
Regards,
K. Mahesh
If I am rambling or getting off topic here, I apologize. You have hit on
a topic that I am passionate about: performance. I have spent the last
few years working to perfect the measurement and analysis of trading
latency. I can offer the following thoughts:
You don’t want to measure timings in your application. System timings
can drift significantly under load, so timestamping should be done in
hardware synchronized with a GPS or CDMA time source. For example,
you can non-invasively watch network traffic at point A and know with
a high-resolution of accuracy (sub-microsecond) when messages were
transmitted and received at that point. In a ping/pong FIX message
exchange, you would have to ensure that the FIX servers at both ends
are not under heavy load and both had high quality Stratum 1 clock
synchronization.You want to measure the actual application messages you wish to
benchmark, not a special set of messages that may ultimately be
treated differently than those you care about. In other words, adding
new messages may give you not get you the metric you want, just add
more traffic.Things can get even more interesting when you consider the network
layer. If a router somewhere along the path is congested, there may
be dropped packets followed by TCP retries, to which application
layer is unaware. So the buy side may see a FIX response-time of
300ms, where the sell-side says it only shows 50ms on their end. By
looking at the network and application layers, you can see both the
network and application delays. Most of the ink given to low-latency
is talking about microsecond level jitter in market-data. The truth
is, that latency on the execution side can be much more significant.
The TCP retransmit timeout is 240ms in most cases! Because TCP
guarantees in-order delivery, subsequent packets are not passed up to
the application until the retransmitted packet is received. So a
little hiccup that affects a single packet containing an order or exec-
report message can affect many other orders in the same time frame.Having rich response-time data can be very valuable. Data mining will
reveal a wide range of interesting things. We have spent a lot of
time and effort figuring out how to best interpret and visualize the
data. For example, we went from peak/avg response-times to
percentiles, baselines, and thresholds. A solution should be able to
determine what is normal, and when performance degrades.
Sophisticated firms can have a smart order router use the information
to determine the fastest route possible. With DMA connections to a
large number of exchanges/banks/venues, your solution needs to be
flexible enough to understand that normal performance for broker A is
different than broker B.Managed service providers are increasingly providing reporting of
network and application response-time data to provide their clients
transparency and proof of performance SLAs.Best Regards, Alex
I have to say I am enamored with this concept and see its value.
It is not clear to me if the Network Status Request / Report
(MsgType=‘BC’ and MsgType=‘BD’) can’t be used for this purpose within
the FIX Protocol now. Or if not maybe they could be extended. It looks
like you could use Network Status Request (with a RequestType=1
(Snapshot)) to perform a ping through a network.It may be better to extend the Application* suite of messages
providing an Application Status Request / Application Status Report
(using PI and PO for message types is interesting - even though it
does violate the current sequential assignment of message
identifiers).This seems like a good topic for continued exploration and
development.If Citi were only interested in the time it takes between FIX
engines, then TestRequest-Heartbeat is the best option.I agree with that tweaking FIX engine to accept a User defined
message which does not start with U is really bad and a User defined
message type which do not start with U is clearly a violation of FIX
Protocol.Definetly FIX deserves better…
Is there any other way to measure the “end-to-end” time including
the time within both the infrastructures - Buy side and Sell side, I
mean the time for a complete round tripBuySide application - BuySide FIX Engine - FIX Network - SellSide
FIX Engine - SellSide applicationSince these message types (PI & PO) are not present in FIX.5.0SP1,
maybe FPL GTC could consider adding these Message Types to newer
versions to provide this functionality since there is no other known
method of measuring end-to-end time, what I would call round trip
time. And the PI message can be sent by either side - Buy or Sell.Regarding comparability across different connections / sessions, I
think comparing end-to-end times would be useful. Traders would want
to know where they get faster Fills for their IOC orders. Brokers
would want to know which Buy side would give faster Orders in
response to IOIs.I would argue that Citi is not interested in additional time
required within the infrastructure of the client behind the FIX
engine but wants to establish the time it takes between Citi and
the first point of FIX entry at the client that is also comparable
across clients. However, this is also speculation on my side. I am
always a little concerned when tweaking the FIX engine is the
solution to solve a problem with the standard. FIX deserves
better…Hanno,
I do not think Test Request - Heartbeat would achieve what Citi
is trying to do with Ping In (PI) and Ping Out messages (PO).
Test request and Heartbeat is from FIX engine to FIX engine
(sent to keep FIX Session alive when there is no other traffic
and Heartbeating is not in proper intervals), but from the
description in Serge’s post states “used to measure round trip,
end-to-end latency time between Citi and its counterparties” by
which I understand they are trying to measure the round trip
time from Application behind Citi’s FIX Engine to the
Application behind counterparty’s FIX engine.Serge,
The only option I see is to modify the your FIX engine to accept
^35=PI^, forward it to application and the application replies
with ^35=PO^ and Citi uses it to measure end-to-end latency.Regards,
K. Mahesh
[ original email was from Saatwik Khanna - saatwik.khanna@bofasecurities.com ]
This might not be relevant but if we use a standardized message to track performance end to end it would be great if each “hop” added its own time stamp from the point of origination to the end point where an order would be acknowledged. Clients and their brokers could run these during different load periods to pinpoint which hops induce the greatest latency with minimal system impact.
Jim,
I agree that PI and PO violates current sequential assignment of
message identifiers, but these values appeared to be intuitively easy
to rememberPI = Ping Into Counterparty
PO = Ping Out from Counterparty
Alex,
There should be a mechanism to measure total round trip time
including applications at both ends. I agree system timings would drift
significantly under load, but thats the idea - benchmark the round trip
time under different conditions (no load, average load & peak load). You
are right regarding clock synchronization.Actual messages are generated for business reasons, not for
performance benchmarking. The idea of adding PI and PO message types
is only for benchmarking performance. Obviously these messages would
be treated differently than messages with business value like New
Order Single, Execution Report, IOI etc. But these business messages
cannot be generated solely for the purpose of measuring performance
because if a business message is sent out and trade happens in
production, it costs money. These PI and PO messages do not cost
money, they only measure round trip time.If the Buy side sees 300 ms and the Sell side sees 50 ms, then its a
discrepancy. So both sides can Ping each other and see how the round
trip times tally.This is precisely what I am trying to measure. If Broker A is faster
than Broker B most of the times, if all other factors like commission
are equal, then my Traders would prefer to FIX most orders to the
faster Broker A. Smart Order routers can use messages PI / PO to
dynamically decide which Broker / DMA to send the Orders over FIX.Regards,
K. MaheshIf I am rambling or getting off topic here, I apologize. You have hit
on a topic that I am passionate about: performance. I have spent the
last few years working to perfect the measurement and analysis of
trading latency. I can offer the following thoughts:
You don’t want to measure timings in your application. System
timings can drift significantly under load, so timestamping should
be done in hardware synchronized with a GPS or CDMA time source.
For example, you can non-invasively watch network traffic at point
A and know with a high-resolution of accuracy (sub-microsecond)
when messages were transmitted and received at that point. In a
ping/pong FIX message exchange, you would have to ensure that the
FIX servers at both ends are not under heavy load and both had high
quality Stratum 1 clock synchronization.You want to measure the actual application messages you wish to
benchmark, not a special set of messages that may ultimately be
treated differently than those you care about. In other words,
adding new messages may give you not get you the metric you want,
just add more traffic.Things can get even more interesting when you consider the network
layer. If a router somewhere along the path is congested, there may
be dropped packets followed by TCP retries, to which application
layer is unaware. So the buy side may see a FIX response-time of
300ms, where the sell-side says it only shows 50ms on their end. By
looking at the network and application layers, you can see both the
network and application delays. Most of the ink given to low-
latency is talking about microsecond level jitter in market-data.
The truth is, that latency on the execution side can be much more
significant. The TCP retransmit timeout is 240ms in most cases!
Because TCP guarantees in-order delivery, subsequent packets are
not passed up to the application until the retransmitted packet is
received. So a little hiccup that affects a single packet
containing an order or exec- report message can affect many other
orders in the same time frame.Having rich response-time data can be very valuable. Data mining
will reveal a wide range of interesting things. We have spent a lot
of time and effort figuring out how to best interpret and visualize
the data. For example, we went from peak/avg response-times to
percentiles, baselines, and thresholds. A solution should be able
to determine what is normal, and when performance degrades.
Sophisticated firms can have a smart order router use the
information to determine the fastest route possible. With DMA
connections to a large number of exchanges/banks/venues, your
solution needs to be flexible enough to understand that normal
performance for broker A is different than broker B.Managed service providers are increasingly providing reporting of
network and application response-time data to provide their clients
transparency and proof of performance SLAs.Best Regards, Alex
I have to say I am enamored with this concept and see its value.
It is not clear to me if the Network Status Request / Report
(MsgType=‘BC’ and MsgType=‘BD’) can’t be used for this purpose
within the FIX Protocol now. Or if not maybe they could be extended.
It looks like you could use Network Status Request (with a
RequestType=1 (Snapshot)) to perform a ping through a network.It may be better to extend the Application* suite of messages
providing an Application Status Request / Application Status Report
(using PI and PO for message types is interesting - even though it
does violate the current sequential assignment of message
identifiers).This seems like a good topic for continued exploration and
development.If Citi were only interested in the time it takes between FIX
engines, then TestRequest-Heartbeat is the best option.I agree with that tweaking FIX engine to accept a User defined
message which does not start with U is really bad and a User
defined message type which do not start with U is clearly a
violation of FIX Protocol.Definetly FIX deserves better…
Is there any other way to measure the “end-to-end” time including
the time within both the infrastructures - Buy side and Sell side,
I mean the time for a complete round tripBuySide application - BuySide FIX Engine - FIX Network - SellSide
FIX Engine - SellSide applicationSince these message types (PI & PO) are not present in FIX.5.0SP1,
maybe FPL GTC could consider adding these Message Types to newer
versions to provide this functionality since there is no other
known method of measuring end-to-end time, what I would call round
trip time. And the PI message can be sent by either side - Buy or
Sell.Regarding comparability across different connections / sessions, I
think comparing end-to-end times would be useful. Traders would
want to know where they get faster Fills for their IOC orders.
Brokers would want to know which Buy side would give faster Orders
in response to IOIs.I would argue that Citi is not interested in additional time
required within the infrastructure of the client behind the FIX
engine but wants to establish the time it takes between Citi and
the first point of FIX entry at the client that is also
comparable across clients. However, this is also speculation on
my side. I am always a little concerned when tweaking the FIX
engine is the solution to solve a problem with the standard. FIX
deserves better…Hanno,
I do not think Test Request - Heartbeat would achieve what
Citi is trying to do with Ping In (PI) and Ping Out messages
(PO). Test request and Heartbeat is from FIX engine to FIX
engine (sent to keep FIX Session alive when there is no other
traffic and Heartbeating is not in proper intervals), but from
the description in Serge’s post states “used to measure round
trip, end-to-end latency time between Citi and its
counterparties” by which I understand they are trying to
measure the round trip time from Application behind Citi’s FIX
Engine to the Application behind counterparty’s FIX engine.Serge,
The only option I see is to modify the your FIX engine to
accept ^35=PI^, forward it to application and the application
replies with ^35=PO^ and Citi uses it to measure end-to-end
latency.Regards,
K. Mahesh
I think Alex’s point #2 is significant. If I were a smart order router and I needed to know who’s matching engine was fastest, I would not care so much about the ability of the target application to handle a PI message. I would need metrics on actual orders, cancels and replacements where the recipient is managing an order book. You can imagine how much more effort it takes to process a CxlRpl (especially one that reduces quantity) than it does to simply echo back a PO message. So I would argue that the PO/PI method is ill suited to determine the shortest round-trip acknowledgment time.
I believe a better way would to include a tag in the order, cancel or replace message that can hold an integer. An order-generating system could fill this tag with temporal info in usec granularity. An order-receiving system would be required to echo it back in the subsequent Execution message (just as they do with tag 1). Upon receipt of this Execution message, the order-originating system could compare the current system time to the time echo’d in this tag. This would give them a true measure of the round-trip time.
As for Citi using PO and PI… aren’t they tired of getting all those reject messages!
-Greg
Greg,
what would be the advantage of an extra tag flowing back and forth unchanged (just echoed) compared to the order-generating system storing the temporal info with the order locally and retrieving it when the response comes in? The order has to be retrieved and updated anyway when an ExecutionReport arrives for it. The ExecType then tells you how much processing took place at the other end (e.g. [Pending] New=simple ACK vs. Filled=gone through the matching engine).
Regards,
Hanno.
I think Alex’s point #2 is significant. If I were a smart order router
and I needed to know who’s matching engine was fastest, I would not care
so much about the ability of the target application to handle a PI
message. I would need metrics on actual orders, cancels and replacements
where the recipient is managing an order book. You can imagine how much
more effort it takes to process a CxlRpl (especially one that reduces
quantity) than it does to simply echo back a PO message. So I would
argue that the PO/PI method is ill suited to determine the shortest round-
trip acknowledgment time.I believe a better way would to include a tag in the order, cancel or
replace message that can hold an integer. An order-generating system
could fill this tag with temporal info in usec granularity. An order-
receiving system would be required to echo it back in the subsequent
Execution message (just as they do with tag 1). Upon receipt of this
Execution message, the order-originating system could compare the
current system time to the time echo’d in this tag. This would give them
a true measure of the round-trip time.As for Citi using PO and PI… aren’t they tired of getting all those
reject messages!-Greg
Hi Hanno,
Yes that would work, but I’m concerned about the overhead of storing a timestamp along with my outbound order. If I were to store it locally, either in a database or local disk, then I’d be adding some latency. If I were to just populate a tag with the system time then I could avoid that overhead and also not have to keep state information. Maybe I’m overstating this cost, but that’s the idea.
Perhaps it would be best to know what kind of applications would be interested in these kind of solutions (Algo engines, CEP, smart order routers, etc.) before we proceed any further. It seems thay all have slightly diferent needs.
-Greg
Greg,
what would be the advantage of an extra tag flowing back and forth
unchanged (just echoed) compared to the order-generating system storing
the temporal info with the order locally and retrieving it when the
response comes in? The order has to be retrieved and updated anyway when
an ExecutionReport arrives for it. The ExecType then tells you how much
processing took place at the other end (e.g. [Pending] New=simple ACK
vs. Filled=gone through the matching engine).Regards, Hanno.
I think Alex’s point #2 is significant. If I were a smart order router
and I needed to know who’s matching engine was fastest, I would not
care so much about the ability of the target application to handle a
PI message. I would need metrics on actual orders, cancels and
replacements where the recipient is managing an order book. You can
imagine how much more effort it takes to process a CxlRpl (especially
one that reduces quantity) than it does to simply echo back a PO
message. So I would argue that the PO/PI method is ill suited to
determine the shortest round- trip acknowledgment time.I believe a better way would to include a tag in the order, cancel or
replace message that can hold an integer. An order-generating system
could fill this tag with temporal info in usec granularity. An order-
receiving system would be required to echo it back in the subsequent
Execution message (just as they do with tag 1). Upon receipt of this
Execution message, the order-originating system could compare the
current system time to the time echo’d in this tag. This would give
them a true measure of the round-trip time.As for Citi using PO and PI… aren’t they tired of getting all those
reject messages!-Greg
[ original email was from Ryan Pierce - ryan.pierce@fixprotocol.org ]
I’m also in agreement with Greg regarding Alex’s point #2. Tweaking systems to do well on certain benchmarks can produce misleading numbers. As an example, a system with slow business logic might persuade their engine vendor to implement PI/PO messages in the engine itself, without persistence, resulting in sub-millisecond responses when orders might take orders of magnitude more time. A smart order router using the PI/PO times might route a lot of traffic to such a system based on an incorrect perception of lower latency than competitor’s systems.
Personally, if I were implementing a smart router and needed to use latency to break ties between equal prices, or wanted to route away from slow connections, I’d look at a moving average of actual measured response times, which would be real numbers, not something arbitrary like how long an application can respond to a message requiring no business processing.
If risking order flow on potentially slow connections is a concern, or if a way is needed to test whether a system that suddenly developed high latency is back to normal without risking real trades, then, depending on the trading platform, it might be possible to place orders using dummy exchange symbols like ZVZZT for US equities markets. Granted, some firms might refuse to accept dummy orders in production, or they may be routed to a slower QA system or a faster unloaded production system. But I still think it would be better than the alternative, since at least it would flow through the other party’s business logic.
This also gets to the heart of one of my concerns on performance: firms often boast on order ack times, which can be a largely meaningless number. Order ack times may vary, depending on the closeness of the component generating the ack to the customer, in which case the meaning of the ack is completely different. Does an ack mean that a FIX front end system received the order? Or that it’s in the book of a matching engine? I don’t care if an ack can come back in sub-millisecond time; if I send an IOC and it matches an order on the book, how long it takes for the fill to come back is, in my opinion, far more meaningful.
And then there’s the question of fill rate. Is a system that takes 10 ms to respond, but lifting an order in their book happens 95% of the time any worse than one that takes 500 us but fills 5% of the time?
I believe a better way would to include a tag in the order, cancel or
replace message that can hold an integer. An order-generating system
could fill this tag with temporal info in usec granularity. An order-
receiving system would be required to echo it back in the subsequent
Execution message (just as they do with tag 1). Upon receipt of this
Execution message, the order-originating system could compare the
current system time to the time echo’d in this tag. This would give them
a true measure of the round-trip time.
Here, I have to disagree. I’m concerned that additional requirements on application systems will slow FIX adoption. Even if this were done in SP2, it wouldn’t affect the vast majority of systems that won’t implement SP2 for some time. It adds additional size to messages. Also, it imposes an artificial restriction that the timestamp must be taken and added to the FIX message before it is handed off to the engine, which would then include the sender’s engine latency. As a value-added service, I could easily see a FIX engine taking a microsecond-level timestamp at the moment the message is sent out the door and providing it to the business layer, as well as taking another timestamp when a message response arrives, before any validation or persistence is done, and passing that up to the application. Yes, it requires storing the timestamp locally in the application, but the results would be more accurate.
[ original email was from Ian Hoenisch - ihoenisch@fastesp.com ]
Hi guys,
One point to ponder, if you try to measure at the application layer, you have to make sure you have consistent application performance, otherwise your test results will be skewed by what the CPU is doing at the time (i.e. shared with other apps).
A better approach may be to look at the network level. Tying the outbound message with the return message, tricky, but do-able. Then measure the latency difference between the messages will give you a much more precise external latency measurement.
Ian
I’m also in agreement with Greg regarding Alex’s point #2. Tweaking
systems to do well on certain benchmarks can produce misleading numbers.
As an example, a system with slow business logic might persuade their
engine vendor to implement PI/PO messages in the engine itself, without
persistence, resulting in sub-millisecond responses when orders might
take orders of magnitude more time. A smart order router using the PI/PO
times might route a lot of traffic to such a system based on an
incorrect perception of lower latency than competitor’s systems.Personally, if I were implementing a smart router and needed to use
latency to break ties between equal prices, or wanted to route away from
slow connections, I’d look at a moving average of actual measured
response times, which would be real numbers, not something arbitrary
like how long an application can respond to a message requiring no
business processing.If risking order flow on potentially slow connections is a concern, or
if a way is needed to test whether a system that suddenly developed high
latency is back to normal without risking real trades, then, depending
on the trading platform, it might be possible to place orders using
dummy exchange symbols like ZVZZT for US equities markets. Granted, some
firms might refuse to accept dummy orders in production, or they may be
routed to a slower QA system or a faster unloaded production system. But
I still think it would be better than the alternative, since at least it
would flow through the other party’s business logic.This also gets to the heart of one of my concerns on performance: firms
often boast on order ack times, which can be a largely meaningless
number. Order ack times may vary, depending on the closeness of the
component generating the ack to the customer, in which case the meaning
of the ack is completely different. Does an ack mean that a FIX front
end system received the order? Or that it’s in the book of a matching
engine? I don’t care if an ack can come back in sub-millisecond time; if
I send an IOC and it matches an order on the book, how long it takes for
the fill to come back is, in my opinion, far more meaningful.And then there’s the question of fill rate. Is a system that takes 10 ms
to respond, but lifting an order in their book happens 95% of the time
any worse than one that takes 500 us but fills 5% of the time?I believe a better way would to include a tag in the order, cancel or
replace message that can hold an integer. An order-generating system
could fill this tag with temporal info in usec granularity. An order-
receiving system would be required to echo it back in the subsequent
Execution message (just as they do with tag 1). Upon receipt of this
Execution message, the order-originating system could compare the
current system time to the time echo’d in this tag. This would give
them a true measure of the round-trip time.Here, I have to disagree. I’m concerned that additional requirements
on application systems will slow FIX adoption. Even if this were done
in SP2, it wouldn’t affect the vast majority of systems that won’t
implement SP2 for some time. It adds additional size to messages.
Also, it imposes an artificial restriction that the timestamp must be
taken and added to the FIX message before it is handed off to the
engine, which would then include the sender’s engine latency. As a value-
added service, I could easily see a FIX engine taking a microsecond-
level timestamp at the moment the message is sent out the door and
providing it to the business layer, as well as taking another
timestamp when a message response arrives, before any validation or
persistence is done, and passing that up to the application. Yes, it
requires storing the timestamp locally in the application, but the
results would be more accurate.
[ original email was from Joseph Horowitz - jhorowitz@aegisoft.com ]
I agree with Ian. If a FIX Driver that receives a message would include the time received & the time the response is sent back, to the microsecond, when it acknowledges, processes (e.g. trades) or rejects a message (e.g. order), then we would know the total time used on the far-end as a black box. Subtract this from the total time as recorded at the sender and you can determine the bi-directional in-flight wire time regardless of any clock-drift between the machines. This information is very useful, and it would be additionally nice to see it collected & returned for each hop.
Thanks!
-Joey
Hi guys,
One point to ponder, if you try to measure at the application layer, you
have to make sure you have consistent application performance, otherwise
your test results will be skewed by what the CPU is doing at the time
(i.e. shared with other apps).A better approach may be to look at the network level. Tying the
outbound message with the return message, tricky, but do-able. Then
measure the latency difference between the messages will give you a much
more precise external latency measurement.Ian
What if the counterparty FIX Driver does not provide the time it received a message truly instead puts a time which “makes it look faster”? as a developer, to make my FIX engine and Trading application faster, I populate this new Tag RelatedRequestMessageReceiptTime (1427 since 1426 is the last used tag number in FIX.5.0-SP1) as Sending time (52) of my out message minus 10 milliseconds - so always my system would appear to have 10 ms response time.
Its the same logic like what Ryan stated “a system with slow business logic might persuade their engine vendor to implement PI/PO messages in the engine itself”. There is no end to this kind of cheating.
I agree with Ian. If a FIX Driver that receives a message would include
the time received & the time the response is sent back, to the
microsecond, when it acknowledges, processes (e.g. trades) or rejects a
message (e.g. order), then we would know the total time used on the far-
end as a black box. Subtract this from the total time as recorded at the
sender and you can determine the bi-directional in-flight wire time
regardless of any clock-drift between the machines. This information is
very useful, and it would be additionally nice to see it collected &
returned for each hop.Thanks! -Joey
Hi guys,
One point to ponder, if you try to measure at the application layer,
you have to make sure you have consistent application performance,
otherwise your test results will be skewed by what the CPU is doing at
the time (i.e. shared with other apps).A better approach may be to look at the network level. Tying the
outbound message with the return message, tricky, but do-able. Then
measure the latency difference between the messages will give you a
much more precise external latency measurement.Ian
[ original email was from Joseph Horowitz - jhorowitz@aegisoft.com ]
Mahesh,
This is not a problem since we still consider the full round-trip time as a ‘black box’ value; therefore we know in our EMS when an execution venue or broker route is slow overall (network sniffers can also be used). It will be more than obvious and another route will be chosen.
Thanks!
-Joey
What if the counterparty FIX Driver does not provide the time it
received a message truly instead puts a time which “makes it look
faster”? as a developer, to make my FIX engine and Trading application
faster, I populate this new Tag RelatedRequestMessageReceiptTime (1427
since 1426 is the last used tag number in FIX.5.0-SP1) as Sending time
(52) of my out message minus 10 milliseconds - so always my system would
appear to have 10 ms response time.Its the same logic like what Ryan stated “a system with slow business
logic might persuade their engine vendor to implement PI/PO messages in
the engine itself”. There is no end to this kind of cheating.