Imported from previous forum
I’m working on an article that will outline a high performance architecture as a layer to K, to handle a K distributed I/O.
This layer, or model, can be developed in C/C++, even Java. This article is aimed particularly towards working with R/T market data.
A working title of this article is: "High Performance, Multiple - Virtual QoS Interfaces for an Input/Output Layer to K Instances".
It’s importance to FIX: K can warehouse Fix messages, and corresponding proprietary order messages. And offer a rapid translation between the two. And provide bookkeeping on Fix msg status.
The 64 bit version of K, is said to be able to parse 100,000 messages per second. It’s imprint is small enough, that K instances can be allocated per groups of FIX clients, or per group of FIX connections, on the same multi-CPU machine. And allocate Fix messages to their appropriate targets. It offers a VPN solution to Fix connections. K is with www.kx.com
My research provides a high speed QoS interface.
If a financial firm would care to budget a project to do this work, for them … please contact me at NicholasFerguson@wingarch.com 917 270 1688. I am currently available.
Some of the issues that this article, using a virtual QoS model, intends to answer:
- How to manage K, on a 32 bit or 64 bit platform, so that it will analyze over 20 GB of Real Time data, on a daily basis… and distribute that analysis, near Real Time.
- How to dynamically manage multiple instances of K; assigning instances of K to various analytical functions;
each instance of K receiving same data sets, or not, in view of point #1, above.
3 How to persist K - analyzed data, in view of point #1 and #2, for further aggregate analysis.
4 How to manage this model, dynamically, through QoS messages.
This model will use a QoS methodology:
QoS Input to K -> Classify data, remark it, police size of data accumulation, insert as K data into K
QoS Output from K -> Classify data, remark it, police size of data accumulation, shape data, insure output of data is fast enough, provide output queues, scheduler that can output desired data formats.
Issues that this model, "Multiple - Virtual QoS Interfaces" will resolve:
- Managing multiple instances of K.
- Scheduling of K instances.
- Dedicating single K threads to its analysis work.
- Size of data that each K will receive for statistical computations.
- Closing of K instances, after its analysis has been completed.
- Persistence of K analyzed data.
- Allowing K scripts to be run against K data.
- Allow dynamic introduction of K scripts to this architecture.
- Allow separate instances of K, to focus on K scripts or to focus on KSQL.
- Manage this distributed model, dynamically, through QoS distributed messages.
- Allow other messages to flow through this model, for additional purposes, such as FIX.
To keep a high speed, these "High Performance Virtual QoS Interfaces" will use K structures
and its code will address these issues:
- Keep data local and temporal to each other
- Avoid temporary variables
- Heavy threading
Thus this model’s functions can be either iterative or recursive over K structures, and keep
its high speed.
Further, this High Performance Virtual QoS architecture will address issues for wire communication:
- Process UDP, with particular labels/addresses, such as Tibco, or MPLS
- Process TCP.
Again, if a firm would care to budget a project to do this work, for them, please contact me at NicholasFerguson@wingarch.com 917 270 1688. I am currently available.
Best regards
Nicholas Ferguson
Mr. Ferguson is a Senior Developer/Architect, C/C++/Java/K and has performed contract work, specifically with market data, and FIX, at some of the major Investment Banks/Brokerage Firms/Commercial Banks/Specialists/Market Makers in NYC. His previous contract, at a NYC Investment Bank, ended 9/2003. He also has his Cisco CCNP/CCDP and is currently working on winning his CCIP. He has been encouraged by DARPA to submit proposals to them.