
Ask HN: How do HFT systems store position and order data? - robinrheem
HFT systems are really sensitive in latency. Which means it can&#x27;t actually use traditional databases like MySQL, PostgreSQL, or etc. But then, how do we actually persist order and position data that the algorithms make? Would we just print it in the screen like application logs? I&#x27;m not sure that will be viable since logs can actually get lost. Maybe write on local files as CSV or HDF5? And perhaps make a cron or something to read those files and persist it into somewhere else? Or is there a solution that runs in memory but persists to disk?
======
pnako
The hot path involves the network card, the CPU, RAM and nothing else. The
trading algorithm will keep various data structures in memory to keep track of
current position and position limits.

Outside of the hot path/thread, you can do pretty much whatever you want. You
typically broadcast your trades on the network for other systems to pick them
(and put that data into SQL databases, typically). The databases used for risk
and compliance, etc. will most likely get their data directly from the
exchange.

The idea is that you don't need to query a central system for every single
trade you want to make. HFT systems have some limits set up at the beginning
of the trading session, telling they can take a position of up to X in
security Y.

~~~
robinrheem
Thank you very much for your kind reply. So does that mean the algorithm,
which is in the hot path will just broadcast via a messaging system like
ZeroMQ or Kafka to another subscriber that’ll just save the order and position
data to the database right? Plus the messaging system will be nonblocking and
just do a fire and forget style.

~~~
pnako
The hot path is the time between receiving market data from the network, and
reacting to it by potentially sending an order. It's typically a thread
constantly polling events from the network card.

In that path the rest of the world does not exist: you are interacting with an
exchange, possibly multiple exchanges, and that's it. The rest of the world
doesn't exist. No ZeroMQ or Kafka or whatever.

After you send an event, you can set a flag or copy some data structures for
another thread of lower priority to do some house keeping. This might include
sending an update on some messaging system, yes. This can be something like
TIBCO RV, or a proprietary equivalent, or perhaps the ones you mentioned.

But the bottom line is: when doing low-latency, high-frequency trading you
don't want to do anything that is not strictly required in the hot path. You
must read the message coming from the market, very quickly decide how you want
to react (i.e. send a new order or update an order, at which price point,
etc.) and that's it. You will in fact even avoid computing too much stuff
here; you would have pre-computed as much stuff as possible to be ready to
very quickly do a lookup and maybe some trivial computation before making your
decision.

