

Ask HN: Is anyone a sysadmin in high frequency trading? - chuhnk

Hey<p>I was wondering if anyone here is a system administrator for a HFT company. A lot of the time you hear about the code, low latency and high powered machines but I've never heard anything about the system administrator's who piece all of this together, monitor it and keep it running. I'm looking for people who work in this area to shed some light on firstly what it take's to be a sysadmin at that level and secondly to share some insight on the architecture of systems in that area.<p>I really hope there are some out there who can talk about this stuff. And for anyone else who is a sysadmin working on high performance systems please speak up. A bit of knowledge sharing and stories would be awesome. There is so much out there on programming but as a system administrator I want to hear about our side of things.
======
staunch
I was a sysadmin at a company that was extremely connected to some very
important finance people. I can't say anything remotely specific since these
guys are _ridiculously_ litigious and I personally witnessed multiple
lawsuits.

Honestly I didn't find it very interesting or particularly challenging from a
technical perspective. I had experience running big datacenters previously and
it wasn't any more or less difficult. All the normal stuff you need to do to
run hundreds of servers.

You have a bunch of servers running a bunch of strategies, similar to what a
human could do, but faster. The strategies are tuned in real-time by the guy
that created them (usually). He watches them run and either pauses, modifies,
or kills them based on performance.

The strategies are a mix of code and configs and run on a bunch of servers.
You stick the servers as close to the exchange's (or other third-party's)
servers as you can to get as much of a latency edge as possible. We had access
to all kinds of advantages in this regard that normal ccompanies wouldn't get.

The programming challenges were somewhat more interesting, but I wasn't super
involved in that. Things like garbage collection performance tuning (so they
could use higher level languages) and hard core math. But a lot of boring
stuff too, like dealing with hundreds of legacy data feeds (many of which cost
tens of thousands per month).

The very core is just really crafty individuals sitting around all day
thinking of ways of screwing^Wtaking a slice of other people's pie and turning
that into a strategy which can be milked as quickly and for as long as
possible.

~~~
veyron
sysadmin stuff is pretty much standard because the challenging work is
actually done by the strategists / developers. Stuff like processor affinity
magic, which at one time would be covered by sysadmins, are now being covered
by the people who deploy the strategies. I guess there's something to be said
about the extent to which roles have melded (the line between strategist and
developer and sysadmin is not clear, and in some circumstances a single person
fills all roles).

"We had access to all kinds of advantages in this regard that normal
ccompanies wouldn't get." <\-- it costs about 15K a month, soup to nuts, to
get the "advantages"

~~~
staunch
The star at a trading company is the traders, as it should be. Nothing has
changed that or probably ever will. The programmers are just one layer closer
to that than the sysadmins, which tends to make it more interesting.

> _it costs about 15k a month_

That's like saying "a piece of meat costs about $25".

I don't think there was one POP (of dozens) that was the same price as
another. Not to mention the equipment and connections required made the deal
unique every time.

There's more to it than that regardless. I'm not going to get into specifics,
but don't presume to know something you would have no way of knowing.

~~~
veyron
Having bootstrapped and built a successful ultra low latency trading operation
from start to finish, I can assure you that in this space the star is the
technology. In other forms of HFT, such as statistical arbitrage, the traders
are the stars. But my understanding is that the question specifically regarded
low latency trading.

To give an example, a whole cohort of traders used a simple lead-lag signal
between futures and equities. Up until mid 2009, it was possible to
consistently make money merely by waiting for futures to move, sending the
information down from chicago (where the CME sits) to new jersey (most of the
equities exchanges are housed in new jersey) and trading the relevant stocks.
At the time, a popular trade regarded ES (the on-the-run emini s&p 500 futures
contract) and SPY (the s&p 500 depositary receipts, designed to replicate the
daily returns of the S&P 500). Here, everyone and their mothers knew that this
relationship existed, and this relationship was published in various papers,
both academic and professional. In this context, the "trader" who brings the
idea isn't really adding any value to the process. The success of the trade
was wholly determined by the performance of the platform.

In a large HFT, the management cleverly structure the relationships so that
the technology staff is lopped together with support and told that the traders
bring value. This is intentional: there are numerous traders out there now who
still believe they have value-add, when by now all of the tricks have been
spilled.

The quoted estimate is averaged over a year of actual expenses, whereby
relationships with clearing brokers and exchanges were set up (bypassing
external vendors)

"We had access to all kinds of advantages in this regard that normal
ccompanies wouldn't get." <\-- that makes it sound like the tools to actually
do ultra low latency trading are out of the reach of most people, and in
reality they aren't. A competent person can build the business from 100K of
personal savings :)

~~~
Lorcal
Vernon is quite correct here, traders are largely redundant though they still
hold the balance of power in almost cases.IT and quantitative strategies still
add value while operational trading can be almost totally automated. There are
definite opportunities for running trading firms as technology firms.

------
veyron
What is your email?

Let me give a meaningful response: in low latency trading, the value is truly
in the platform. The strategies are simple, and you can pick up a textbook (ie
Aldridge [http://www.amazon.com/High-Frequency-Trading-Practical-
Algor...](http://www.amazon.com/High-Frequency-Trading-Practical-Algorithmic-
Strategies/dp/0470563761)) to learn the strategies employed at that level.
That being said, the traditional "sysadmin" role is oftentimes filled by
people traditionally described as "developers". The roles have melded over the
years.

Hence, the question you are asking really delves into the "alpha" of the
platform. This is something a practicing HFT sysadmin wouldn't want to divulge
(or can't because of the highly litigious nature of the industry -- take
Sergey Aleynikov or Samarth Agarwal as examples of what could happen if people
talk too much)

~~~
HockeyPlayer
Be careful of the Aldridge book he cited, most of the helpful reviews are very
negative, and the gang at nuclearphynance has similarly nasty things to say.

~~~
veyron
I don't think the book is particularly good for what people buy the book for
(learning some strategy that can immediately be applied to make money), but it
was chosen as an example of a book that describes some of the trades. I would
like to emphasize that I wouldn't recommend any of the high frequency trading
books.

------
zmmz
Not a sysadmin at a HFT, but am in personal contact with many of them. The
only thing that I can tell you is that getting any information out of anyone
in the industry is extremely difficult. Every variable (hardware and software)
is considered something that can be turned into a competitive advantage.

To get an idea of the sort of platforms that are dealt with, check out
<http://www.stacresearch.com/>

------
djb_hackernews
My bet is the code is so closely related to the hardware that there isn't a
clear distinction between software developers and sysadmins, they are known as
devops, something that is becoming more and more popular.

I bet any "software developer" at a HFT firm would be able to entertain any of
your questions.

