I think I actually saw these folks present at JavaOne a couple years ago? Either that or there's more than one shop branding itself as "HFT" that uses Java.
I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants.
Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of trades and vice-versa there's barely any point in being ultra-fast in the "non-speed-critical" bucket because your alphas last for minutes to hours.
Just from this read, I feel like these folks are just a hair to the right of "fast" in the [fast]---------[smart] continuum. I mostly make this appraisal based on these paragraphs:
>To gain those few crucial microseconds, most players invest in expensive hardware: pools of servers with overclocked liquid-cooled CPUs (in 2020 you can buy a server with 56 cores at 5.6 GHz and 1 TB RAM), collocation in major exchange datacentres, high-end nanosecond network switches, dedicated sub-oceanic lines (Hibernian Express is a major provider), even microwave networks.
>It’s common to see highly customised Linux kernels with OS bypass so that the data “jumps” directly from the network card to the application, IPC (Interprocess communication) and even FPGAs (programmable single-purpose chips).
That's nice but that's where the cutting edge of the speed game was in 2007ish. Everything mentioned here is table stakes at this point (colocation, dedicated fiber, expensive switches, bypassing the kernel in network code, etc). The fact that "even FPGAs" is listed as "even" is the biggest thing I focus on. FPGA's and/or custom silicon is where the speed game is right now. Similarly, "even microwave networks" is also table stakes at this point (you can get on nasdaq's wireless[0] just by paying).
This is the kind of game where capex for technology is dwarfed by the margin you're slinging around every day in trading, so you see some pretty absurd hardware justified.
I discovered something amazing when working with some people who were writing HFT software.
Why do you need 1TB of RAM in these machines? Because when you're Java based, you want to avoid stop-the-world GC pauses. These trading systems only have to be up from 9:30AM-4:30PM EST, so they simply disable GC altogether! At the end of a trading day, restart the app or reboot the system.
Speaking from experience with JVM HFT applications (we used Scala).
There are a lot of tricks though to not require 1TB.
And allocation in general is a bad idea even if you don't collect because it scatters stuff all over memory and messes up cache locality. You really, really don't want to allocate in a performance sensitive jvm application if you can avoid it. It's the opposite of a lot of what I was told and taught (e.g. never do object pooling), but empirically, in my experience, allocations are the biggest slowdown. You can get an application a lot faster just by opening up the memory allocation tab in a jmc flightrecording and refactoring the biggest allocators, usually there is a lot of easy to optimize low hanging fruit that will give good performance improvements, even better than focusing on hot spots in code (in my personal experience).
By far the biggest allocator in trading is going to be marketdata and calculations on it. For reading marketdata from the exchange it's best to leave raw data in memory and access it with a ByteBuffer / sun.misc.unsafe. Under this pattern classes have 1 value, the memory address to pass into sun.misc.unsafe, then everything from there on is done with offsets onto that address. For calculations it's better to write things as static functions, or use object pooling.
In the course of optimizing a trading engine I wrote lots and lots of code to get allocations down to zero. It's definitely doable, but best done from the start, I refactored an existing trading engine to do that, it was not very fun.
> 1TB of RAM is cheap though, versus spending engineering hours.
It's not about the amount of RAM, it's how fast and predictable the overall system is. Note in particular the remark about cache locality; that can be the difference between nanoseconds and microseconds.
What the GP is referring to (Misc.Unsafe) is basically going back to manual pointer reads/writes (IE writing more or less C code in Java), the benefit is you get C speed/memory layout/cache-locality but with the downside of writing it in a less suited language (Java).
HOWEVER If you DO _allocate_ much then 1TB seems like a great choice since GC pauses are killers w.r.t. latency in comparison to cache issues. A cache line (often 64 bytes) would probably only hold at most a couple of Java objects (minimum is probably like 16 or more bytes for each small object) so cache locality won't be improved much by a GC (Yes, the G1 GC in newer JDK's does neighbour compacting iirc so you get a little cache locality but only if the patterns were bad from the start, but avoiding GC entirely is better)
Probably because they started out in Java and don't want a full rewrite once they've gotten this far (might be that they don't have many C++ devs anyhow). I remember working on similar codebases 10 years back when J2ME games were still a thing (J2ME runtimes usually had horrible GC's and some people really went overboard in trying to avoid them) and it wasn't that fun at all (Even if the questDB codebase that is linked elsewhere in this thread seems to be saner).
Also JNI for calling C/C++ code is fairly bad and error prone (if you want to combine code), JNA is better but i think this is one of the bigger reasons that Oracle is funding GraalVM is that it promises "seamless" interoperability and breaking up things into Java and C/C++ parts might be a good option in the future.
JNI replacement is called Panama it doesn't have anything to do with GraalVM.
GraalVM is the new name of MaximeVM one of the JVM meta-circular JVMs. Other well known ones being JikesRVM and SquawkVM.
What Oracle and others in the Java community are doing is reducing the need to drop down to C with support for value types and more fine grained control over native memory.
Java 16 will have the first preview release of the new low level memory APIs.
The meta-circularity is part of the impl details, cross-language calling (Java/JS/LLVM) seems to be the main selling point of it, so if you want GC-less logic then Clang->LLVM then running on the JVM is an option, no idea if it also simplifies native bindings though (that is more of a Panama issue and improving that is a good idea aswell).
Ha, these are many of the same things one might do to make a chess engine fast on the JVM. This only strengthens my belief that the best computer science education you can get is studying chess programming.
I had a side project of 'flattening' classes at classloading time into large mappedbytebuffers, removing all allocations needed for ser-des and recursing on belongs-to references, copying buffer slices from sockets to the buffers (and reusing the socket buffers), and using netty for sockets (selector allocates crazy). Very fun project with javassist...
Do you know of any public examples of how this sort of Java looks? I imagine you lose out on being able to take advantage of much of the JVM ecosystem and I struggle to see what using Java even adds anymore.
here is one: https://github.com/questdb/questdb. Disclaimer, I work on this project. Main reason we use Java is speed of development (which increased with amount of base libraries written) and ease of testing.
Would you do it again? Coming from C/C++ first, Java then and coming to C# it just feels so much more pragmatic with f.ex. struct(s), slices and stackalloc (and unsafe blocks with pointers in a pinch) allowing for GC less programming w/o resorting to turning pointers to integers and using function calls for memory access all around. (Noticed that you do i guess query compilation via the ASM toolkit?)
It is a good question. I feel there is a happy medium where boilerplate is in Java and more intricate data processing routines are in C++. Makes both worlds simpler. I like C++ better. Those things you mentioned - Java truly sucks at indeed, but you don't always need them. When it comes to IDE, testing, compilation speed, cross platform code and finding talent - Java is way easier than C++.
Yeah i see that outlook, my comment was actually mostly about C# (been using it for the past year and these things above are in it as well as all the base being Java-ish)
RAM is not the only aspect of a Garbage Collector....
Over a decade ago, we used a third-party pre-trade risk system that was implemented in Java. Since it was a "service" (we connected to a TCP port), the underlying tricks they used to make it "fast" were transparent to us... until it was not.
They highly tuned their GC to where there was seldom any GC. One day, the third-party made a change to a supervisory service to generate more periodic monitoring emails. The file handles from this actor were apparently not GC'd and held by the process until the system ran out of file handles. That made the service stop working properly.
But, in addition to alerting, that service had a more important job: it was a post-trade risk "watchdog" to the pre-trade risk gateways, which we sharded across.
The various pre-trade risk gateways, upon not hearing from the watchdog authority then 1) began cancelling outstanding orders and 2) not allow new orders. We saw this happening haphazardly over 10 minutes and had little time and capability to recover; this also happened near the end of the trading day when some orders (MOC, LOC) are not cancellable. This was across our whole organization so it affected many strategies.
So it wasn't as simple as "stop all" and although we had many checks and recovery procedures, this was a pretty special Chaos Monkey.
We ended up with a >$1B basket of random stocks that cost >$10M to liquidate over the next few days.
$10M evaporating in 10 minutes because of "GC optimizations" and poorly considered OS settings.
The #1 risk in algo/HFT trading is not financial, but operational.
[Some of the technical details may be slightly off, as it was third-party; my outlook is pieced together from post-mortems. Also, no other party (e.g. broker, SIPC, market participant, the third-party) was financially affected besides our firm.]
Even if you completely disable GC, Java is still allocating tons of ephemeral objects on the heap. Which in turn is leading to expensive and unpredictable page faults. In contrast, C++'s default is to allocate objects on the heap unless you knowingly call new/malloc.
Of course, it's possible to write Java in such a way to minimize this type of heap-thrashing. But by that point, you're already doing the equivalent of C++'s manual memory book-keeping anyway. It's not like "just provision a ton of memory and turn off GC" is a free lunch.
If you add -XX:+AlwaysPreTouch to the JVM arguments, the JVM will pre-touch the entirety of the heap at startup time to avoid unpredictable page faults through the life of the application.
I'd imagine other HFT companies may also pay for the Azul JVM which goes even further and comes with a kernel module that the JVM coordinates with for memory allocation. The kernel module pre-reserves x% of system memory at the time it is loaded.
It is commercial, and sadly their GC algorithms are patented. I'm really anxious for those patents to start running out because the functionality is quite clever.
Basically one main reason you need to stop the world in common GC's right now (and thus cause pauses) is because if you want to compact memory a GC thread has no idea if other threads reads/writes to an object while moving leading to worst case corrupt/lost writes.
What Azul does is to make the pages in question invalid for reading/writing so any access to them would page-fault and then proceeds to move the objects without worrying about other threads. Since the hardware provides the page-fault "for-free" this protection doesn't cost anything in terms of runtime performance in the common case (compared to the new ZGC made by Oracle that has read-write barriers that steals some mutator performance), worst case if another thread tries to read/write memory that is being moved a specialized page-fault handler detects if this was an page that was in motion and can then do a slower read-write but ONLY in the seldom cases where this occurs (compared to all the time of f.ex. ZGC)
Page-faults aren't for free (actually kinda expensive) but rather the paging functionality is giving "free" read/write-barriers instead of those being instruction sequences being inside the main program (ie read/write barriers are handled as page-faults by the paging system that the OS already provides to avoid requiring every read/write of the user-code to be aware of moving pointers).
Scalar replacement has several requirements, and escape analysis is only part of the story. One of the restrictions with scalar replacement for arrays is that indexing has to be constant at JIT time, which your example clearly isn't.
The point would be that escape analysis could have proven that the array was not used outside of the function and allocated it fully on the stack, thus not providing any GC pressure. Scalar replacement isn't the point of interest really.
Which version of Java are you using? And what is Rnd? The class included with JDK is Random. We rely on escape analysis to elide such object creation for us and it works reasonably well. If something this trivial doesn't work for you, file bug with them. We have had success with that as well.
This ran on Java 11. This isn't so much of an issue for us. We are trying to avoid allocations, even as trivial as those. There are other examples that allocated where they should not, for example this lambda will allocate.
() -> System.out.println(1)
I lost hope in escape analysis quite frankly.
Rnd is something I have written because Java's Random is slow and clunky.
The JVM does scalar replacement, not stack allocation. The former is much more limited in functionality, and much more finicky. You also have about zero control over it, compared to, say, C# structs.
Last-drop latency improvements. For ultra-latency-sensitive applications, where developers are conscious about memory allocations and know the application memory footprint exactly, or even have (almost) completely garbage-free applications, accepting the GC cycle might be a design issue. There are also cases when restarting the JVM -- letting load balancers figure out failover -- is sometimes a better recovery strategy than accepting a GC cycle. In those applications, long GC cycle may be considered the wrong thing to do, because that prolongs the detection of the failure, and ultimately delays recovery.
So essentially you need to know the memory footprint of your applications. To err on the side of caution just get as much memory that is available in the market.
At this point like another reply mentions here aren't you just better off writing C++ code ?
> * I guess what you mean is the no-op garbage collector which is available in Java 11*
I think I heard of the same pattern before 2018, so I guess there are other ways to do it. Perhaps Java has some option like memory usage value at which the GC is run.
Depends if your returns are based on writing faster code or shipping correct code faster. Once the advantages of the former are eliminated, optimize for the latter.
Game devs face similar challenges, especially on phones. Though they have to be less radical in their solutions—which afaik usually boil down to ‘preload the level into memory, use object pools, and don't do any new allocations’.
Malloc is usually slower than GC-based allocation (which basically just increments a pointer). Of course, one can emulate this via custom allocators in C++. My guess is that Java development is just easier and quicker, and the JIT may even result in more effective optimizations than AOT compilation.
In C++ you can link with your own version of malloc (one which just returns the next consecutive block, and can be implemented simply by adding the allocated size to a pointer).
You generally choose the language which has the libraries and ecosystem which solves your problem. For instance, you'd be silly to use anything but Java to work with Hadoop. This is the pragmatic choice.
But as soon as you make use of almost any library from the ecosystem, you get allocations. Doesn't that defeat the purpose? Why not isolate the Java-using part in a separate process from the non-allocating fast part?
There are still a number of players in the HFT space that use JVM languages.
I think as with any space, if you really zoom into the details, you see a lot of diversity. You're obviously right in that people who are still hitting the CPU can't compete with people that do everything on FPGA, but it seems like there's still plenty of money to be made by people who are just a bit less fast than that.
I've heard people argue that they prefer being in that sort of space, too, as it gives them a bit more room to compete on their own alphas, and tends to be a bit less winner-take-all.
Yeah absolutely. There's money to be made across the spectrum of fast and smart. My experience is just way on the left side, where the special sauce is only barely statistics-y but extremely technical (both from a software/hardware standpoint and market microstructure saviiness standpoint).
I'm not trying to gatekeep who can call themselves HFTs or not. The main thing I find funny is if you ask 10 firms if they're HFT or quant shops, it will probably not actually line up all that well with exactly how many orders they send or how speed-sensitive they are.
You're making good points, but one thing I want to emphasize is that latency is not the sole dimension of competition in HFT space.
Certain things may just be "table-stakes", but for many strategies table-stakes is the only requirement. Simply being "fast enough" may be fine if you have a smarter model, exploit niche opportunities overlooked by others, or are willing to shoulder certain risks that other HFTs are trying to offload.
To take this to the extreme, look at Renaissance's Medallion fund. It's certainly the case that much of what they're doing is "HFT-ish" in the sense that they're executing high turnover, high Sharpe strategies with short holding periods. Yet they've managed to continue to be successful from the 90s well until the age of HFT, without competing very hard on latency at all.
Markets are an ecosystem, and like biological ecosystems there isn't just one trait that predicts dominance.
correct, there is a peculiar nuance on financial trading topics where people conflate "computer executed trade" with "high frequency trading"
you may need to rapidly adjust the expected fill price at a high frequency on a multi leg strategy being run in 100 positions, but the actual sending orders to the exchange doesn't need to be high frequency, and it is an important distinction that you wouldn't be competing with others on the frequency at this stage, it doesn't matter if this occurs in 1 millisecond of 500 milliseconds, even a couple of seconds. Very different ballgame than the wishful femtosecond game.
its probably better that people don't understand that, but it is annoying that there are this many roadblocks to a nuanced conversation
either way your servers still have to have all the authentication code, and various algorithms running to monitor the tape.
That's right, the more to the fast end you are, the more your strategy is obvious. I worked at an HFT where it was literally buy-here-sell-there for one of the strategies, and of course you can't do that without being ridiculously fast.
DSquare is reasonably famous in London, somehow everyone knows them. They are on the smarter end of things, because the founders were plugged into the early development of the electronic FX market at the beginning of this century.
There's a sort of engineering-vs-business culture thing to this as well, but I think it blurs over time.
There is no singular definition of HFT. The extreme end of low latency is really only applicable for market making systems, but there are plenty of systems in the HFT sphere that do not rely on market making latencies. For example, those microwave links you're talking about? Those are specifically for cross exchange arbitrage. In that domain acceptable latencies go up quite a bit, and general purpose computing is still quite competitive.
Nice overview. The fast vs smart comparison is apt. But something that gets missed in these conversations is what trading system architectures actually look like. The part where latency is measured in nanoseconds is just the tip of the iceberg. Java and 2007 tech is absolutely acceptable for most of the architecture.
Another thing is the structure of Equity and Derivative markets vs FX. The former is fairly standardized. You'll need infrastructure at a handful of exchanges in 2 or 3 cities. Microwave networks and FPGA's are the norm.
But for FX markets, where these guys trade, the picture is much more messy. There are countless places to trade. Co-location means different things at all of them. The entire structure is much less regulated and understood. As a result, strategies and trading system architecture looks much different.
> Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of trades and vice-versa there's barely any point in being ultra-fast in the "non-speed-critical" bucket because your alphas last for minutes to hours.
Depending on the overall strategy, you can be smart and fast at once. If you have occasional alpha harvesting opportunities that must be acted on quickly (very low latency but also low-ish frequency), then it is possible to spend the time in between trades modeling market context and developing optimal short-term plans for how to react in case of various market triggers.
I think he means smart as in "how long your model prediction takes". If your neural net (haven't actually met anyone who uses these in trading) takes 5ms to make a prediction that'll lock you out of a whole lot of trading opportunities/strategies.
Speed always matters, no matter where on the smartness spectrum you are, but it's relative. If your model prediction takes 5ms you're not getting much ROI out of investing $1M into shaving off 50ns in your data processing. But if your end-to-end latency without prediction is 1ms, you better invest in getting that down.
> If your neural net (haven't actually met anyone who uses these in trading) takes 5ms to make a prediction that'll lock you out of a whole lot of trading opportunities/strategies.
Let's say you have trading opportunities once every 100ms. They need to be acted upon within 2ms or they vanish.
You don't have the time budget to run a NNet every time the state of the market changes. You can, however, train a NNet to output a very small decision tree that can run in under 1ms. The NNet can then decide which "micro-strategy" in the form of a much faster and more reactive decision tree is more appropriate for the current market context.
Flexibility in code is important. When markets change it is good to be able to quickly rework your code. Sometimes depending on the market this is more important than pure speed.
Market data is the slow monster - especially in the options market - not order entry. This is where the fpgas come in to help.
Often I have seen trading groups throw money at infrastructure when it was really just that a competitor (a virtu or citadel) has a better way of internalizing order flow or managing risk so they can pay more for the same trade.
PHP or Perl are called interpreted because the interpreter (installed on the destination machine) compiles each line of code as it goes.
Yeah, that hasn't been true of most "interpreted languages" in decades. The most common interpreted languages that I can think of where you parse as you go are shells like bash.
Languages like Perl and PHP are called interpreted because an interpreter runs the script. But the interpreter actually compiles the code first and then runs off of the compiled representation. Which is byte code - just like the JVM. If you put energy into it, you can use all the same techniques that the JVM does. As a practical example, both nodejs and pypy (a fast variant on Python) have a JIT. Julia regularly hits speeds for numerical calculations that is fully competitive with C++ and FORTRAN.
That is amazing. Bash doesn't even load the script into memory? It just keeps a line pointer and chunks through the open file? There's something comforting about that. :D
On a common PHP setup (with OpCache today, any predecessor in the past) this happens only once on startup, thus is neglectible. (Java's VM and JIT are faster, no doubt, but where the translation happens matters only a little ... especially since Java does optimisations and JITting on the VM at run time only as well)
It argues that it operates as an interpreter, in exactly the way that I described.
From your link, The Zend Engine compiles PHP source code on-the-fly into an internal format that it can execute, thus it works as an interpreter. In other works at startup itreads code, compiles to an internal form of byte code, and from then on it executes that. So it works as an interpreter (you give it code, it runs). But internally it does not go line by line as you go.
As a simple demonstration, a syntax error on line 10 of a PHP script will keep it from executing code on line 5. By contrast a syntax error on line 10 of a Bash script will not keep it from having executed line 5 first.
I'm surprised this article doesn't talk about the common issue with writing a trading bot in Java, which is the floating point types. I worked on a trading bot in Java over a decade ago when it was common in the industry to do so. We wrote a fixed precision decimal library which was about 4x faster than the alternative BigDecimal type. It worked out quite well, but Java doesn't allow operator overloading, so the whole code base looked like a mess with tons of a.mul(b.add(c)) looking lines of code.
When C++14 came out the project was ported to C++. It needed to go fast, but more importantly it needed to be stable and readable. An error in code could cost us, which wasn't something we could accept. Moving to modern C++ wasn't fun, but the code is clean, easy to read, works and works well.
Today, the code base has started to be ported to Rust, but unlike going from Java to C++, this move isn't as big of a deal, but the added safety is appreciated given the nature of the project.
Today it's pretty common for firms to create their own lisp like or Haskell like programming language. By having it FPP it can mirror mathematics a bit closer which reduces bugs, yet by having it in house it can be fast, otherwise these companies would use a common language. We are the only ones I know of moving towards Rust. I imagine a lot of firms who are aready C++ are going to stay with modern C++.
From Java to C++ it was more about messing up the logic, because it was so difficult to read. This was a while ago, but there was a Perl script where you could write a kind of psuedo code and it would convert it into the Java syntax, so we could look at the math behind it in a clear way.
>Does rust buy much for HFT?
Going from C++ to Rust is low priority. It may not ever be finished. That low of a priority. It's more about refreshing the code base.
One thing that surprised me is Rust can be faster than C++ at certain tasks.
The big benefit is if you have a new dev coming on and they try to modify something that would introduce a bug, the compiler is all over it. Currently acceptance tests are all over it and work fine and C++ can be pretty strict so it works, but Rust is more ideal for this.
One thing to keep in mind, is that HFT is not just "one thing". It's a collection of many different strategies that are generally characterized by short holding periods, high portfolio turnover, and latency sensitivity. But depending on the specific strategy, the degree of latency sensitivity could vary by orders of magnitude.
A desk trying to gain first queue position at the opening auction is probably running in FPGAs or ASICs and aiming for 100 nanoseconds. A spot market maker needs to respond fast to new level formation, and is aiming for 10 microseconds on a C++ stack. An alpha-driven liquidity taker may be using complex ML models, and is running a Scala stack at 50 microseconds. An ETF market may just need to re-mark their quotes when the underlying shifts, and is fine with 250 microseconds on OCaml.
A (relatively small) subset of HFT is purely latency driven. But the largest sub-category is probably market making. I.e. leaving open a buy order for $0.10 and a sell order for $0.11, and hoping that both get filled and you net the difference.
You're taking real risk, in the sense that your buy order could get filled, your sell order doesn't, and the market moves down, so now you lose money. Latency is pretty important to market making for two reasons. One is because you want to quickly adjust your quotes before the market moves through you. If you're too slow, opportunists will pick you off in the wrong direction. Two is because exchanges mostly follow price-time priority. E.g. if me and you both enter buy orders for $0.10, the front of the "queue" to get matched, will be whoever's order arrives first at the gateway. Not only is that important for getting more opportunities, but generally orders in the front of the queue are less likely to be trade against "toxic flow", where the market moves through you.
There's definitely a baseline minimum of latency that you need to play the game at all. But there's tons of ways to compete besides just being the fastest. You can build your own signals to internally predict which way the market moves before it does. You can have superior models to tell you when the risk of "toxic flow" is lower. You can use clever tricks to get to the front of the queue when others aren't paying attention.
I'm a HFT quant trader and HFT is not purely latency driven. You don't want to be slow, but most firms are increasingly perusing longer and longer term strategies that aren't as latency sensitive. Pure risk free latency arb isn't really a profit center anymore. The speed wars are pretty much over.
The industry has consolidated and any player that didn't adapt to the new, more cerebral game is dead.
No one "won" the speed wars: the top firms are all competitive with each other and have a gentleman's agreement against spending billions more for nanosecond gains.
I've used Java and C++ in a few low latency applications and I do prefer Java in certain scenarios. The whole ecosystem around Java makes rapid development easier meaning we could make safe changes quicker which is a big benefit in trading systems where you need to react to unpredictable market conditions.
The article focuses quite heavily on Zing vs Hotspot but it'd be interesting to see an analysis of a variety of the standard JVMs GC methods (namely Shenandoah).
For anyone interested in low latency Java I'd recommend watching some of Martin Thompson's talks on building LMAX and his blog Mechanical Sympathy is a great start too.
>The article focuses quite heavily on Zing vs Hotspot but it'd be interesting to see an analysis of a variety of the standard JVMs GC methods (namely Shenandoah).
Also, ZGC, which is considered production-ready in JDK 15.
LMAX Disruptor is a really powerful abstraction if you are trying to build something with consistently-low latency. I've been using a variant in .NET Core for purposes of processing UI events.
It was very interesting, I ported it to C++. However, I lost interest when the whole thing could lock up if the RingBuffer size was very small yet still a power of two.
It also uses sun.misc.Unsafe to do the latency critical aspects, so yes it's Java, but most certainly not vanilla Java.
I come from a place that used Java as the main language for its strategies, and the developers' overall sentiment was very close to yours.
I saw people comment on the "fast<->smart" continuum, and in this context, I believe they mean smart=computationally intensive. (An extreme example is a stat arb shop running its portfolio optimizer.) But there's another way to gain an edge, which is to iterate quickly on your "fast" algorithms and develop them so they take advantage of opportunities that are only partially exploited by competitors. Java seemed pretty good for that purpose for about a decade. Things like JVM warmup, GC, individual ultra-low-latency responses whenever necessary, were all dealt with after the fact.
If you want something that's as fast as C++, but safer to work with, then there's this new language, has been on HN frontpage once or twice.. I can't remember the name, but I think it had something to do with oxidation of metals.. Something about shellfish as well..
The person you're replying to is almost certainly not using the word "safe" in the same sense you are. In particular: safety in trading has to do with risk management, strategy correlation, side effects, and correctness.
Software security matters, in the abstract, but trading firms don't care about it nearly as much as tech companies do. The other thing is that being as fast as C++ is not compelling enough a reason to replace C++ for their use cases. If anything, Rust's dependency management story would be the thing to highlight (this is part of what's compelling about the JVM).
I think the responder's version of 'safety' really boils down to 'null pointer safety' - not 'software security'.
And even then, people keep talking about it like it's 'the thing'.
Java has null 'references' as do most languages frankly and it's just 'a thing' almost never 'the thing' to be concerned about.
'What devs want' is basically something kind of like Java, but that compiles, predictable/controllable performance and memory management i.e. no GC. That's it. It will probably end up being Rust, but that's because Rust will eventually provide all the nice, clean, modern package management, idioms, libraries etc., not specifically because of the 'safety'.
Granted I don't want to diminish that in the attempt to create 'proper memory management' you probably end up writing better software anyhow.
That's fair. Instead of saying "software security" I should have been more specific in talking about memory and pointer issues. In that sense, yes Rust does offer some safety guarantees beyond preventing memory corruption vulnerabilities.
Why would you think that? Rust's major selling point for safety guarantees is the borrow checker. It does not provide most of OCaml's type and functional semantics.
That being said: Jane Street is pretty avant garde in this respect. I expect there already is, or soon will be, a successful trading firm which likewise builds its tech brand on Rust and alternatives to Rust standard library primitives. But overall adoption will probably continue to be anemic.
It has immutability, no null pointers, and error types. It has escape hatches available for everything, but unnecessary use of these should be caught in code review, so I don't think it's a big deal. Not sure what correctness features OCaml could add beyond those, but then again I've never programmed in OCaml, only Scala, so I might be wrong.
This is perhaps the most "blub" comment ever written. There are whole classes of bugs that can be caught with more advanced type systems that the Rust type system + borrow checker will never catch. No null pointers and immutability is the bare minimum.
Does OCaml have dependent types? I thought that was a Coq and Idris (and Haskell, to a lesser extent) thing.
I am aware that dependent types exist, but quite frankly I've yet to see a popular/ergonomic language implement them. As much as we'd like absence of null pointers, error types, and immutability to be the norm it is not and it will take decades until it is.
If I said Rust is rarely used in trading I'd be still overstating its adoption. There is approximately no Rust code in buy side finance. It exists in the sense that you could find something written in it somewhere, even it it's just an infra tool used by one of the software engineers. But I would be very surprised to hear of any trading firm that uses it in the hot path.
There's a major historically-contingent component to this, too. The Island ECN, later absorbed into NASDAQ's ECN, originally ran its matching engine as one of these low-latency Java processes, writing everything using explicit object pools to avoid doing any GC on the critical path, as this was long before Zing was around.
This was a pretty reasonable choice at the time, as this was long before C++11 was around, and Java just offered a lot of advantages for developer ergonomics.
A natural consequence of this is that a number of HFT shops that were seeded by ex-ISLD devs just fell into using something like the ISLD tech stack, which at least at first involved a lot of GC-free Java.
More color on the original FoxPro matching engine:
"Levine found a simple but hugely consequential 'trick' to speed matching. In Levine’s later paraphrase, what the 'enter2order' procedure did when the Island system received a new order was to: 'See if there was a record from a recently cancelled order that we can reuse for this new order. This is hugely important because that record will likely still be in the cache [fast internal memory] and using it will be much faster than making a new one. (Levine, n.d.)'"
Less than you'd think – it comes up in serializing/deserializing data, somewhat, because JNI has a fair bit of overhead, but this isn't really like Rust's "unsafe". To have fast Java, you really need the JIT to be happy, and that's not a safe bet if you're doing random things with Unsafe.
One bit of useful background knowledge: The technical demands for high frequency trading can vary wildly depending on both what kind of trading strategies you're using, and what market you're in. This translates into real-time needs that vary considerably, depending on context.
At one firm I used to work at, the spread was several orders of magnitude. On one end, people were counting nanoseconds, and even C++ wasn't fast or predictable enough. At the other extreme, some teams didn't care about anything finer than a millisecond, and a big chunk of the stack was written in managed languages. It all counted as high frequency trading.
Probably a hardware team using FPGAs. Software tends to be way less predictable than hardware, so to even get to the point where you can reasonably count nanoseconds it often requires hardware
My friend spent two years working for a large bank as a (his words) "trader's bitch". Every day the trader would request a new feature and my friend had at most two hours to implement it.
They used a combination of Python and highly GC-tweaked Java.
This was six years ago, but I see that similar techniques are still used to this day.
In my experience this is a pendulum. Traders hire a few desk developers who are paid by the business, and who work directly with the traders in the same physical area. They crank out code under a lot of pressure for fast turn around, and eventually it becomes completely unmanageable and / or there is some disaster in production.
At that point the technology organisation starts taking by over, the desk developers are let go or they move into technology and start getting paid by technology instead of by the business. Things are now done in a more normal enterprisey software development way, and things are generally done more safely, with more design up front, and more documentation. Of course it also takes a lot longer to get anything done.
So traders decide to hire a bunch of developers, answerable only to them, and back to the start we go...
When you do HFT, nobody cares about that. The only thing that matters is to make money.
I also worked on HFT software written in Java (not the one in this article), and the mindset was completely different from other industries. F QA, F unit tests, F your F-ing AbstractProxySingletonFactory, just write the code that makes it work, see you after lunch to deploy.
Yes, in a lot of ways it is awful, but OTOH it is really great intense bootcamp to learn how to write very modular, easy to change and to read code. No time to over engineer, no time to build these fancy generic abstractions to support the one use case you have and the dozen hypothetical use cases product never asked for.
I'm guessing most people are familiar, but if you're interested in this, look up "Knight Capital flash crash". They lost $440MM in about 45 minutes. As I recall, they reused an old feature flag accidentally reviving some old code.
I'd be curious about knowing the technical details, as the article is pretty much an advertisement. Example: the vast majority of the GC section is about GC in general; only a small part is specific about C4, and with little details.
Specifically, the C4 looks very interesting, although I'm a bit perplexed because, if it was universally good as it's written, they would have essentially solved one of the major problems not just for Java, but for a large amount of languages.
> if it was universally good as it's written, they would have essentially solved one of the major problems not just for Java, but for a large amount of languages.
In a way, low-latency GCs like C4 and ZGC have done that, but implementing them requires a large and experienced team, and not many languages even try to target high performance to begin with.
I don't like how the author of the article said that PHP is an interpreted language and thus does not execute anything like Java is executed.
I'm not saying PHP executes just like Java, but PHP is more like Java than he realizes. PHP does not execute line by line like an interpreted language. But takes the source and compiles it down to an intermediate representation (IR), just like Java. Java calls it bytecode, PHP calls it OPCodes. Also like Java, PHP then takes that IR and executes it on a VM. Java's is stack based and PHP is register based, if I remember correctly. Many PHP accelerators / caching solutions save the IR and thus, the next time the code is needed, no compilation of source code is done.
I can't count how many times I have seen someone effectively say "My thing is better than your/that thing" but they have not actually researched what the other thing is or how it works, they just simply like their own thing so much that they automatically assume it's better or this case different.
Often in corporate environments people are asked to justify a choice of technology, and an approach to a problem. It is quite natural for the people making the decision to choose technology they're most familiar and comfortable with, and positioning it in a favorable light. I imagine this is what happened here.
The actual merit of the competing technology is simply not part of the evaluation. This is at least a step up from government procurement processes where the procurement officer, who has zero technical knowledge, procures based on a checklist he or she is unable to correctly assess bidder responses to, and of course price.
Maybe the article means that PHP interprets the OPCodes as opposed to java that uses a JIT compiler that produces machine code for part of the bytecodes.
Anyway JIT is coming with PHP8 :
The key difference between Java and PHP is the lack of a compilation step. I can change a line of code, press cmd+r and I see the change - whereas in Java more often than not I have to wholly restart the application container as hot-reloading is a complicated mess and tough to set up.
Also, in Java it is easy to end up with resource leaks whereas PHP always starts from a clean slate for each request. Way less headache for operations.
> Also, in Java it is easy to end up with resource leaks whereas PHP always starts from a clean slate for each request. Way less headache for operations.
That assumes you're running a new separate PHP processes for each request and an application built this way is unlikely to scale terribly well.
Right, but in a corporate production environment where everything is probably deployed compiled or compiled shortly after, does that matter? No one is altering prod code.
It certainly makes zero or almost no downtime deployments easier. Assume a Java application with a database that accesses said database during initialization - for a schema change you have to take great care in code that both the old and the new code can concurrently work on the same database to achieve low downtime.
With a PHP application, it's easier - spin up the new Docker container, disconnect the loadbalancer, do the schema change, and connect the loadbalancer to the new Docker container.
When i finished my theoretical Physics PhD, HFT was THE way out of academia for making money in my field. I didn't follow this path as it felt at the time a pretty evil thing to do.
Some people argue that HFT provides liquidity for retail investors, but it's debatable whether that liquidity is real or not since it'll be gone during black swan events. At the same time HFT profit from uninformed/retail flow. So it's debatable wether the actual activity provides values. I'd say probably not.
But there can be indirect value in working in HFT, just like there is with other demanding jobs. There's interesting systems and ML research going on in the field - that's knowledge you can take with you and share with others. Some time ago, HFT was driving a lot innovation in networking, infrastructure, and systems.
Also, over time I've come to realize that "providing value" is really difficult to quantify. Does working in Academia provide value? Maybe if you get lucky and publish something that actually turns out useful. The majority of academics publish papers that are nothing more than noise - negative value. Does working at FAANG provide positive value to society? Again arguable. You may inadvertently end up promoting ads and misinformation. Not everyone does - but it's not guaranteed that you don't without realizing.
"Providing value" is subjective and most of the time it's nothing more than a story people tell themselves so they can feel good about themselves.
I'm not an economics or finance expert, so I may not have a very extensive view of this, but I think there is a meaning to the expression of providing value. If a product has more value to a person than the price of the product, assuming they have correct knowledge, that is the creation of value. In high frequency trading, or stock trading in general, people may be willing to pay more than the listed price, but I don't think value is created because this is generally due to incorrect knowledge. I think stocks are pretty much a zero sum game. If one person is making money, someone else is losing money.
As I mentioned, I'm not an economics or finance expert, so I'd be interested to hear what more experienced people would have to say.
HFT may not, but trading itself definitely provides positive value. People's utility function is not purely monetary over an infinite time horizon. Trading allows you to trade off price and risk over multiple time horizons.
For example, a trade where someone needs to convert his assets into cash due to a family emergency benefits both sides. The person with the emergency takes liquidity from the market and pays a premium because the trade is time-sensitive - he needs cash the next day. Other liquidity traders may profit from such "uninformed" flow in the long term, but both parties are happy because they got what they want.
Another example is trading off risk and hedging against certain changes in the world that would affect you.
I disagree. Trading may seem zero sum on a short timescale, but over the long term markets (empirically) trend upwards. Thus, simply being invested in diverse indices should yield some positive return in the long run. This makes sense as long as we assume GDP growth over time, which is an assumption that has kinda been baked into government/economics to my understanding.
At a short time scale, yes there tends to be a winner and a loser, but even then it's not so clear cut. Most of the trades happening at a short time scale are institutional rather than retail (i.e. people who have the risk tolerance for this type of trading & who have profited enough from it to keep doing this type of behavior). The activity of most HFT firms is some combination of market making & arbitrage, where the profit can be thought of as a fee for trading for other participants (e.g. Robinhood doesn't charge commission but sells order flow, which tends to be cheaper than commission per trade). In general, price improvement is also possible since these make markets more efficient, so as an example it would not be unusual to see firm A sell to firm B who sells to firm C, where both A & B see a profit from their sales. This is possible because prices across markets are not automatically synced, so A has access to a better price than B who has access to a better price than C. And because institutional trading dwarfs any kind of retail trading, the trades can appear high frequency from the perspective of the HFT firm while only occurring, say, once every six months from the perspective of the retail trader.
Regarding the short term versus long term, I agree with you here. I meant to differentiate between getting profit by investing, which certainly has a lot of value and is the whole point of capital markets, and from getting profit from fluctuations in the current price not tied to an inherent value of the company, which is where I think you have the winner and loser, even if the trades involve multiple parties. Clearly this is what is being done in high frequency trading.
I can buy the idea of high frequnecy trading serving as market making even if it is not something I can fully appreciate. But it still seems like these high frequency traders (and others invovled in investing based on fluctations) are mainly skimming money from the market without providing value.
I agree somehow, in academia we often create glitter (we call that "innovation") to make us feel useful for civilisation. In the end everyone just fight for her/his own ass.
But you know, i'm more willing to write application for Earth Sciences (like im doing now) and living a misery than transferring cash & stocks with obscure algorithms..
That might just be stupid, maybe you're correct, that's just a story i tell myself.
I think that's getting a little high minded for the purpose of the OP's question. I think it's fair to interpret the question as "Is there a good faith argument that HFT is beneficial to markets in a way that also benefits everyday citizens? Are there positive downstream effects? Is the intent of HFT strictly selfish?"
I don't think your comparison to academia holds up there.
Yeah, fair enough. The truth is that I'm just a little frustrated by people touting the horn of "making the world a better place" when in reality most end up doing nothing of the sort but then criticize people working in Finance :)
The market is a machine that routes society’s resources into making nearly everything. Making better routing decisions has immense value.
When someone can profit from the same decision everyone else would make by doing it 0.000000012 days more quickly, IMHO he’s exploiting a design flaw in the way the market clears. That’s amplifying noise rather than signal.
ok, that was a bit of my original question (not well asked).
i understand any civilisation needs funding to grow (do we need growth is another core question), and possibly market to do so.
yet any meaningful decision won't be made at the millisecond scale, sure you can train a model to learn short term rules and generate long term profit. but does that profit society as a whole, i doubt it.
Trading facilitates more efficient allocation of resources at both a geographical and temporal scale. HFT market makers facilitate more efficient trading. If anyone tried pitching automated trading to you as some form of altruistic endeavor then obviously that’s bullshit. These people are out here to make a buck. Those milliseconds to nanoseconds moves are just competition between MMs happening in realtime. Is it done for the sake of a better world? No. But it just so happens to benefit society as a byproduct (on average) in the form of tighter spreads and liquidity. It’s up to you to decide if that’s good enough for you.
If we’re going to paint anyone as “real” villains in the world of trading then I’d argue it’s the informed big block traders. ie hedge funds, banks trading exotics with a high barrier of entry, etc. At least MMs are largely market neutral.
They're part of a the market making ecosystem. That market making allows everyone to buy/sell stocks easily at the true price. HFTs make that more efficient by reducing the bid/ask spread. We had market makers before HFTs so it's not an earth shattering innovation. But it is an incremental improvement.
You can go on Robinhood and buy stocks for free. Courtesy of some HFT in return for getting first dibs on your trades. If we compare that to the fees of yesteryear it's brought real value to long-term retail investors.
Unsophisticated investors only using market orders will also pay less spreads than they used to.
Going out on a limb, I think the losers are mutual funds and similar who are less able to monetize their fundamental analysis by buying or sell big chunks of stock without moving the market.
When bitcoin was young there wasn't much in the way of HFT between exchanges, which sucked because if you wanted to transfer coins to someone across the world you would on average lose 7-12% on price differences. Western Union, in comparison, was a flat 10%, making it bitcoin at the time almost pointless.
When there isn't enough volume HFT strategies tend to not run, and so to give an example, my first time trading options was an iron condor on /GC (gold). I bought at a reasonable price, but when I went to sell, where I would have netted a cool 50%, because of low volume and I didn't know to set a limit order, I lost about 50%, so even if my trade was "successful" I lost. If there was a bit more volume HFT could step in and we could have normalized prices where limit orders are not required. On the other end, without HFT, it's easy to play that game and setup orders of opportunity where someone else might mess up and you can make a pretty penny, not that I've personally tried this.
No, but there are many things that don't provide any net value that is even more prolific. All the sugary drinks that just give you health issues, the big monopolistic companies that replaced smaller decent-wage shops with big low-wage machines that suck the value out and doles it out at the top. If everything is designed to take value from one place and never give back, is that beneficial? Of course not. If you only look at the product, you see only half the picture.
Evil is everywhere, really. Our brains are not made to really understand these things well. The easiest thing is to resign yourself to at least make you and your familys life easier. Which is acceptable enough.
> All the sugary drinks that just give you health issues (...)
> Evil is everywhere, really.
i know, i live 10km from Nestlé HQ. and the banks in my country manage a significant part of the cash in this world.
> Our brains are not made to really understand these things well.
Agreed. I was recently reading several text saying that our brains are adapted to small hunter-gatherer communities of 300 ppl max.
> The easiest thing is to resign yourself to at least make you and your familys life easier.
Not now i know how the Earth System react, because a look at the latest IPCC projections doesn't make me want found a family.
> Which is acceptable enough.
Listening to some climatologists, there is a chance i see a world with less than a billion person from my own eyes. My brain cannot accept such a perspective acceptable (without actively fighting it at least).
High margin businesses rarely if ever provide (equitable) benefits to someone other than shareholders, but the innovations they create may/usually trickle down to the rest of us over time, one way or another.
Yes, HFT provides liquidity to capital markets, reducing transaction costs of financing. The social benefits here can be a bit hard to see, but they are certainly real. When HFTs make money, it's because they're buying low and selling high. That means that they are making prices slightly more efficient for the next person to come along, which in turn means that that next person gets a slightly better more accurate price.
Yes, it absolutely does. Rapid efficient capital allocation allows our economy to react nimbly to anything that gets thrown at it (like covid).
The sharp v recovery in the markets was surprising, but not so much when you realize what's backing it. People gave credit to the wrong group. They said it was the fed, it wasn't, it was the HFT. All the fed did was add some zeros to accounts and relax some interest rates. HFT had to figure out where the money goes.
Without HFT, it likely could have taken a year for the market to recover. It would have taken forever to unravel all the changes (some ephemeral, some not) and figure out how to efficiently allocate all that money.
If I was the democrats, I'd find a way to connect the market to main street rather than try to create ftt. I believe the first country that does this will win the 21st century.
Imagine being able to come up with a business idea, partner with some friends, do a bit of proving out and publicizing your team and money just flows into it based purely on the merit of your real potential. No politics, no long lunches and dinners with VC, no irrational idiots. Just pure, unfettered meritocracy.
Or imagine having a long standing mainstreet business that gets hit with a one time, out of its control, disaster. No need to go the bank with hat in hand, the money just shows up in your account, with hyper competitive interest rates. The same could be said if you have an idea for expansion or efficiencies. Just send out a press release with a few specifics, and next day you are underwritten.
SPAC is sort of like that. Sadly, democrats have their own brand of xenophobia. They tax what they don't understand. It's very unfortunate that the GOP didn't vote with the democrats to impeach Trump. Historically, I think it will go down as the most colossal political mistake since Nixon.
One way to think about all this, is DeepMind / starcraft. AI could already out-micro any competitor, but now it's getting to a point where it can out-strategy as well.
Having tools like that to manage our markets is a very powerful tool and makes the US not only hyper efficient but a far more competitive than any other nation.
That said, the whipsaws of computers figuring out things too fast can be tough on people. pure, unfettered meritocracy can be hard for those without merit to offer.
Yang and his UBI isn't a bad idea.
Where it provides less value are traders who intentionally inject volatility. Some guy went to jail over that (flash boys)
Except that HFT firms don't call themselves HFT any more, they re-branded to ultra-low latency trading.
And FPGAs are the norm now, not the exception.
Sure, Java still has its place, but implying that it is anywhere in the hot loop of a contemporary ULL system is wrong at best and disingenuous at worst.
The importance of FPGAs is generally oversold. They definitely have their place, but the problem is due to the complexity you can only evaluate very simple triggers and/or must sacrifice a ton of agility where strategy development is measured in years instead of weeks.
The biggest manifestation of this is that FPGAs really struggle with sizing. Say there's a given market event that produces a very obvious opportunity, but it's not so obvious how big that opportunity is. Archetypical case is when the price changes, and there's a new level formed on the book.
It's almost certainly the case that adding an epsilon of liquidity at the first position of the queue is profitable. What's harder to determine is where the breakeven point is. Should you add 1 lot? 100 lots? 10,000 lots? That's a lot more complex because you generally have to be sensitive to the typical sizes in that particular market and that particular time, as well as how strong the level formation event was. E.g. was it in response to something that looks like a very big order, or just some random liquidity depletion.
What happens is that the FPGAs know they should quote an epsilon, but can't really confidently quote anything bigger than that. So if the right answer is a thousand lots, the FPGA will quickly capture first queue position with something like a hundred lots, then the software-based players with take the rest of the 900 shares for themselves.
What you see with FPGAs is a segment that makes very consistent, very good money relative to the volume that it executes. But fails to capture the sizable majority of the market share.
It's an entire industry devoted to wasting enormous energy, resources and intellectual capacity.
There are simply no material external gains from HFT etc, it's completely zero-sum other than maybe a 'job creation program'.
The site ridiculously talks about 'levelling the playing field' by extending HFT opportunities to other players, which is rich, considering the playing field could be 'levelled immediately' by some really basic regulation and a few smart queuing algorithms for placing orders.
That is not un an entirely unfair characterization. However in my mind HFT practicioners are model citizens when compared to so many of the bright young minds working in the worlds largest tech companies. HFT is a neutral force in the world, while Ad driven tech is causing harm at a scale that is hard to overstate. History will judge the companies profiting from manipulating the worst in us in a much harsher light than they will HFT.
Its not the need for advertising. Its the unreasonable effectivness and devastating side-effects of the attention sequestration techniques developed in the last 10 years. Its not only that years of our of lives are being directly stollen by vapid bullshit, its that the effects of this manipulation extend well beyond the actual time spent engaging with it, with absolutely horrific consequences to our mental health, our coesion as a people and ultimately our physical safety. In a generation, if not earlier, the ad bussiness will be looked upon in a worst light than the tabaco industry. And this will be entirely appropriate.
Again, business cannot exist without advertising. Even money and finance is negotiable - business can survive on barter alone, even though the overhead would be monstrous.
But advertising is not negotiable, it must exist in one way or another if the economy exists.
If you want to go down this road, argue about the "how", not the "what".
The idea of banning advertising outright is too silly for words.
Most traders I've met have been fairly honest about contributing nothing to society, or that the world would probably be a better place is large segments of the finance industry disappeared tomorrow.
I can't say the same for the majority of developers from Facebook or Google I've met.
I run ads and campaigns for my company and they are an absolutely essential form of communication.
... as almost everyone running and actual business knows.
Yes - it can be very inefficient - but it's not inherently wasteful.
You're making the wrong comparison - the 'Digital Ad Industry' could be compared to 'Digital Banking Industry' perhaps - i.e. useful overall but deficiencies exist.
HFT is one of the ares in which there are no reasonable externalities.
Better/faster tech for CDNs etc. can be developed without the HFT industry.
So uh, HFT is a zero-sum waste of intelligence, but paying for the privilege of shitting into people's brains to improve the chances that they waste money on garbage that they don't need is somehow positive-sum and good for society?
Actual campaigns can be good, or bad, and aren't really being referenced here.
The ad tech in question is the issue; the data collected, the platform itself. Think Facebook - in pursuit of creating a platform and getting data that allows for very specific targeting, they have created bubbles that magnify extremist messages.
If you have to ask the question I would encourage you to take $50, open a FB ad account, same for Google, and run some ads to drive some traffic to your blog or whatever.
Running an ad campaign, or having to 'get the word out' for something possibly meaningful can be an enlightening and transformative experience for so many people who live adjacent to, but otherwise 'totally outside' the ostensibly opaque world of marketing.
It's one of the most stupidly represented issues in pop culture, like 'reefer madness' memes form the 1960s' where 'smoking weed will make you gay!'. Well - just 'smoke weed' once and you realize it's frankly just not that big of a deal.
Ads are ancient, normal, universal. The industry has serious efficiency problems, we're starting to face privacy issues (which are overstated but nevertheless legit), and I don't like some mediums (I for one, think we should dump most billboards) - but the nature of the industry is not wrong or evil, it's actually beneifical.
'Good ads' are a win for everyone. HFT is only a win for those directly involved and it's zero sum.
I don't think that's quite right. It's the same way Formula 1 racing contributes to regular automotive industry. Technological innovations could be adopted by more mainstream industries in a few years. For instance SolarFlare NICs used to be almost exclusively used by HFT players and can now be found in most CDNs or latency sensitive cloud providers.
The F1 comparison is not bad, but its not great either. F1's tech innovations are much more publicly visible (HFT shops are extremely secretive) and F1 has the added benefit of providing entertainment value to millions.
This amounts to explaining that a $20 hammer is better than a $10 hammer when you frame your own deck. It really is, but that tells you nothing about the equipment a contractor needs to operate a construction business.
The majority of programming tasks don't need anything better than Java, and don't need anyone more expensive than the typical Java coder.
Low-latency trading is not one of those tasks. In low-latency trading, as in poker or any other zero-sum game, you either rip the other guy's heart out and stamp on it, or get your own heart ripped out and stamped on. Using any but the best tooling and programmers means you will be that second guy.
So, anybody serious is using C++17, coded by people who understand cache interactions, branch prediction, TLB shootdowns, and core isolation, and who avoid lockless queues because of the cache-line pingponging. If you don't know what all that means, you are not the right person to code low latency trading.
They are not the hardest core: the sharpest players farm the simplest, most time-sensitive operations to FPGAs. But FPGAs take tradeoffs that make them not right for everything, and FPGA coders are not just programmers.
> dedicated sub-oceanic lines (Hibernian Express is a major provider), even microwave networks.
This statement as a definitive "this is how it works" makes me think the person who wrote this doesn't really work in anything at the intersection of networking and HFT.
Much of the serious HFT for transoceanic stuff at low bitrates, but very important signalling messages, moved to shortwave radio with giant-ass aimed yagi-uda antennas several years ago.
You can go out to some places not very far from the CME datacenter in Illinois and find big, suspicious shortwave band things aimed at Tokyo.
To add to this, I know for a fact some people who are really serious about Java + HFT, and they would never ever use a GC. I can’t believe how much this article talks about Zing and other GCs, while the de-facto standard way of doing this is to just always reuse all objects and never deallocate them.
I get a very real feeling that the people writing this article have no idea what they’re talking about.
You use special libraries designed for this purpose, that use a static buffer under the hood themselves. Or you just make sure that memory usage increase is low enough that it isn’t a problem. You sometimes have places where they do for example one GC call a day.
Does anyone know if C#/dotnet is used in HFT to any extent? I'd imagine proper support for user-defined unboxed types (structs) and non-hacky methods for manual memory management (unsafe) could give it major advantage over Java with much of the QoL improvements java has over C++.
I've worked for one of the big trading firms and they used C# including for their ultra-low-latency trading strategies.
I'm a bit surprised by all the chatter about garbage collection because it's not as much an issue as it sounds.
The thing is not to do allocations (i.e. GC) on the _critical path_.
In a trading strategy you react to a stream of market data. It appears constant to the human eye, but in fact it's a series of "blips", sporadic events.
When you get a new message you need to react as fast as possible. The fastest code is, of course, no code, so it doesn't really matter if it's not written in C++, C# or PHP. Here you're just going to do the very minimum required. You will find that generally there is no difference in the code compiled whether it' done by the C# JIT, the Java JIT or a C++ AOT compiler, because the code is simple: you read a few fields from the message and select a pre-computed outcome to decide whether or not to send a (buy|sell) order.
Now, after that event has been handled (and perhaps you sent an order), you need to recompute stuff. You generally have plenty of time (a few milliseconds) before the next event arrives, and that's where you want to do all your fancy calculations, and here a high-productivity language like Java or C# helps, just because it's quicker to iterate and deploy modified trading strategies (typically deployed daily, or at least weekly).
Many people assume that you need to do all your fancy calculations very rapidly right after a message arrives. But you don't have to because the message is not completely random. The order book is not going to change completely after one message. You can pre-compute a few likely scenarios, and when a message arrives and confirms one of those scenarios, you just mechanically go with the precomputed result.
This is definitely an area of recent improvements, and I don't think modern .net (.net core 3.1 - .net 5) is comparable to older school .net here. Whilst GC isnt quite on JVM level yet (no serious custom GC's afaik), the .net team is clearly paying attention to performance(https://devblogs.microsoft.com/dotnet/performance-improvemen...). As you mention, increased ease of use for structs in the form of Span<T> is having quite significant impact in a lot of common .net libraries, aswell as the recent availability of hardware intrinsics for more performance sensitive tasks. I can certainly say F# has had a pretty decent impact in the finance sector in London and likely elsewhere (not sure about HFT however), mostly as a result of this attention to both performance and developer experience.
I had a conversation with someone who works at Credit Suisse in IT in some capacity.
He said they experiment with C# but went back to C++ due to latency. This was about 12 years ago.
I don’t think Java was chosen because it had ways to do manual GC. I think the developers were comfortable with Java, so they wanted to find a way to fit it into something “fast”.
That was my assumption, considering how dominant Java is in finance. Also, as the article notes, Java has a pretty great runtime ecosystem, allowing devs to choose between hotspot, azul, graal, etc. depending on their runtime needs, while dotnet devs are more-or-less limited to Microsoft-controlled offerings of Mono, dotnet 4.x and dotnet core/5+.
- Until late in it's lifecycle, .net runtime performance was worse than JVM.
- Also, the Windows TCP stack, re-architected with Vista, had severe latency issues, which probably led to wholesale abandonment of that platform for low-latency applications. Some of these problems were fixed in Windows 8, but it was probably too late by then.
(These two points are based on my own experience with Windows/CLR/JVM in the mid/late 2000s and some anecdotes I heard second-hand. So I can't really cite any public sources for them)
I'm sure the fact that you can customize Linux to the extreme and the lack of .Net on Linux in those days probably had a lot to do with the choice of Java, in addition to the above.
However, based on my reading of blogs like Mechanical Sympathy [1] a while ago (~8-10 years), there does seem to have been some demand for .Net-based solutions in the HFT space. They had code for both Java and C# in their public code.
If you are publishing a JVM GC benchmark, you should really include the parameters you used for the program as well as what steps you took to find those particular parameters. While I think it has gotten better in recent years, there are still many knobs for most JVMs which can make a big difference in performance.
Ideally, you would want to have the benchmark developers cooperate with the team creating the JVM you are evaluating in order to be as far as possible with the benchmark sort of like how the TPC benchmarks are done.
This article starts very interesting but the rest sounds like infomercial for Zing by Azul Systems.
Both issues related to low latency metioned here (JIT and GC) can be solved with very simple and common methods with the JVM provided by Oracle.
We had similar issues in my old company, where we implemented a graph database in Java. It was very slow and somtimes freezed for alomost a minute (512 GB heap).
Here are the strategies we used for solving these issues.
- performance - we built warm up logic in our app, which called the most time sensative parts on startup. It delays the start up process by 10 sec, but after that the application is very fast.
- GC - we pre-alocated big chunks of memory outside of the heap and put there the 'heavy' objects - those which require most effort from the GC. We also optimized the code so that all other allocations are in the stack (escape analysis). This requires some discipline, but eventually we ended up in situation where there was nothing left for the GC to do.
I'm not sure if it's worth paying hunderds tousands $ in license fees for something that can be achieved with good programming practices and discipline, which most companies need anyways.
Very cool to see Java continue to be used in high perf situations.
Something about warmup: this is an area where JVMs still have catching up to do versus JavaScript VMs. JVMs have suboptimal startup time in the non-cached case (I.e. not what they are doing with Zing) because it’s just not been an optimization target for JVM hackers. In the JSVM world, we were forced to optimize for this and optimize we did (otherwise page loads would suck). I suspect that Zing with the ReadyNow thing is way better than normal Java, but I don’t know if it’s better than what JSVMs do. It’s probably still way worse if you’re not cached. It might be better in the cached case but then again JSVMs also save some profiling in the cache (JSC definitely does). That suggests that JVMs can probably improve further. No reason why Java - an overall very performant language - should start slower when uncached than JS. I suspect this will get fixed since AOT is a thing now, but getting start times right is about way more than just JITing less.
All HFTs I'm aware of have a continuum along developer_speed to execution_latency. On the developer_speed end we have things like Python (or Perl, Ruby, etc) -> Java/C# -> C++/C/assembly -> programmable nics/fpgas/gpus with access to the nic -> custom hardware. At each part of the path development gets much harder, dev times go up, talent gets harder to find, and the whole thing becomes much more expensive. The trick is use as little of the bottom as you can get away with and control it from as high level as you can.
It would surprise me more to find an HFT that doesn't use Java at some part of their execution lifecycle. Not everything needs to be in microseconds, nanoseconds, or picoseconds, especially if you can factor it out of the critical path and do the work/build some data structures ahead of time.
I am trying Swift in the hope of integrating developer speed while with not too terrible performance. Julia would be another interesting choice in this space. Of course, as you get to nanoseconds, any language choice is irrelevant.
I feel like there's a pretty big change in scale that happens over the course of the article. It starts off talking about a field that needs to "gain those few crucial microseconds" and then talks about a GC with an average and max pause rate of hundreds of microseconds.
I mean yeah those GC pause times are extremely impressive, but you're clearly operating against a different set of performance requirements than those teams that care about several microseconds of difference and it's a bit disingenuous to compare your own tech stack to them.
That being said I do appreciate articles like this that show just how far Java performance has come over the last few decades.
The Azul JVM has been around for I'm guessing at least 10+ years? I think originally you had to buy it on their own custom hardware. No idea what it's like now, but an ex-sales guy told me then that it wasn't super reliable.
They had hardware in 2005, but started transitioning to JVM only in 2010.
They used some of the intel instructions meant for running VMs to get fast concurrent GC. It’s probably a lot easier to court customers with a software only solution than with custom hardware they can’t really repurpose...
Several commenters have brought up garbage collection.
I'm curious. How much heap memory do HFT applications typically allocate in a day?
I'd expect them to be doing most of the work, or at least most of what actually needs low latency, in locally allocated primitives which would be on the stack. If that's the case, you might be able to turn off garbage collection and just stick enough RAM in your computer that it takes a day to fill up with garbage. Once a day, reboot.
Alternatively, you could have a pool of Java servers and coordinate GC with your load balancer.
That is, some system would:
(1) coordinate with the load balancer (or directory service of endpoints) to ensure that a Java node is completely out of the pool
(2) coordinate with the Java node to ensure all its in-progress requests have completed
(3) run a full GC on that node
(4) return it to the pool
(5) move on to the next node, cycling through them.
You probably wouldn't need to increase the number of Java nodes very much, either. If a GC is required every 10 minutes and it only takes 30 seconds to do this, then you only need 5% more Java nodes.
> you might be able to turn off garbage collection and just stick enough RAM in your computer that it takes a day to fill up with garbage. Once a day, reboot
Assuming the code can be made not to allocate much, this approach could probably be made to work, [0] but I imagine it would make more sense to just configure the GC to run once a day.
I wish there were some better open source pauseless GCs. To me, the main value proposition of Rust is memory safety without GC pauses[1]. But the complexity tax you pay for the borrow checker is enormous. If pauseless GC is possible (and Azul seems to demonstrate that it is, or close enough) then I would much prefer to just use GC.
[1] I'm aware of many other benefits, but this is the one most unique to Rust and the one I care about most by far.
Speaking about JVM garbage collection, why isn't automatic reference counting being used instead of traditional garbage collection? It seems kind of like the best of both words in a automatic memory-managed environment of providing safety as well as performance. Languages like Objective-C and Swift have proven that this is very viable. I wonder what's keeping the JVM from adopting it?
Because allocation is slower with just reference counting, and the JVM is all about throughput. The downside is predictability, but newer memory managers are almost fully concurrent so soon it may not matter.
Because it is way slower despite cargo cult about its performance, and any optimization algorithms, just turn reference counting into poor man's tracing GC.
"An improvement can be discussed in the morning, and be implemented, tested and released in production in the afternoon."
I feel like they've buried the lead here. If the above statement is really as it sounds, I find that way more impressive than the fact that they were able to make java fast.
If you avoid overloading yourself with unnecessary complexity, and you as an engineer spend time understanding the problem/business domain, there's no reason why this speed of turnaround should be unattainable.
I have frequently had the experience of sitting in a meeting with (internal) customers and/or a product manager, and shipping discussed changes during that meeting.
If you can't do this, it's because you've put barriers in your own way. Ask yourself if they're really worth it. (They might be!)
I have been doing HFT for many years now, so I have an acute appreciation for exactly what is likely to be involved in doing what they claim to be able to do. If you want to put an algorithm in control of large amounts of real money, then some very thorough testing is required to ensure that it's not going to blow up due to some corner case that no one thought of (of which there are typically a surprising number).
I suspect there are some significant caveats to the claim, and even considering that I still think that's a far more impressive accomplishment than just making java fast enough.
Money corrupts. Wall Street money corrupts absolutely.
In a sane market, there would be no advantage to making an investment and then selling it a few milliseconds later. It introduces unnecessary volatility and incentivizes insider trading and other illegal activity[1]. It defeats the original purpose of the market: to make long term investments. It also leads to some unbelievable waste:
Traders’ need for speed has grown so voracious that two companies are currently building underwater cables (price tag: around $300 million each) across the Atlantic, in an attempt to join Wall Street and the London Stock Exchange by the shortest, fastest route possible. When completed in 2014, one of the cables is expected to shave five to six milliseconds off trans-Atlantic trades.[2]
The problem is that our 20% of our economy is "financial services". It's doubled in the last hundred years or so[3]. Technology isn't shrinking this sector because the people who own it also own lobbyists who own Congress. They write the rules so they can play stupid financial games, and then force the government to bail them out when the latest ponzi scheme hits maturity.
This is a really frustrating comment because it has so little to do with the real world of HFT.
>In a sane market, there would be no advantage to making an investment and then selling it a few milliseconds later.
And yet exchanges (NASDAQ, NYSE, CBOE, LSE, CME, Euronext, etc.) actually _PAY_ HFT firms to do this exact behavior since it provides actual, quantifiable benefit to the exchange and its customers.
>It introduces unnecessary volatility and incentivizes insider trading and other illegal activity[1].
The firm mentioned (Trillium) is not a player in HFT, they are a day trading proprietary trading firm where the firm basically gives individual traders (borrowed) money to use however they want. In this case, some of the traders decided to break the law because there is no real oversight at this kind of firm. When looking at reputable proprietary trading firms (i.e. the kind that don't just hire random people and give them money to play with) this type of behavior simply does not happen. I agree that these types of places are extremely scummy (and are probably responsible for a ton of insider trading crap & the like), but they are not representative of the HFT industry at all.
As for the comment on volatility, the idea behind HFT is actually to help reduce volatility. Outside of a few bugs (such as the Knight Capital debacle you mention) and the Global Financial Crisis, it seems like volatility is trending downwards in part due to HFT. In the same vein, liquidity has exploded thanks to HFT (which is why exchanges pay for HFT). It means that you no longer have to wait minutes or hours for your trades to execute and you can usually get the most up to date price very easily.
> Outside of a few bugs (such as the Knight Capital debacle you mention) and the Global Financial Crisis
I don't know anyone who thought the Global Financial Crisis was a "bug." People went bankrupt, lost their retirement, lost their homes, lost their jobs. These are not small stakes.
> It means that you no longer have to wait minutes or hours for your trades to execute and you can usually get the most up to date price very easily.
Warren Buffet was able to build his empire before HFT, because he was investing, not trying to skim market moves. If the point is to have a stock market where people make investments, HFT literally serves no purpose. It's like getting a weather update every second. If you're investing in a company for its merits, those don't change by the second or even by the day.
I didn't mean to call the GFC a bug, I'm saying that the instances where volatility has significantly and immediately spiked have been:
1. The GFC which wasn't caused by HFT
2. Certain instances where HFT bugs led to "flash crashes"
>If the point is to have a stock market where people make investments, HFT literally serves no purpose
This is patently false. Imagine a situation where the price of AAPL suddenly plummets because of an influx of sellers. If you want to close a long position with AAPL to lock in your gains, there will simply not be enough buyers for your shares to sell most of the time. The purpose of market makers is to make this sale possible because they are still able to profit or nearly break even from buying your shares (and are often contractually obligated to do so). Without someone taking the risk that market makers take, the stock would just free fall indefinitely and nobody would be able to take a profit off their investment.
A similar situation is options market making. People/institutional investors like Warren Buffet hedge their bets by purchasing options to protect them when their expectations fail to pan out. In such a situation, it is imperative that options are able to be purchased for a fair price and due to volatility the price from 10 minutes ago is almost certainly wrong. Likewise, it's important that a contract even exists for your desired purposes.
The purpose of HFT isn't just to give the most recent update, but rather to provide the essential pricing of various derivative instruments which depend on immediate values like the price, volatility, etc. of the underlyings. The immediate ability for your order to execute also protects you from the market's volatility, since you can essentially lock in your price as soon as you see the market moving. The blatant truth is that many modern portfolios consist of more than just unhedged long positions in a variety of stocks and as such a lot more complexity is required than "investing by a company's merits," even in the world of "value investing."
I work in HFT and base comp is pretty much irrelevant. In fact at the firm I work at, every single employee gets the same base salary (low 6 figures). Not that's not important: everyone makes way more money through their part of profits depending on their equity and their amount invested in the fund.
Total cash compensation easily gets above 400-500k. People probably make less money at large firms, tho.
It's stressful in the sense that the amount you earn is a linear function on the performance of the strategy. Unlike at other firms, bonus isn't discretionary: its a pure function of strategy performance. When the company loses money, you lose money. This incentivizes everyone to work together and solve problems.
Also working with some of the smartest people in the industry can be intimidating: if you're used to being the smartest guy in the room, it's very unlikely you will be anymore at a top notch firm. This is a form of stress in itself.
It's not stressful in the sense of shitty clueless bosses, long unpaid hours (people work long hours, but that work is often highly compensated for through pnl improvements).
Everyone I know feels extremely lucky that by just putting our heads together, we can mint money out of thin air. I don't think anyone I work with would rather be doing anything else. Trying to make money out of nothing certainly humbles you, but when something works, its the best feeling in the world.
Keep in mind that things might be different at a larger firm: lower compensation and upside, more hierarchy, etc.
So for my last question, did you have to know someone to get into this field. I'm in fintech right now, and I'm well compensated, I love my job but if I can reach the Pinnacle programming, IE pulling 400 or 500k a year. That would be simply awesome.
Would it be easier to target bigger firms and then get into or start my own firm ?
I don't think you have to know someone. Moreover, if all you're looking for is a straight up dev role (vs quant research or trading), I don't think it'll be that hard. Not sure if you'll pull in 400 or 500k, but if you're good, you should do pretty well for yourself.
Just apply for roles, if you're a good dev, I don't think you should have trouble landing interviews.
I think it’s easier to hit that level in FAANG due to the number of opportunities and less elitist hiring practices.
I do a lot of intern interviews and I haven’t seen a single non-target resume, or someone without FAANG internships, high placement in global hackathons, math olympiads, etc.
Experience is a way in but it depends what “fintech”.
There are a lot of companies that brag about their last victory, rather than their current one. It’s not so much disinformation as minimally actionable information.
> This is why, according to Oracle, Java is the #1 programming language, with millions of developers and more than 51 billion Java Virtual Machines worldwide.
Oof this didn't need to be added to the end of the article.
Maybe to some extent? The teams doing proper HFT are all dealing with C++ or hardware. This is true across the industry AFAIK at places like HRT, Citadel Securities, Jump, etc. Some of these might use some Java for things that don't require low latency but those aren't really considered HFT.
by working there. I would say sure a lot of the platform is written in C, much of the architecture and front end market-making apps are written in C but when it comes to the algo's and tools to spin up new pricing models in a way that is fast, those tools are written in Java.
A sting of nostalgia and coincidence perhaps, was thinking about Java and HFT only a few days ago.
I worked as an offshore employee for E*Trade back in 2010 - 2012. Their desktop app for HFT was written in Java Swing and their web client too was running on the JVM but the 'services' layer the one that interacted with the database was written in C++.
Four years ago heard that they were re-writing the desktop app in JavaFX.
I worked in the industry and it's always a little funny to see who calls themselves HFTs vs quants.
Basically, there's a bit of a spectrum of fast vs smart. In general it's hard to do incredibly smart stuff fast enough to compete in the "speed-critical" bucket of trades and vice-versa there's barely any point in being ultra-fast in the "non-speed-critical" bucket because your alphas last for minutes to hours.
Just from this read, I feel like these folks are just a hair to the right of "fast" in the [fast]---------[smart] continuum. I mostly make this appraisal based on these paragraphs:
>To gain those few crucial microseconds, most players invest in expensive hardware: pools of servers with overclocked liquid-cooled CPUs (in 2020 you can buy a server with 56 cores at 5.6 GHz and 1 TB RAM), collocation in major exchange datacentres, high-end nanosecond network switches, dedicated sub-oceanic lines (Hibernian Express is a major provider), even microwave networks. >It’s common to see highly customised Linux kernels with OS bypass so that the data “jumps” directly from the network card to the application, IPC (Interprocess communication) and even FPGAs (programmable single-purpose chips).
That's nice but that's where the cutting edge of the speed game was in 2007ish. Everything mentioned here is table stakes at this point (colocation, dedicated fiber, expensive switches, bypassing the kernel in network code, etc). The fact that "even FPGAs" is listed as "even" is the biggest thing I focus on. FPGA's and/or custom silicon is where the speed game is right now. Similarly, "even microwave networks" is also table stakes at this point (you can get on nasdaq's wireless[0] just by paying).
This is the kind of game where capex for technology is dwarfed by the margin you're slinging around every day in trading, so you see some pretty absurd hardware justified.
[0] http://n.nasdaq.com/WirelessConnectivitySuite
Edit: Also shout-out to a different comment in this thread mentioning ISLD, a story I considered telling as well: https://news.ycombinator.com/item?id=24896603