Hacker Newsnew | comments | show | ask | jobs | submit login
Advice on the implementation of a real-time trading system [closed] (stackoverflow.com)
38 points by wslh 996 days ago | 43 comments

He needs to get in touch with the stock exchange (and brokers) before jumping into any implementation conclusions.

He is not a broker, you can't really interact with any exchange directly. He needs to use some bank (which will already do 99% of what he is implementing).

Without knowing exactly how the market systems work, this is a toy, he should have talked to someone BEFORE writing any code. Things are way more complex and simpler than it looks. It's extremely simple to use bank services to do that for you and close to impossible for a developer to just jumpt into writing trading software without a bank backing it.


Considering that users in this thread might be interested in trading software, let me add this link I posted yesterday:


"Markets too complex for high frequency trading software"


Someone ought to create a "stripe for stock trades" to make building stock bots easy-peasy for any developer looking for a challenge. I'd certainly use it.


Sounds like an idea PG would fund. The problem here is not even money, is having enough influence to get people and banks to trust you - and get a deal sweet enough to be profitable.


And that's exactly why is not that easy to get in touch with banks and brokers too.

"Financials" sucks and they hide every inner workings...


This isn't quite accurate. There are brokerages such as Interactive Brokers that provide an API so that you can write automated trading systems. And automated trading isn't limited to just high frequency trading. It's true that most banks have the simple features he mentions in the post but as soon as you want to get a bit more complex what the bank offers just isn't going to cut it.


Yes indeed. The problem with using smaller brokerages would be the pricing (because he is trying to be some kind of broker himself). Banks (investment banks) can even expose the exchange 'API' directly to you. EBS kind of does that for FX


You might be interested in zipline, an open source backtester written in Python. Zipline, Quantopian's open source backtester (yes, I work at Quantopian).

The backtester is designed to also be a trading engine. You feed zipline data in "events" where each event is a point in time. Zipline doesn't care if the events are live or replayed from a database - it processes the data and executes any trades in the order book.

The tricky part here is that zipline is more aimed at implementing trading algorithms. You'd have to write a simple algorithm to implement the buy/sell orders coming in.



Some of my similar architecture questions were closed by Mods. I don't know where one can ask such questions?


Yes, I was asking the same thing. Too bad that we can't use SO to discuss things. HN is good, be the feedback here is not that technical.

What I ended up doing is to cross-post the questions in some mailing list/groups that are slightly related to the topic (celluloid, any active ruby/my-lang group).


"I don't know where one can ask such questions?"

May be start a SO type site which focusses specifically on Artchitectural how-tos ? Like "How to design a real time trading system", "how to design a supply/chain software" etc. What do you think ?


I think the How To format might be too limiting as their are many possible answers. A wiki format with embedded diagramming targeted at architecture would probably suffice once you arrive at some standard for page structure.

This small toolset could also elaborate various business models and business processes as well.




I often see well-thought and well formulated architecture questions to be closed on SO. Why do mods do such thing?


The SE sites have a strict rule - only questions of a certain type are allowed.

That's a good rule. It means SE can focus on that thing. But it's a great shame that the huge knowledge of the SE sites is locked in that format.

There's quite a lot of heat generated around the fuzzy borders of acceptable and unacceptable borders, and having a question closed or deleted doesn't feel good.

Meta is death. I kind of agree, yet SE does have active meta. SE also has chat stuff.

So, would it really kill them to have "freeform SE question site"? It could be a paid feature (although recommending people buy a sub to post a question would cause all kinds of furore).

Thus, you'd have:




It would still have strict rules, and mods, and etc. But it would allow more open form discussion. They might need to have a different structure to allow threading and better replies?


One of the objectives of SE is to build a vast Q&A, which relies on the assumption that all questions can be definitively answered. There's typically no canonical solution to design/architectural questions (there can be many solutions, with varying degrees of quality).

Because of this, http://programmers.stackexchange.com exists, which is a site that allows more free-form questions.


you could try quora


Interesting feedback I already got:




Lmax was the first thing I thought of too but I don't really see how it applies here.

Celluloid looks interesting have you played with it yet?


While it's certainly an interesting question, I presume you're posting it here to garner support for it being reopened (as far as the rules go, I agree with the closure, it's very open-ended). It might fit better on http://programmers.stackexchange.com.

I know little about stock markets but I quite like the idea of spawning workers to monitor orders after being placed. These could then update a page live to the customer.


I too think this is an interesting question and I'd love to hear how people would approach it.

How long do the orders need to be monitored for? Indefinitely? If so, is it really feasible to keep them all running? How much data do they need to analyse / aggregate to make their decisions?

Very interesting question.


For instance orders would be open from week into week if the user enables it (each week).

In my case "decision making" is only based on some fixed price threshold in order to provide some risk management. So the amount of data to aggregate would be minimal. But the data to analyse came directly (as a huge stream) from the markets.


Hi there, I'm the SO owner of the question. It's Ok that the questions was closed I guess... But I couldn't find a way of make it more "to the point".

But now that it's here it would be great to have some comments from HN though ;)


It's not the orders you need to monitor from the sound of it (at least once the trade is complete), it's the position that you hold in the market vs the current price.

* Actually the previous statement could be wrong: You might also want to do this on a per order basis, although they are on some sense functions of each other.

I would suggest the following:

For any market which you have a position in, subscribe to that market's price feed. I'd use zeromq for this, but there are plently of pub/sub libraries and products. The previous point implies that you are republishing your market data feed into specific channels. How they are mapped depends on many factors, but basically you want to de-multiplex the market data so that you are able to pick out one market's price data as a stream of updates from the whole.

Once you have a feed of the market with a position you can use it to transform the stream of market data to multiple streams of values. You might have new steam of PnL values for each order you have on the market, or a stream of price differences or whatever, you then re-publish those streams. Depending on how many of these derivative streams you have and the rate of data, you might want to do some downsampling too.

What I've described above is pretty similar to the market data architecture in hedge funds that I've worked at. Have a look at the ideas behind function reactive programming http://en.wikipedia.org/wiki/Functional_reactive_programming for more info on stream processing.

For the front end, you'll bridge the particular stream of data to a WebSocket (probably translating it to json or a binary data format that javascript can understand at the same time). I don't know how you do that in RoR, I'd use gevent or eventlet in the python world. You'll then need to route this data in some way to a representation of the order/position in the front end. I'd suggest looking at knockout.js, ember.js, angular or something similar as those declarative style libraries will massively reduce the complexity of your client side coding.

Last bit is visualization and there's really only one sane choice for this imho: D3 and libraries built on top of it. Look out for http://square.github.com/cubism/ and http://square.github.com/crossfilter/ and http://code.shutterstock.com/rickshaw/ for those bits.

Bit of a whirlwind here, hope it helps!


Regarding the front end, it's exactly what I'm doing right now. I receive the data, do some processing, pack it into json format and wire that using WebSockets to the client. It works really well.


Great! I think you nailed it. Function reactive programming was being developed in my head somehow. I really really appreciate your comment. Thanks!


From some of the comments I see, let's not read too much into this. He's not asking how to build an -automated- trading system, just one that's realtime and all analysis and decisions are made by the user. The best I've seen that's commonly available is "Trader Workstation" and it's free with your account at Interactive Brokers. It's written in Java.


Yesterday I learned about Postgres' LISTEN/NOTIFY functionality. I'm not sure about your architecture, or how many listeners/watchers Posgres supports, but it might be that you can just listen to changes in your database -- that might be the most lightweight solution.

Edit: Here is a small tutorial: http://bjorngylling.com/2011-04-13/postgres-listen-notify-wi...


If you're trading, keep it all out of the database. It won't scale and latency is terrible and unpredictable regardless of the engine.

Keep the database for storing stuff that you don't need in ram. Keep the rest in ram and message queuing/pubsub bus architecture.

Plus events in a database engine are a pain in the butt to write tests against.


I have made one in ruby for my trades. The key point of the architecture was to do all asynchronous with queues and workers. It worked for me because the network time was big and I don't do high frequency trading. I have used rabbitmq and ruby with event machine.


You could use one of the realtime api's, like pusher or realtime.co. They can handle the scale. Otherwise erlang/otp looks like a good fit


Use Java, of course.


Java is certainly being used, but it is not an optimal language for trading software.

Trading software has the bureaucratic aspect (IO, etc) and the really money making aspect (calculations).

Let me show an example how f'up pretty much all non-functional languages are in solving SIMPLE interest (total cost of operation) calculations:


This can be solved with a few lines of OCaml - not that current O(nˆ2) approximation crap.


Why "of course"?


Off the top of my head, the op is probably talking about the off the shelf stuff available for mq stuff and integration which is unsurpassed in java, plus the very low latency IO.

TBH when I write java, I usually write virtually no code: I just glue other shit together.

I don't get why people have so much against Java - it gets shit done in incredibly short amounts of time.


I find that people who "hate" java or any other language for that matter, haven't even used it in any meaningful way.

I find its a great language for integration in general and talking to "legacy enterprise" systems.

I do find it verbose, and have other remarks about it, but it does the job well enough.


Yeah, I actually kind-of think like that. So I use JRuby to integrate things with Apache Camel for example...



Until you have to come up with your own algorithm. Then you're stuck with ohgods-so-many-.java-files


Depends. I wrote a rather complicated corridor based routing algorithm for a courier company in the UK which worked on real time traffic data. It was only 11 files and 12,000 loc and that was their entire business model.


You mean Scala, right?


Wouldn´t it be much more interesting to spend that precious brainpower into something that actually solves real problems like war, disease, crime, militarism, xenophopia, lack of education, etc. instead of building another greed-engine that will trigger even more poverty, exploitation and repression?

Of course everybody is free to act on his own karma (in a few countries) - but you will have a much better life by doing some more useful work that will benefit all human beeings, not only a few criminals that build weapons of mass murder just for the mentally poor reason of making more and more money.

Programmers haz powers to push human evolution, don´t waste your life on golden lies, that will turn into deadly dust in the end!


This kind of sanctimony is tiring. Every time one of these threads comes up here, it inevitably devolves to a point where someone brings up something like the above. There are those of us who actually enjoy these kinds of problems in and of themselves. I don't consider what I do particularly noble, but I'm paid to solve very interesting and unusual problems in a very fast-paced environment, and I don't feel the need to apologize for it.

I'm under no illusions that I'm doing God's Work, but if the bar for useful work is something that will end "war, disease, crime, militarism, xenophopia, lack of education", I think >99% of us fall quite short of it. Maybe I fall shorter than you do (you don't mention what it is that you do), but I still enjoy my job.


A "Quantitative Analyst Salary" starts at 100k/year. Trying to build a trading system is like trying to invent the next 10x battery technology could end up golden dust in the end as well. You're distracting people from proper cost benefit analysis and assessment of risk/reward.


Programming getting into high frequency trading is "high risk, higher reward". It would be better to use our Neural Networks to find risk factors for prostate cancer you'll get in 25 years, but the money just isn't there. AKA high risk, low (possibly no) reward.


Applications are open for YC Winter 2016

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact