
Building a Forex Trading Platform Using Kafka, Storm and Cassandra - jecs321
http://insightdataengineering.com/blog/Building_a_Forex_trading_platform_using_Kafka_Storm_Cassandra.html
======
fxthrowaway
This is a somewhat related question about MetaTrader, which is the most
popular FX platform.

It could be considered paranoid and jingoistic, so I apologize in advance to
all the great Russian hackers out there.

Does anyone know anything about MetaQuotes (the maker of MetaTrader)? They're
a trusted solution provider to many of the biggest banks in the world, and yet
in my experience it's difficult to find anything about them online (the
company, not the software).

I know it's a broad stereotype, but many Russian companies don't have the best
reputations when it comes to business ethics. It's an open secret that many of
the top oligarchs formed more or less a kleptocracy with the Putin
administration, and that the rule of law is pretty shoddy when it comes to
Western companies or individuals seeking justice.

Traders all over the world plug in their algorithms in plain text right into
MetaTrader. Realistically, do they have anything to worry about, or am I being
completely paranoid?

I don't mean to disparage MetaQuotes. As I said, I don't know anything about
them, and they partner with big financial institutions all over the world.
Many of these companies provide customized versions of MetaTrader, so I would
imagine they might have access to some or all of the MetaTrader source code. I
guess I'm just looking to be reassured.

What steps could a trader take on any platform in order to reduce exposure to
potential bad actors? Is using an API from a broker the only solution?

Anyway, again my apologies if I offended anyone.

Update: their Wikipedia page has a few more articles on them than I
remembered. Also, technically they are registered in Cyprus, but were founded
in and are primarily based in Russia. Still, I'd love to hear people's
thoughts on this issue:

[http://en.wikipedia.org/wiki/MetaQuotes_Software](http://en.wikipedia.org/wiki/MetaQuotes_Software)

~~~
twelve40
Just like Intellij! Omg the Russians own our codes

~~~
random3
JetBrains (the company behind IntelliJ) is headquartered in Prague - Czech
Republic.

------
pas
> It is a truly global marketplace that only sleeps on weekends.

Sure, there are 4 trillion reasons why this is enough and yes, I know it's
very much about traders themselves, and that if there were an incentive to
change it probably would change pretty fast, but anything that stops on the
weekend seems very silly in the 21st century. (Even EVE Online's daily
downtime is a bit ridiculous.)

~~~
xxs
You need liquidity on Saturday and the banks don't work.

~~~
pas
Why is liquidity for settlement so strongly linked to making contracts
(trading)?

~~~
beagle3
Banks are also counterparties to effectively all large contracts, and there's
an avalanche effect that keeps most traders out of the market.

------
fasteo
I am no expert in forex platforms, but this statement seems a bit off:

"individual investors only have a few simple tools at their disposal, e.g.,
Meta Trader or Ninja Trader"

Both meta trader and Ninja trader have powerful event-based scripting
languages (MQL5 [1] and NinjaScript [2]) with a solid library of charts,
strategies, indicators and order execution. More than that, they have a huge
community providing all kinds of services around those platforms[3][4]

Other than that, Wolf seems like a nice piece of software.

[1] [http://www.mql5.com/en/docs](http://www.mql5.com/en/docs)

[2]
[http://www.ninjatrader.com/support/helpGuides/nt7/](http://www.ninjatrader.com/support/helpGuides/nt7/)

[3] [http://www.mql5.com/en/signals/mt5](http://www.mql5.com/en/signals/mt5)

[4] [http://www.metatrader5.com/en/automated-
trading/mql5market](http://www.metatrader5.com/en/automated-
trading/mql5market)

~~~
dclusin
Lots of brokers also offer vanilla FIX connections to their systems as well,
so a GUI isn't necessary.

~~~
kolencherry
Interactive Brokers offers a FIX interface to their system.

------
genericacct
i fail to see the point in using hadoop if your universe is limited to foreign
exchange... you can probably fit all ticks of all pairs in existence since the
70s in less than a TB

~~~
dodders
Assuming 40 pairs, 2 ticks per second and 1kb to store a tick (including
timestamps, indexes etc), I reckon that's about 70TB for 40 years worth of
data, not counting weekends.

I've assumed only outright ticks, hence the 40 pairs - in reality most banks
store the crosses as well, which can be up to 1,600 pairs (40 x 40), which
will get you into the PB range for the 40 years.

~~~
btn
One kilobyte per tick seems quite generous. All of the EURUSD tick data for
2013 from histdata.com (the source mentioned in the article) is only 515MB
(~20GB for 40 years, ~824GB for 40 pairs).

~~~
throwaway283719
I would say a "tick" comprises the timestamp (stored as a long int) and ten
levels of the order book (bid price, ask price, bid size, ask size) each
stored as a double-precision float or a long int, so that's

    
    
      (1 + 4 * 10) * 8 = 328 bytes
    

per tick, so 1KB isn't far off. Obviously not every level changes on every
tick, so there are opportunities for compression, that can be significant.

Note that the "tick data" from histdata.com gives you prices sampled every 1
second (so not every tick) for the top level of the order book, and doesn't
give you any size information at all.

------
yanev
The library definitely deserves attention considering the fact that the
current MetaTrader and NinjaTrader platforms are like dinosaurs, they feel
very old. In the article you said that the stream of information is essential
to the operation of Wolf. I also think that this is the fundamental part of a
forex platform - the data should be accurate because you have to validate the
forex strategy as much as you can during the backtesting phase. Have you
considered replacing the current data provider (HistData.com) with something
better? Actually we are on the same track because I am also implementing a
forex trading platform and I have experienced a lot of problems just because
the HistData database is not accurate enough for backtesting - it has a lot of
empty records, sometimes a whole day of tick data can be missing.

~~~
jjHey
Thank you for the comment. I agree that having sound data is essential to
building any trading algorithm. Probably that's why they are so expensive. I
won't complain about HistData.com, it's a free service, they only charge a
small fee for a convenient ftp access to their site. At the moment, that's the
most you can get for free with a millisecond resolution. Despite it's
drawbacks, works well for backtesting, IMHO.

A great alternative is finance.yahoo.com. It is reasonably rate limited, see
[http://stackoverflow.com/questions/9346582/what-is-the-
query...](http://stackoverflow.com/questions/9346582/what-is-the-query-limit-
on-yahoos-finance-api). You won't be able to do any high-frequency trading
anyway without dedicated hardware in a dedicated data center.

------
itaifrenkel
Were you actually able to get 1ms latency from Kafka? How long does it take an
event to enter then exit Kafka?

~~~
jjHey
No. I only had boxes with SATA drives. I don't see how you could get below 1ms
latency with spinning disks. You would have to use in-memory queue to get this
sort of latency. Secondly, I had Storm consuming off Kafka every millisecond,
so in the worst case scenario a message would have to wait one millisecond in
the queue anyway. Here's a great article about throughput/latency of Kafka
[https://engineering.linkedin.com/kafka/benchmarking-
apache-k...](https://engineering.linkedin.com/kafka/benchmarking-apache-
kafka-2-million-writes-second-three-cheap-machines).

------
scottlocklin
"The foreign exchange market, or forex, is the biggest and the most liquid
exchange service in the world with over $4 trillion worth of trades made every
day."

Yes, and you are shut out of that market unless you're on the interbank (which
generally means you are an actual bank). Retail forex "markets" are bucket
shops that get first look at all their customer bids. The actual forex market
is OTC.

[http://www.ecnforex.co.uk/interbank-forex-markets-
explained/](http://www.ecnforex.co.uk/interbank-forex-markets-explained/)

Trading retail forex is a mug's game.

~~~
threeseed
I would disagree that it is entirely a mug's game.

There are some EAs for MetaTrader that will check if your broker is
manipulating your SL/TP and stop the trade. Also if you're a medium/long term
trader then it doesn't really matter what your broker does.

~~~
scottlocklin
In that case: you should be trading futures.

------
mattfrommars
Anyone know how I can run this? I was into FOREX few months back and this tool
looks very interestnig. I need to exchange large sum and want the most of the
volatile FOREX market.

~~~
pas
[https://github.com/slawekj/wolf#getting-
started](https://github.com/slawekj/wolf#getting-started)

It suggests to deploy modules on different machines, but you could probably
try this out on 3-4 VMs (use SSDs).

There are rather simple deployment scripts (it uses apt-get, so get Ubuntu
VMs), for some of the modules.

Set up a Hadoop cluster (HDFS, Hive, Camus) use YARN, you can run Storm on it,
but there is a deployment (install & run) script for running Storm standalone
(with a standalone Zookeeper node).

So, it's a lot of work, but sounds really interesting, fun and immensely
didactive!

~~~
jjHey
A pilot installation of Wolf was made on 10 ec2 boxes, mainly m1.medium and
m1.large if I remember correctly. 3 boxes for C*, 3 boxes for Hadoop, 1 for
Kafka, 1 for Storm, 2 for everything else. You might be able to squeeze it in
less machines as Amazon is upgrading specs all the time. Deployment scripts
are very simple at the moment, just a bunch of shell commands. That might be
upgraded in the future to some magic automation. Nevertheless, scripts were
tested on Ubuntu 12.04, but are not specific to Ubuntu, other then the package
manager (apt-get). Probably if you sed s/apt-get/yum the scripts, they will
work on CentOS/Fedora. If you have a really strong machine you might even be
able to squeeze everything in one box. I wouldn't recommend it though, keeping
the state of all the clusters in one Zookeeper is a very risky business, plus
reliability issues. Unfortunately, it's not so simple to get Wolf from
nothingness to a working/trading system. To actually trade you will also need
a data feed, and a brokerage account.

~~~
mattfrommars
I'm not ultra tech savvy. Just getting into programming but I know bits and
pieces. What the guy made, is it superior to metatrader 5 or 4? How is it
different from them. I'm running Ubuntu 14.04 as a default machine. Also why
is a huge cluster, Amazon servers required? Isn't it possible to run it on a
home computer? I don't see any heavy computation. From what I understood, it
harvests data from sources and displays them in a plot form.

~~~
pas
And it's able to go and get any historic data. It can scale up to thousands of
machines (petabytes range), it can scale up to simultaneously ingest thousands
of data flows, and so on, and continue to do whatever you want based on event
processing (Storm can watch for any strange rules you can come up and fire off
events [triggers] to an other component to execute trades), without stopping
the flow for even a second.

It's a proof of concept in distributed systems engineering.

------
sorincos
Because running a trading algorithm on a non-idempotent stack is such a good
idea.

~~~
jjHey
Very good remark! Storm by default implements at-least-once semantics. If a
bolt process executed an event and failed right away without acknowledging, an
execution would get replayed (so you would buy/sell twice). However, there's a
remedy! By sacrifacing latency you can have exactly-once-semantics, see
Trident, Summingbird.

------
badname
Yes. Because that's what was missing in this world. Yet another foreign
exchange lottery platform for the rich kids to gamble around.

Edit: Oh boy - the all-I-wanna-be-is-rich egotistical nerds downvoted me. My
soul is crushed.

