Having gone through this learning exercise myself, it always makes me giggle to watch others do it. People first learn about arbitrage and think they can automate it to make money (tm).
In reality, the complexities involved are enormous and the biggest issue is simply that it is almost impossible to predict when the opportunity opens. Therefore, you're always dealing with what is effectively stale data. The next step is to try to trade faster, which then exposes bugs in apis and issues with dealing with connection failures.
That said, I do think that with the proper architecture and enough effort, it is definitely possible to build an arbitrage bot. Especially when you get into more complex solutions, like 3+ way. Those that have done it and are successful with it, don't sell or give away their bots because, it is arbitrage, and to make money, you have to be better at it than everyone else. Why sell a money printer when you can use it to print money?
The other realization is that the exchanges must be running arbitrage bots against themselves. I also would not doubt that there is cross exchange collusion in place as well. They help create volume and heck.. why not? No trading fees.
>And suddenly something changed – just in one single day we no longer were able to place orders in time for few currencies. Either some one plays with scalping (which, in theory, exchanges tries to prevent), or someone were able to prepare efficient ML model to predict price movements within second intervals, or someone just can do it faster.
Something I experienced too. I would still spot profitable trade but would be late to pick them up, most of the time. With arbitrage you also have to beat the fees and the time.
Arbitrage in the past 2 years is a loser game - (or maybe not, and I am that loser). It was still a good learning experience that eventually led to other profitable ideas.
I also shared this experience. I moved from a Python prototype to Java for improved concurrency but still ended up being beat out by somebody closer or faster.
yeah, choice of language is not so easy in this case: c++ seems to be industry standard, but I am curious to measure how golang or scala would improve time of development and what would be their performance.
>> With arbitrage you also have to beat the fees and the time.
I have to add - coins that you use for trading is also important factor, volatility may lead to situation when you were able to increase volume of some alt-coins, but next day their price severely drops
This is why you need to build a global balancer that withdraws funds into more stable coins once they reach a certain threshold. A minimum (e.g. Max trade size * 3) should be kept in each active coin for trading only.
My eyebrows went up when I read "It was early 2017." I'm pretty sure I remember chatting about this at work in 2013 or very early 2014 and even then my understanding was that it was too late and that automated cross-exchange arb trading had already been done. It's exceptionally difficult to squeeze any profit out of this kind of very crowded trade. There have even been Twitter bots that have come and gone which automatically identify opportunities and broadcast to the whole world how an aspiring entrepreneur can lose a lot of money trying to make fractions of a penny if they act now.
I did something similar (https://github.com/Metabot-Tech/artis), I stopped when my bot wasn't fast enough to place order. I had a lot of fun trying to optimize round trip time with various datacenters around the world.
Out of curiosity what was the full timing - since the moment we get data till the moment when we sent request to exchange?
>> I had a lot of fun trying to optimize round trip time with various datacenters around the world.
To be honest - I am thinking crypto exchanges are not at this state yet: many of them relying on public clouds,
some of them still send non-compressed plain text json data, etc.
So I was trading between a market in Ukraine (Liqui.io, now defunct) and one in Japan (Binance.com). My server was in the US in the mid-west on a cheap provider. I found it was the best to reach either side. The round-trip to each was about 100-120ms. It's quite high, but my tests showed me that going anywhere else would increase one side so the best is to have the same time on each side.
I am sure there is a lot of optimization to be done to my bot, but it had a good balance of sales/buy and I would rarely lose money.
I built a simple BTC arbitrage bot some years ago in Python. I'm basically illiterate when it comes to programming, so it was a bit of a slog. It was possible to use it profitably for a time, netting maybe between 10 and 200 USD a day, due to arbitrage opportunities between a certain small exchange that no longer exists and Bitstamp.
My bot basically only traded on two exchanges, and only triggered trades. For any real profit you had to tie down quite a bit of money and risk having them on the exchanges. But the biggest issue I faced was execution. The connection to the API for both exchanges was unreliable. Arbitrage involves executing a trade on both exchanges, and much too often, one trade would fail to execute and instead of arb the bot would effectively take a position in BTC or USD. I concluded it was beyond my abilities and willingness to learn to try to mitigate the risks that came with. Frankly I wasn't sure the problem could be worked around reasonably.
Interesting because they mention many of the same issues our crypto arb had to resolve.
As far as data, there are lots of ways to handle it. Initially the old team had used kdb mainly because they literally thought that was the only possible solution. I made something called `timequerylog` that uses newline-terminated JSON files separated out by date and hour.
It's on npm. It has some simple query abilities. Also there is `tql-cli`. The basic idea is pretty simple, just that you use log files and query by date and time range and key and so you avoid querying or analyzing the entire dataset at once. Note that since it uses the disk with lots of files so much, the assumption is that you are on an SSD.
I tried to do some same platform triangular arbitrage, on paper and in simulation mode my bot was earning a few cents every second. When I started testing with real money I hit a wall when I could not find any platform that placed a bid in under 500ms. My bot was 3 milliseconds away from the server, processing everything in under a single millisecond but the exchange was just too slow to do anything meaningful. In the end I did not lose anything but the fees made it not profitable enough.
I complement with our detailed experiences in "What I Have Learned From My Arbitrage Experiences with Cryptoassets"[1] and in "An Efficient Algorithm to Exploit Arbitrage Opportunities in Crypto Markets" [2]. TLDR: it is very difficult to exploit arbitrage opportunities.
This is a really spot on take on this moment in time. I also wrote a bot circa 2016/2017 [1] and the constant exchange error handling killed any profit I would eek out of the arbitrage. I was only doing this part time while a student so wasn't a huge deal to me, but losing to an exchange throwing up an error that I had never seen before and never would again, or my all time least favorite exchange cryptsy having one side of their books 'stuck' for hours was fabulous.
At peak (December 2017 to February 2018), my small python bot was making between 500 to a couple thousands dollars per day. It was even possible to do arbitrage by hand at that time since you sometimes had a 10-20% difference between the markets.
Can confirm. Early January 2018 I made $10k in a week because the BTC/JPY price was 10% higher than the BTC/USD price. It required a partner in Japan and lots of international money remittances though...
Been writing a trading bot over the past 6 months and I was shaking my head vigorously at literally every single point the author brought up.
At this point the only real “arbitrage” opportunities are through derivatives (swaps and futures). However, there is still enough inefficiency in the market for individuals to squeeze out a bit of profit.
Even internship at a large company was at best 20% of this kind of difficulty. This entails so many little issues (technical + business) on top of the usual data storage and logic.
Maybe I fell asleep during that class I don't know.
Exactly, really my point should have been that you can't expect college to teach you explicit step by step knowledge for most real world problems that will require solving. The idea is that they make you solve a small set of problems and hope that you can extract a general framework for problem solving and communication. Of course there is a lot of rote memorization of reusable axioms that are taught as well.
In reality, the complexities involved are enormous and the biggest issue is simply that it is almost impossible to predict when the opportunity opens. Therefore, you're always dealing with what is effectively stale data. The next step is to try to trade faster, which then exposes bugs in apis and issues with dealing with connection failures.
That said, I do think that with the proper architecture and enough effort, it is definitely possible to build an arbitrage bot. Especially when you get into more complex solutions, like 3+ way. Those that have done it and are successful with it, don't sell or give away their bots because, it is arbitrage, and to make money, you have to be better at it than everyone else. Why sell a money printer when you can use it to print money?
The other realization is that the exchanges must be running arbitrage bots against themselves. I also would not doubt that there is cross exchange collusion in place as well. They help create volume and heck.. why not? No trading fees.