
Open sourcing crypto-arbitrage bot in Python – back to the dawn of crypto hype - villager
http://my-it-notes.com/2019/08/chronicles_of_one_crypto_arbitrage_bot/
======
latchkey
Having gone through this learning exercise myself, it always makes me giggle
to watch others do it. People first learn about arbitrage and think they can
automate it to make money (tm).

In reality, the complexities involved are enormous and the biggest issue is
simply that it is almost impossible to predict when the opportunity opens.
Therefore, you're always dealing with what is effectively stale data. The next
step is to try to trade faster, which then exposes bugs in apis and issues
with dealing with connection failures.

That said, I do think that with the proper architecture and enough effort, it
is definitely possible to build an arbitrage bot. Especially when you get into
more complex solutions, like 3+ way. Those that have done it and are
successful with it, don't sell or give away their bots because, it is
arbitrage, and to make money, you have to be better at it than everyone else.
Why sell a money printer when you can use it to print money?

The other realization is that the exchanges must be running arbitrage bots
against themselves. I also would not doubt that there is cross exchange
collusion in place as well. They help create volume and heck.. why not? No
trading fees.

~~~
Pmop
It would be nice and time saving to have a book documenting thoughtfully these
issues.

------
michalu
_> And suddenly something changed – just in one single day we no longer were
able to place orders in time for few currencies. Either some one plays with
scalping (which, in theory, exchanges tries to prevent), or someone were able
to prepare efficient ML model to predict price movements within second
intervals, or someone just can do it faster._

Something I experienced too. I would still spot profitable trade but would be
late to pick them up, most of the time. With arbitrage you also have to beat
the fees and the time.

Arbitrage in the past 2 years is a loser game - (or maybe not, and I am that
loser). It was still a good learning experience that eventually led to other
profitable ideas.

~~~
tastyfreeze
I also shared this experience. I moved from a Python prototype to Java for
improved concurrency but still ended up being beat out by somebody closer or
faster.

~~~
villager
yeah, choice of language is not so easy in this case: c++ seems to be industry
standard, but I am curious to measure how golang or scala would improve time
of development and what would be their performance.

------
brianpgordon
My eyebrows went up when I read "It was early 2017." I'm pretty sure I
remember chatting about this at work in 2013 or very early 2014 and even then
my understanding was that it was too late and that automated cross-exchange
arb trading had already been done. It's exceptionally difficult to squeeze any
profit out of this kind of very crowded trade. There have even been Twitter
bots that have come and gone which automatically identify opportunities and
broadcast to the whole world how an aspiring entrepreneur can lose a lot of
money trying to make fractions of a penny if they act now.

~~~
villager
That's the thing that was difficult to overcome - it was common belief and a
common perception that everything were settled already.

Probably in other fields situation might be the same - judging only by facts
and data can bring potential opportunities.

------
Sytten
I did something similar ([https://github.com/Metabot-
Tech/artis](https://github.com/Metabot-Tech/artis)), I stopped when my bot
wasn't fast enough to place order. I had a lot of fun trying to optimize round
trip time with various datacenters around the world.

~~~
villager
Out of curiosity what was the full timing - since the moment we get data till
the moment when we sent request to exchange?

>> I had a lot of fun trying to optimize round trip time with various
datacenters around the world.

To be honest - I am thinking crypto exchanges are not at this state yet: many
of them relying on public clouds, some of them still send non-compressed plain
text json data, etc.

~~~
Sytten
So I was trading between a market in Ukraine (Liqui.io, now defunct) and one
in Japan (Binance.com). My server was in the US in the mid-west on a cheap
provider. I found it was the best to reach either side. The round-trip to each
was about 100-120ms. It's quite high, but my tests showed me that going
anywhere else would increase one side so the best is to have the same time on
each side. I am sure there is a lot of optimization to be done to my bot, but
it had a good balance of sales/buy and I would rarely lose money.

------
rebuilder
I built a simple BTC arbitrage bot some years ago in Python. I'm basically
illiterate when it comes to programming, so it was a bit of a slog. It was
possible to use it profitably for a time, netting maybe between 10 and 200 USD
a day, due to arbitrage opportunities between a certain small exchange that no
longer exists and Bitstamp.

My bot basically only traded on two exchanges, and only triggered trades. For
any real profit you had to tie down quite a bit of money and risk having them
on the exchanges. But the biggest issue I faced was execution. The connection
to the API for both exchanges was unreliable. Arbitrage involves executing a
trade on both exchanges, and much too often, one trade would fail to execute
and instead of arb the bot would effectively take a position in BTC or USD. I
concluded it was beyond my abilities and willingness to learn to try to
mitigate the risks that came with. Frankly I wasn't sure the problem _could_
be worked around reasonably.

A fun exercise, for sure!

------
ilaksh
Interesting because they mention many of the same issues our crypto arb had to
resolve.

As far as data, there are lots of ways to handle it. Initially the old team
had used kdb mainly because they literally thought that was the only possible
solution. I made something called `timequerylog` that uses newline-terminated
JSON files separated out by date and hour.

~~~
villager
>>timequerylog For storing only or for data analysis as well?

Currently our postgres have something around 1 Tb which is already not so easy
to play with.

~~~
ilaksh
It's on npm. It has some simple query abilities. Also there is `tql-cli`. The
basic idea is pretty simple, just that you use log files and query by date and
time range and key and so you avoid querying or analyzing the entire dataset
at once. Note that since it uses the disk with lots of files so much, the
assumption is that you are on an SSD.

------
t4ko
I tried to do some same platform triangular arbitrage, on paper and in
simulation mode my bot was earning a few cents every second. When I started
testing with real money I hit a wall when I could not find any platform that
placed a bid in under 500ms. My bot was 3 milliseconds away from the server,
processing everything in under a single millisecond but the exchange was just
too slow to do anything meaningful. In the end I did not lose anything but the
fees made it not profitable enough.

------
wslh
I complement with our detailed experiences in "What I Have Learned From My
Arbitrage Experiences with Cryptoassets"[1] and in "An Efficient Algorithm to
Exploit Arbitrage Opportunities in Crypto Markets" [2]. TLDR: it is very
difficult to exploit arbitrage opportunities.

[1] [https://blog.coinfabrik.com/what-i-have-learned-from-my-
arbi...](https://blog.coinfabrik.com/what-i-have-learned-from-my-arbitrage-
experiences-with-cryptoassets/) HN thread:
[https://news.ycombinator.com/item?id=17249163](https://news.ycombinator.com/item?id=17249163)

[2] [https://blog.coinfabrik.com/an-efficient-algorithm-to-
exploi...](https://blog.coinfabrik.com/an-efficient-algorithm-to-exploit-
arbitrage-opportunities-in-crypto-markets/)

------
maps
This is a really spot on take on this moment in time. I also wrote a bot circa
2016/2017 [1] and the constant exchange error handling killed any profit I
would eek out of the arbitrage. I was only doing this part time while a
student so wasn't a huge deal to me, but losing to an exchange throwing up an
error that I had never seen before and never would again, or my all time least
favorite exchange cryptsy having one side of their books 'stuck' for hours was
fabulous.

[1] [https://github.com/mikem5/arb2](https://github.com/mikem5/arb2)

~~~
villager
That was the most funny thing - we thought if we work only with major
exchanges - we will avoid any issues related with immature systems.

Thats make me severely reconsider acceptance criteria of product to be ready
to face customers :D

------
space-dandy
if anyone wants to disclose... how much can/did you make with a system like
this (towards the dawn of the crypto hype, not now)? awesome!

~~~
Sytten
At peak (December 2017 to February 2018), my small python bot was making
between 500 to a couple thousands dollars per day. It was even possible to do
arbitrage by hand at that time since you sometimes had a 10-20% difference
between the markets.

~~~
Railsify
How did you handle the delay of moving coins between exchanges?

~~~
villager
If you have enough money - you can spread it among various exchanges into pre-
defined coins and just rebalancing it along the usual trading.

------
jotakami
Been writing a trading bot over the past 6 months and I was shaking my head
vigorously at literally every single point the author brought up.

At this point the only real “arbitrage” opportunities are through derivatives
(swaps and futures). However, there is still enough inefficiency in the market
for individuals to squeeze out a bit of profit.

------
agumonkey
overwhelming amount of things to manage

nothing in college prepared me to do even 10% of this

~~~
Railsify
College _should have_ taught you how to study and learn do to something like
this or any other new process.

~~~
villager
\+ how to overcome uncertainties, deal with unknowns and what questions to ask

~~~
Railsify
Exactly, really my point should have been that you can't expect college to
teach you explicit step by step knowledge for most real world problems that
will require solving. The idea is that they make you solve a small set of
problems and hope that you can extract a general framework for problem solving
and communication. Of course there is a lot of rote memorization of reusable
axioms that are taught as well.

