
Languages for online betting: an investigation of Go, Erlang, and Elixir - iamd3vil
https://www.erlang-solutions.com/blog/programming-languages-for-online-betting-an-investigation-of-go-erlang-and-elixir.html
======
yupyup
"Erlang Solutions" says Erlang is the bee's knees...

I really like when someone that doesn't have a horse in the race reviews some
language (better if it has been used in anger). You gotta value the
perspective of a pragmatic engineer that has misapplied a tool.

There's a lot to learn about where a language is best used, but when someone
pay is somewhat tied to a language I don't trust the analysis as much.

~~~
batrat
The 1st thing I thought when I so the article. A "Solutions" company
advertising their product?

Use the tools that get your job done and stop bragging about it.

------
jstimpfle
I'm currently writing a GUI for industrial automation, where we have to talk
to a number of machine control threads and image processing threads (all
through a procedural interface that connects to other processes).

I'm writing in C and decided to stay in a single OS thread and to instead
write my own simplistic user thread scheduler. User thread ("green threads")
work by simply switching call stacks and saving registers. In my case this is
as simple as making a few Windows API calls like CreateFiber() or
SwitchToFiber().

It's one of the best things I've ever done. Maybe 1 day of work to get the
threads rolling, no bugs, and still in a familiar C environment. No bindings
required to interface with the libraries that I _need_ to interface with (like
Qt) and perfect deterministic control over what happens (just not the other
processes of course).

I think I can understand people are not using C doing web stuff (lots of
strings) but for what I do I'm convinced it's the easiest, fastest to develop,
most performant, most robust thing you can do.

~~~
lucidguppy
How can you be convinced if you're only running the control in an experiment?

~~~
jstimpfle
It's not an experiment.

~~~
todd8
I don't believe that lucidguppy was saying that your system was an experiment.
Instead, it would be interesting to try implementing your system in a second
language so you could compare and contrast the differences. (Of course, in the
real-world one often doesn't have the luxury of doing this.)

~~~
jstimpfle
Ok, so my conviction comes from the fact that the code is super straight-
forward and I don't have to deal with the complexities of a managed language's
run-time system (like tuning the garbage collector or scheduler, using
synchronisation primitives, depending on the ecosystem/using terrible
bindings, etc).

All investment is just a page of code using the Windows Fiber API, and the
scheduler ist just 10 lines of code doing round-robin scheduling. Which is
totally fine right now, and there's the added benefit of being in total
control and being able to make adjustments if they are needed in the future.

Since the scheduling is non-preemptive, threads are not allowed to block. But
only a synchronous, blocking coding style is maintainable, so threads use
wrapper functions to execute slow operations. The wrapper functions have a
synchronous interface, but internally they call asynchronous functions, and
unless the request can be served immediately, they call the scheduler. The
wrapper functions make another 50 lines (so far) of really simple bullet-proof
maintainable code.

It's really like Javascript technically (non-preemptive scheduling), but you
don't have to do callback chaining. You just write your code in an ordinary
synchronous blocking style. Think about that. You write thread-heavy
interactive C code that is much more maintainable than Javascript code. And of
course, much more maintainable than clumsy OS threads - no management, no
synchronisation primitives and no excessive copying between threads. On top of
that, the scheduling is deterministic. All with 100 lines of straightforward
infrastructure code.

The only boilerplate in the project comes from using Qt/Qml.

(If you're on POSIX and can't use the Windows Fiber API, you can depend on the
deprecated makecontext/getcontext/setcontext API, or write a little inline
assembly. It's not a big deal either way, since it's really just a few lines
of code that are neatly separated from the rest of the system).

~~~
ac2u
Would be nice to see a blog post expanding on this a little more with more
examples and some basic diagrams.

------
misja111
The article briefly mentions the similarity of online betting to stock
markets, when it is about the requirements for stability and real time
processing without long GC delays.

This makes me wonder why Erlang/ Elixir are used so little in the financial
industry? I have been working for a couple of financial firms and even a HFT
company but I have never seen Erlang being used, despite its apparent
advantages.

Does anybody know why this is? Or is Erlang actually being used a lot in
finance and have I just been working in the wrong places?

~~~
user5994461
A betting exchange is a stock exchange. Same thing.

Let's say that there is a football game Liverpool vs Manchester.

One client puts £5 that liverpool will win. The exchange takes the money and
adds it to the order book.

Another client puts £5 that manchester will win. The exchange takes the money.
It's matched with the opposite order and frozen. The money will be
redistributed to the winner at the end of the game, minus a fee.

It's exactly like buying and selling in a normal exchange. You can do real
time trading during the game and arbitrage between exchanges.

Disclaimer: I worked in betting before I moved to finance.

Erlang is super weird and hard to recruit for. When used right, it's not
faster or more productive than C++ or Java. I wouldn't advise anyone to use
Erlang for anything.

~~~
fractallyte
_> I wouldn't advise anyone to use Erlang for anything._

Try concurrency on multiple cores in C++ or Java, and let us know how you get
on.

I'm quite happy and doing well with my 'super weird' secret weapons, thank
you!

~~~
pjmlp
By making use of tasking libraries that took ideas from Erlang, and
async/await patterns

------
eternalban
That was poorly written. And amazingly, the author neglected to shed light as
to why Erlang has a more a 'complete' story in regards to distributed f/t
systems ("reliability", "concurrency").

Erlang has OTP, Go does not. Erlang is FP, Go is not. These are the critical
differences.

Silver lining (in context of OTP) for Go is that the du jour 'cloud' based
architectures will provide the OTP like capabilities ("elastic", "fail-over",
etc.) for deployed (micro-)services. It seems that in the (future) long-term,
Go is better positioned than Erlang for cloud based systems.

------
tybit
Would be curious to see Pony added to the mix. It has the best of both worlds
for my tastes, from Go: statically typed and native code, and Erlang: actor
model, GCd at the actor level. Plus more features that neither has like
capabilities.

~~~
digitalzombie
Pony isn't even version 1 yet iirc and instable API.

I mean would you like to bet on that stack on top of your betting platform?
>___>

Pony promise great things and the language have a proof and all that but it
still needs to be battle tested and get to some sort of stability.

[https://www.ponylang.org/discover/#what-is-
pony](https://www.ponylang.org/discover/#what-is-pony) (under why not pony
section, API unstable)

The degree of tech hype and adoption of bleeding edge is crazy. The thought of
programmers just betting on these thing without thoughts on the schmuck who
would inherit these technical debt still boggle my mind.

~~~
holydude
That just means most of the software is not that critical or people are more
careless i do not know.

------
ilitirit
As someone who works in this industry, the biggest issues are (besides like
stuff like match-fixing and illegal internal trading/leaks):

1\. Highly domain-specific legacy pricing model running on... Excel
spreadsheets. These present huge performance bottle-necks and aren't easy to
migrate.

2\. Reliance on 3rd Party data. Did a scout just send incorrect data? Did
their device disconnect? Did they rollback the wrong information? Is the
source lagging behind the TV/online feed? GGs. Suspend the game and refund
bets.

The tooling really isn't that important. Scalable software can be written in
any software, and while some languages do make things simpler, often you can
just also find a suitable paradigm (or middleware) that can provide similar
advantages. Having said that, we do make extensive use of RabbitMQ (which is
written in Erlang) in our environment. I suppose this is also just an example
of my previous point though...

~~~
user5994461
1\. I don't know what you are talking about. The major betting exchanges all
support real time trading, they have market makers and are arbitraged against
one another.

The only issue is with low popularity events. Noone cares about them, there is
low liquidity and volume, it's hard to price them.

~~~
twic
I think ilitirit is talking about bookmakers, not exchanges.

~~~
ilitirit
There are companies that do a combination of things: Real-time trading
(manual, semi, auto), pricing (model-driven, or sourced), risk-management,
etc. They can either provide their own prices using models, or apply prices
from partners. The business is usually separated into discrete units which
feed into each other, and outward to partners/customers.

Yes, exchanges support real-time trading, but not every company has _all_
their models running in modern paradigms. It's not uncommon to find companies
that run multiple VMs for the sole purpose of... running Excel calculations.
Yes, this is exactly what it sounds like. A server that has a several, multi-
meg Excel files in memory with external APIs that plug values into cells and
read outputs. As I said, within the grand scheme of things, for sports who
still run on legacy models, this is a major bottleneck. Remember, many of
these companies were started in the 90s, and that's when these models were
created. The people who created them are usually swallowed up by the financial
industry and now come with a hefty price tag. So the question for the business
is "Do we spend millions recreating these proven models and run the risk of
them failing? Or do we simply just add another VM?"

At my company things have slowly started changing, but the new models (written
in F#) still need a lot of work.

~~~
user5994461
The bookmakers are far behind in technology but it's not critical to their
business. They only price once a day and have large spread (fat margins).

I agree that the finance industry is competing for talents of quants and
developers. Anyone who works in betting will find out that finance has better
work conditions and better pays.

------
gaius
I spent a few years working at the UK's largest betting exchange, I could tell
you what language we used for bet matching and so on but you wouldn't believe
me...

~~~
PaulRobinson
I remember when Betfair seemed to be a bunch of shonky crap stitched together.
I can easily imagine a load of Perl scripts before they invested in Oracle in
a big way.

~~~
gaius
_before they invested in Oracle in a big way_

Yep, got it: it was all done in PL/SQL. Clicks on the website went through a
simple Java layer and called stored procs. At the time we were doing more
transactions/sec than all the European stock exchanges combined, and often the
Japanese too. It makes me chuckle when people say to me now "but SQL databases
don't scale" when they are running 1/10th of the workload we were running 10
yeara ago...

No idea what they're using now, mind, I left after the IPO but before the
merger.

------
mbertschler
I am really interested in some numbers to show how much the Go's GC harms
program performance. I wonder if this is really an issue for an online
service, given the stop-the-world phases are usually <1ms. Does anyone have
some material on that?

~~~
fcbrooklyn
Sasa Juric, and the folks at pusher wrote a couple of blog posts on this:
[http://theerlangelist.com/article/reducing_maximum_latency](http://theerlangelist.com/article/reducing_maximum_latency)

------
PaulRobinson
What a terrible article. It's spam designed to get hits for bookmaker CTOs
scouting around.

No benchmarks, no numbers, no real detail, just empty praise for Erlang from a
company called "Erlang Solutions".

------
Thaxll
What a garbage article, Go is faster that Erlang and not by a small margin,
same for the GC. Erlang has definitely its place in some scenarios but the
article reflects it poorly by trying too hard to dismiss Go.

~~~
iamd3vil
Go maybe faster than Erlang in raw throughput but Erlang can compete with Go
in Latency (mainly predictable latency). As you said it all depends on the
scenario. The article is mainly in the scenario of betting infrastructure
where I think erlang is better due to OTP (saying this as a person who works
on building trading systems and daily user of both).

~~~
Thaxll
One example:

"When you pit Go’s requests per seconds against the likes of Ruby on Rails or
Django, Go returns some impressive benchmarks performing 3x better. Go can
scale to hundreds of thousands with relative ease, in much the same way that
Erlang and Elixir scale to millions."

He's talking about raw performance where Go is much faster than Erlang and
still talking about different order of magnitude to diminish Go.

