
How Primedice was exploited for $1M in Bitcoin - jbardnz
https://medium.com/@Stunna/breaking-the-house-63f1021a3e6d
======
jstanley
I don't think he did much wrong here. It doesn't appear that he actually broke
in to your computers, he just submitted lots of bets.

If you're going to run a casino (whose entire business model is based on
exploiting weaknesses of others for profit), don't be surprised if people try
to exploit your weaknesses for profit.

I'm not saying there's anything wrong with running a casino, I think it is
fine. I just think that what he did is fine too.

EDIT: Except the part where you apparently didn't pay out all of his winnings.
I'm not fine with that part. Also doxing one of your customers just because he
made a profit out of you.

~~~
fweespeech
> EDIT: Except the part where you apparently didn't pay out all of his
> winnings. I'm not fine with that part. Also doxing one of your customers
> just because he made a profit out of you.

They asked him to stop playing at that point and he created a new account to
evade that. I'm not sure why you feel people don't have the right to refuse
service.

~~~
s73v3r
It's not that person's fault that these guys suck at enforcing their own
policies

~~~
fweespeech
If you are explicitly contacted and refused service, its actually illegal to
do what the user did.

------
genericuser
"To understand how Hufflepuff beat our system, one must understand how our
provably fair system (RNG) works."

"Our database had seeds that were both inactive and in use at the same time
all connected to Hufflepuff."

To me it sounds like your company simply didn't understand how the system you
created worked, it sounds like you merely understood how you wished it to
work. Gambling has always favored people who actually understand the system in
place, by not understanding the system you yourselves were implementing you
created the opportunity for a customer to understand it better than you. What
you describe with your database having seeds that were both in use and
inactive at the same time, sounds to me like you did not do a good job
managing concurrency on records which were crucial to your business.

While I can not say whether the users did anything illegal, the post you have
created to call attention to this does not make an adequate case for wrong
doing on the users part. However your response does make an amazing case for
you having responded to an implementation issue in your software very poorly,
and I think would make future users who are aware of this less likely to use
your service.

That is just my opinion given the information you have presented, as you seem
to focus more on maliciously providing information of the person you are
accusing as opposed to proving your accusations that the person in question
committed a crime while doing so.

~~~
jballanc
I'd be willing to place a bet on what sort of database technology they _weren
't_ using. Hint: it has three letters, starts with an S, and ends with an L.
;-)

~~~
jsprogrammer
SQL won't force you to respect flag columns in your application layer (unless
your entire application is in SQL?).

~~~
nyir
From the article it sounds like constraints could have been used to prevent
having multiple seeds linked to the same account?

~~~
jsprogrammer
Yeah, I guess it depends on how good your constraints engine is versus what
your constraints actually are. If your actual constraints can't be modeled, or
if you choose not to model them at the database layer (e.g. for performance
reasons) then you have to implement the constraints somewhere else.

I worked on a database that allowed (almost) every table to mark a row as
deleted with a boolean-type column. This caused problems when you wanted to
create a new row that had the same values for the table's key-columns as a
deleted row. You can't just add the deleted column to the key and have all the
functionality you want (multiple deleted rows with the same values in the
table's key-columns). Each table could use (or not) the flag in different ways
and there was no simple way to enforce consistent behavior across all tables.
The constraint has to be placed somewhere else, either in explicit code, or by
creating new abstractions inside the database that allow you to represent the
actual constraints of your application.

~~~
ewang1
There are benefits for marking a row as deleted w/ a boolean column though.
Undo-ing deletes would be easy to implement, and retaining references to
deleted records would be simple as well.

The solution the problem you described would be to use a surrogate key column
(typically a UUID/auto-increment), and not natural keys.

~~~
jsprogrammer
There are reasons for everything in the database. Undo and historical records
were the primary use case.

All tables had a unique, integer primary key.

However, if you want to enforce a uniqueness constraint across your data [eg.
UNIQUE(name, location)], the constraint breaks when you introduce the boolean
deleted column [and UNIQUE(name, location, deleted) does not provide the
appropriate semantics]. The application semantics must be provided at some
other level than SQL column constraints.

------
imaginenore
> _and time again to investigate and each time our developers could not find
> any wrong-doing._

> ...

> _This was done by sending it more requests than it could handle in a small
> time period, think hundreds of requests in under a second_

Sounds like your developers really messed up. The server logs would be the #1
place I would look. Where else really? How do you not notice hundreds of
requests per second (from the same IP I assume)?

~~~
disordr
This was my question as well. How do you not notice 100's of qps from the same
IP, and then go on to say, he was betting $8k/second. server logs, and basic
networking chomps seems like it would have found this much quicker.

------
Lazare
I'm reflexively sympathetic to the small, scrappy startup. But...

...man this is not a sympathetic story. You're running a gambling site
(morally shady and an obvious magnet for abuse), using bitcoins (another huge
abuse magnet), you had very supicious gambling patterns (a massive red flag
that no real casino would accept), and you just let the guy keep playing?
That's not how you're supposed to do that.

And as for the whole "we had a timing bug in our code, and we couldn't find it
even with hard proof that it existed, and then we finally thought we'd fixed
it, and then we _asked the guy for the million dollars back_ (??!?), and then
it turned out we hadn't fixed it at all, and he hit us up for another few
thousand"? Like, that just sounds screaming amateurish.

If you can't be trusted to write secure code, or at least fix the bugs you
find in it, maybe online bitcoin casinos aren't for you? And if you think
asking people nicely to give you your money back works in casinos, maybe you
don't understand the industry?

Edit: I'm not trying to be a dick, but I feel like the proper blogpost to
write would be a grovelling "hey guys, I know it's super obvious, but if
you're doing an online gambling website, monitor your transactions for
specific patterns. Know what a real punter looks like, and aggressively
throttle anyone who doesn't behave like one. Otherwise you'll be stupid idiots
who lose a million dollars over a stupid bug, and then have to write a
magnanimous letter congratulating the guy who exploited you for winning so
much money (man that was painful to write)." You made some huge mistakes, and
it doesn't sound like you really learned from them, or even identified them.
Hint: Your core mistake was not a timing bug that emerged when your system was
under heavy load.

------
justonepost
Lol. They (as any sane Casino operator would) should have booted this guy
immediately on day 1, claiming they felt he was scamming the system. Sure, pay
him out, but don't let him keep betting. Yeesh. I have to wonder if this was
an inside job and they let him siphon those winnings. I am highly suspicious
of most bitcoin companies, and I wonder if they're all just shells around
vulnerabilities looking to siphon user deposits / investments.

~~~
danneu
That doesn't make much sense.

If they have a legit exploit, then kicking them will just make them create a
new account. Or perhaps distribute the +EV betting over a bunch of accounts so
that it was much harder to detect.

If they don't have an exploit, then you want them to keep betting and lose.
And if you have no real basis for kicking them, then, what, your casino just
doesn't pay out people that win big?

The casino's only real option here was to discover the exploit.

~~~
justonepost
No, pay him out, but just don't let anyone bets that way continue betting.
Betting that rapidly has to be a huge flag for even the dimmest of sys admins.

~~~
scintill76
What is "bets that way"? It seems like initially all they knew was that some
accounts were unusually active (betting every second, for hours) and
implausibly lucky. They saw the flags, they just didn't know any useful way to
react (which is also incompetence, but IMO different than what you said.) (I'm
assuming a dumb rate-limiting solution would be off the table, as it would be
ineffective at stopping the attack, and/or lose money as other players stop
betting when they hit a limit.)

~~~
Lazare
> What is "bets that way"?

I dunno, maybe anyone who is clearly automated (ie, betting every second for
hours) and probably cheating (ie, implausibly lucky)?

> It seems like initially all they knew was that some accounts were unusually
> active (betting every second, for hours) and implausibly lucky.

Exactly.

> They saw the flags, they just didn't know any useful way to react

The useful way to react would be to not let them continue betting.

~~~
scintill76
But how do you reliably identify one anonymous entity? Like I said, putting a
rate limit is going to turn away legitimate bettors (if too strict) and not
make much of a dent in the attacker's profits (if not strict enough). Doing an
"implausibly lucky" check is going to piss off legitimately lucky winners who
might have given you more money (if the limit is too conservative), and/or not
hurt the attacker much. Any per-account limits can be circumvented by the
attacker making more.

Maybe they took some measures like this that they didn't tell us about. With
that much money at stake, maybe "do something, anything" does make sense -- my
point was there's no way for them to ultimately prevail until they
understood/addressed the root issue, and knee-jerk responses could backfire.

------
lcswi
Uh, that post includes some doxxing.

From an observer's view, after fully reading it and seeing how hostile their
response is, I can only applaud the player. To me it seems fair game. She
exploited a weakness in the game, something that in meatspace is often hailed
as genius. In bitcoin online gambling, race conditions are a part of the game
that has to be expected to be attacked.

With their doxxing they lost any bit of sympathy I had.

------
mmosta
The (costly) lesson learned here doesn't revolve around Primedice's RNG
architecture but the strict importance of testing.

We all face pressure to deliver but at the end of the day -like brakes on a
car- testing is one thing you should never cut corners on.

Although another 2 weeks of testing may not have explicitly exposed the
vulnerability, it surely would have offered a better baseline from which to
evolve better heuristics for the analysis of the exploit when it did occur.

~~~
andrewvc
Testing is important, but I don't think that's enough for this sort of
problem. What this needs is in-depth code review. Tests are only as good as
the tests you write, and if there was no test to account for this particular
issue nothing would have been caught.

For concurrency issues someone thinking real hard is as important as testing.

~~~
PhantomGremlin
_someone thinking real hard is as important as testing_

With one caveat.

The "thinking real hard" has to happen before and as the code is written.

If you try to catch fundamental design problems and misconceptions during a
code review, you're much too late. At that point there is a "status quo" i.e.
the existing code, and a defensive resistance to change, even if you can
demonstrate clear problems with said code. The end result is band-aids on top
of band-aids, rather than a proper solution to the problems.

Edit: the article itself alludes to band-aids:

    
    
       our developer had improperly patched the glitch
    

i.e. they attempted a quick fix, instead of understanding at a fundamental
level the overall mistakes in what they were doing.

------
tiatia
"We heavily explored what we thought was every possibility, ran simulations
and did the math and came to the conclusion that he was just incredibly
lucky." ROTFL

------
csomar
_Any information that leads to the return of the coins from this incident will
be greatly rewarded. We invite you to analyze the above bitcoin addresses and
find out where the bulk of the coins ended up if you have the skills._

What does this mean? Are they trying to _steal_ the thief (according to them).
Shouldn't they inform the local authorities and let them handle the case?

~~~
danneu

        > Shouldn't they inform the local authorities and let them
        > handle the case?
    

In which jurisdiction? The one containing the VPN IP addresses? Or the
internet police?

An open letter to the internet asking for help ends up sounding about 1000x
more effective than phoning this in to the "authorities".

------
baakss
Strange choice to put a picture from the Ocean's Eleven series at the top of
the article, seeing as how the thieves in those films are considered the
protagonists/heroes.

I think a bit more information included with the article to verify this was
actually a 'hack' would be helpful. I mean, was this akin to counting cards or
what? It's hard to tell from the article.

~~~
danneu
Since the player was making hundreds of simultaneous requests, it sounds like
a race condition in the casino's server-seed mechanism where the player was
able to get the server-seed of future bets.

Obviously, the server only wants to divulge the server-secret of closed-out
bets so that you can verify them as provably fair.

> Strange choice to put a picture from the Ocean's Eleven series at the top of
> the article, seeing as how the thieves in those films are considered the
> protagonists/heroes.

It's not so strange after reading a number of these HN comments.

~~~
baakss
I think the strangeness for me is that the article is written from the
casino's perspective. They went so far as to dox the 'culprit' and are
portraying him in a negative light in the article.

~~~
danneu
It looks like the attacker had some sort of celebrity status and fanfare
throughout all of this:
[https://bitcointalk.org/index.php?topic=843892.0](https://bitcointalk.org/index.php?topic=843892.0)
so maybe Ocean's Eleven isn't so far-fetched.

------
junto

      ... did the math and came to the conclusion that he was just 
      incredibly lucky
    

Woah. Red flag. Just incredibly lucky doesn't exist. Odds are odds.

~~~
csbrooks
I think they just mean, if you have 10,000 customers, it's not surprising if
someone hits the 1 in 10,000 odds. You could say that person was "lucky",
though they have no greater chance than anyone else of hitting those odds
again.

~~~
Lazare
They don't give us enough info to say for sure, but something tells me
Hufflepuffs winnings were a few orders of magnitude less likely than one in
10,000.

------
obstinate
It is really interesting to me how different Bitcoin users' understanding of
morality and the typical person's are. To almost any joe or Jane on the
street, exploiting a weakness in someone else's security -- be it an unlocked
door or an unchecked error code -- to acquire money from them without their
intent, is Wrong. On the other hand, taking retaliatory action, such as
providing what detail you can about that person in the service of returning
the ill-gotten gains, is right.

Judging by the contents of this thread, the Bitcoin/hacker community feels
just the opposite. How odd. I feel like this more than anything will impede
the mass adoption of Bitcoin.

~~~
mikeash
I'm not sure the difference is really there. The fact that this place is a
casino matters a lot. My understanding of typical morality is that it's
_totally_ cool to exploit a weakness in a casino's games to shift the odds in
your favor, and the casino's recourse is limited to trying to make this
impossible, and banning people who do it (while still paying out whatever they
won before they were banned).

However, it's not cool to break the casino's games from within. To make an
analogy, if you discover that e.g. a casino's blackjack decks have twice as
many 7s in them as they should, and you use that knowledge to make a ton of
money playing blackjack at the casino, that's legitimate. However, if you
break into the casino and replace all their normal blackjack decks with ones
which have twice as many 7s, then use that knowledge to make a ton of money at
blackjack, that's bad.

So the question is, which category does this particular attack fall into?
Apparently they exploited a race condition, wherein outcomes became
predictable if lots of games were played simultaneously. Is that exploiting
the game within its rules, or is that breaking and entering?

To me, this feels legitimate. An analogy to physical games here might be
finding a roulette table where the dealer under some circumstances lets you
place a bet after it becomes apparent where the ball is going to land.

~~~
PhantomGremlin
_if you break into the casino and replace all their normal blackjack decks
with ones which have twice as many 7s_

I've always wondered about this from the other perspective. Why can't the
_casino_ remove a few cards?

E.g. in Blackjack in a six-deck shoe the house edge is perhaps 0.5% to 2.0%
against a "pretty good player" (depending on exact rules). Remove a few face
cards and you would probably hurt the player by another 1% or thereabouts.

But you've now moved the house edge from 1% to 2%. Double the profit!

When is the last time you've ever seen any player at a casino ask the dealer
to reconstruct the six or eight starting decks that are in a shoe? I've never
seen that.

But I still do enjoy taking the occasional trip to Vegas.

~~~
dmurray
>When is the last time you've ever seen any player at a casino ask the dealer
to reconstruct the six or eight starting decks that are in a shoe? I've never
seen that

You see the decks fanned out, each card visible, before they get shuffled and
put in the shoe. It would require some sleight of hand from the dealer to
cheat that way. If you're going to do that, it's much less risky to rig your
slot machines or your craps dice or your Wheel of Fortune, where no one will
ever notice any wrongdoing short of an audit by the gaming commission.

------
pavel_lishin
> _Sorry for the long read_

Didn't feel long at all; it was concise and had enough detail to satisfy a
casual reader (me) with links to more detail if I wanted it.

------
scintill76
Some pretty hilarious YouTube comments on the linked video[0]: "could you
please share your strategy in that video", "Please, Send your method of play
and programs for this need". Sorry, but I'm pretty sure nobody's going to
share their bot for making 2000 BTC in an hour.

[0]
[https://www.youtube.com/watch?v=lSLXv5Tz1ZY](https://www.youtube.com/watch?v=lSLXv5Tz1ZY)
, linked from "$8000 worth of bitcoin every second for hours on end"

------
curiously
So bitcoin casinos are legal ?

