
Stripe-CTF 3 Writeup - henrikm85
http://muehe.org/posts/stripe-ctf-3-writeup/
======
jeremymcanally
I got stuck on Level 3, partially because Scala is a little wacky (but I
learned to appreciate it!) and partially because their specs were super
terrible. They didn't mention it required substring matching or was case-
sensitive until I'd been working on it for 2-3 days and then was out of time.
Given the bad debug output (the head of a diff which often didn't show
anything and didn't include what term the failure was for) it was really
frustrating to make any progress.

And he's right about the servers and scoring; that, too, was frustrating. I'd
pass a test case locally then push and have it fail because it was slower for
no apparent reason. Overall, it was awesome up until Level 3. I wish I'd made
it to 4 because that strikes me as an actual distributed systems problem
rather than a minor tweak to an existing example like the others.

It wasn't a bad experience and the system they have to manage it all is cool.
I just wish a little more care had gone into making that level a bit smoother.

~~~
brown9-2
To be fair though, there was an update to the level 3 README published on 1/25
which made some of the requirements (substring matching) a bit clearer.

~~~
jeremymcanally
Yup, but that was kind of after I didn't have a lot of time to devote to it
unfortunately. They did clarify but only after I whined. ;)

------
ajdecon
I got stuck on level3 over the weekend and then had to put it down due to life
stuff. But it was still a ton of fun, and I learned a lot.

If anyone from Stripe is here, do you plan to post the original problem
statements and repos for the problems anywhere public now that it's done? I'd
love to get a look at level 4... :)

~~~
recentdarkness
[https://github.com/ctfs/write-ups/tree/master/stripe-
ctf3/le...](https://github.com/ctfs/write-ups/tree/master/stripe-ctf3/level4)

~~~
ajdecon
Ha, I missed that this existed. Thank you!

~~~
recentdarkness
Well it was published about half an hour ago ;)

------
mkaufmann
Great article, I am really eager to see the solutions in github. I guess there
won't be many with c++ solutions.

I only found the challenge two days before the end. My goal with the challenge
was not to get the best solution, but rather use it as an exercise to practice
the different languages and finding intuitive solutions.

Level 0 as you wrote was really just a two line change.

Level 1 was fun, because I tried to work with the existing bash structure as
far as possible and just replaced the hot loop with inline python (something
new learned). I found it interesting that this way I could directly inject the
variables into the python program without having to load them as env variables
(see [1]).

In Level 2 I did too much work, I did not think simple enough. I assumed that
the different test cases would differ a lot and that a static divider would
not help. My assumption was that the histogram of requests per second across
the different IPs would have a bimodal distribution[2], thus I used the
excellent "fast-stats" library to get a histogram and ban clients based on
that. The library even offers approximate histograms, so even if there would
be thousands of requests it would still scale.

To solve Level 3 with the given code required to changes:

-(i) Partitioning the data across the three server. I only loaded a subset of the files based on the server id (which was conveniently already available in the search servers) and

-(ii) Loading the files in memory so that they don't have to be read from disk for every search term. This was enough to pass this level. The code was too long for the gist but I can put it on github if someone wants to take a look at it. No fancy index structure required. I just scanned The files linearly on each request.

Mastering Level 4 was not possible for me, I tried working with the go-raft
library and got the integration with unix sockets working, but somehow the
system got unstable after a while. Fixing this was really frustrating and in
the end I gave up because I could not find the problem and time was running
out.

I congratulate all ~200 persons which also passed this last level. This was
much more difficult than the other ones!

[1]
[https://gist.github.com/mkaufmann/8716922](https://gist.github.com/mkaufmann/8716922)
[2]
[http://en.wikipedia.org/wiki/Bimodal_distribution](http://en.wikipedia.org/wiki/Bimodal_distribution)

~~~
fragmede
Mind posting your code for level3? I got that partitioning was the main trick
but I am too unfamiliar with scala to reassemble the responses properly.

~~~
brown9-2
Here is how I merged the responses:
[https://gist.github.com/mattnworb/8717911](https://gist.github.com/mattnworb/8717911)

~~~
necubi
Or a bit cleaner using lift-json:
[https://gist.github.com/mwylde/8724700](https://gist.github.com/mwylde/8724700).

------
dkhenry
I really enjoyed the parts of CTF I was able to work on. I didn't have enough
time to finish, but I got through the first two levels and the second flag (
gitcoin ) was actually really fun to implement. I got a gitcoin hasher up to
2M hashes/second on a single machine, but even with that I was no where near
fast enough to get any coins when competing against the rest of the field.

I limited myself to doing everything using only "C++11". I had to pull in
boost_regex since no one has the c++11 regex support, but I am really liking
the new standard and looking forwards to what we can get with c++14

------
bostik
I think this particular bit says it all. To get blazingly fast results...

> _I can say with great confidence that we all cheated._

But I disagree with the word used.

Pre-computing the hash-table and embedding that into the final binary to
eliminate relatively expensive calculations when it matters? That's not
cheating, that's effing _brilliant_.

Hard-core software engineering, that's what it is. There's also a saying that
the only people who play fair are those who don't know how to cheat well
enough.

Well done.

(edit: typos)

~~~
henrikm85
Op here, just wanted to say thanks. Of course it is not cheating in the sense
that I tricked the system to give me a better score than warranted but my
solution only works for this ONE dictionary so.. maybe it is not cheating but
using tighter assumptions than those which were provided by the original task
description :-)

~~~
bct
The problem description did say:

    
    
        # Our test cases will always use the same dictionary file (with SHA1
        # 6b898d7c48630be05b72b3ae07c5be6617f90d8e).

~~~
dclara
The cheat is that he used a separate process to build hash table for real time
look up, which is different from reading the data from the file at run time.

------
zoz
After many hours I failed to complete Level 4. How frustrating! But what a
great experience. I think Stripe did a great job at building a programming
challenge. By going through the challenge I learned something new in each of
the following: Python, Ruby, Scala, Go, Node/Javascript, Git, Hashes, DDOS,
Consensus Algorithms, raft, transactional logs, Inverted Indexes, BitCoin, IRC
Chat, and more. Thanks Stripe you made me a better programmer.

~~~
gdb
You're very welcome. Glad you enjoyed :).

------
Geee
I gave up on the last level, as I didn't have the time to commit. I'd say this
CTF was too difficult/too much work for me. Last time with smaller challenges
was more fun. The challenges were interesting in their respective subjects,
but maybe they should have been a little bit more condensed in scope,
particularly the last 2 levels. Although learning Scala and Go beforehand
would have helped a bit.

------
alexmic
I went up to level 3 and then work/life took over. Here's my approach to the
problems:

(1) Python. Used a set() for the dictionary. Got ~200 points. I was planning
to go back and implement this in Go to see the difference in performance but
never got around to do it.

(2) Python. I wrote a single-threaded miner which could do around 300K
hashes/s. My miner calculated 1M hashes (~3s), then fetched origin, then
continued until it got a hit. I stopped after the first Gitcoin.

(3) Node. My solution was very simple and pretty much like the OP's. I kept a
map between IP and count and always let through IPs with count <= 4. If the
count was > 4, then I let them through with 0.3 probability to keep the
backends working.

(4) I spend most of my time on this problem, largely trying to keep the memory
within the constraints. I started with Node and created a simple inverted
index of all words to their positions. Since we needed substring search I had
to loop through the keys of the index which was too slow. My next attempt was
to index all substrings but that used too much memory. My third, and best,
Node attempt was using a prefix tree to index all suffixes, effectively
building a suffix tree. This was quite fast, got ~2200 on local tests but was
still over the memory limits.

I then gave up on Node and moved to Python. I used Flask for the server and
built two solutions. A simple inverted index and a prefix tree (using datrie)
with all suffixes. To keep the memory footprint low, I stored file/line
locations as bit-shifted longs. Amazingly, the simple index and the prefix
tree performed the same (!), so I submitted the inverted index solution and
got a passing score.

Thanks for a great week Stripe. I learned tons! Looking forward to the next
one.

------
gatehouse
I bailed out on level 3 after about 10 total hours into the contest, as I see
was pretty common.

I really enjoyed level1. I ended up doing an ugly little modification to git
itself to mess around with the last few bits of the commit until I got the
desired number of prefixed zeros on the sha1:
[http://pastebin.com/RwTjjtTU](http://pastebin.com/RwTjjtTU)

------
ajtulloch
Looks great! Nice scores.

I focused on getting the minimal passing solutions (e.g. Level 3, the
distributed grep problem, only took ~5 LoC changes to pass, Level 4 was 162
LoC additions, 69 LoC deletions).

I wrote up my solutions at [http://tullo.ch/articles/stripe-ctf-
golfing/](http://tullo.ch/articles/stripe-ctf-golfing/) if anyone was
interested.

------
dreamdu5t
I really liked CTF-3 but I'm astonished at the amount of time people are able
to find for it. Just reading this write-up I kept wondering, "does this guy
have a day job!?"

I got to level 3 but then figured I had already spent my entire weekend plus a
day and had responsibilities like grocery shopping, going to work, laundry,
etc.

------
lunixbochs
Scored in the top 10 on level0 using a Bloom filter in C. Once you had an
optimal solution, getting on the leaderboard came down to luck. 90% of the
runtime of my program was operating system overhead that also happened on `int
main() {}`. Today I realized I could've compiled with -nostdlib or
-nostartfiles and halved that overhead - lesson for the future. If your binary
needs to run faster than 1.5ms, libc is the next thing to cut :)

I held the top leaderboard spot of level1 for a day or so (my hashrate varied
between 1-5GH/s) but was pushed out in the last few hours by pushrax and the
Stripe server load. Used an OpenCL + Go solution. Was a close race there by
all.

[http://bochs.info/~aegis/rounds.tar.gz](http://bochs.info/~aegis/rounds.tar.gz)
\- tarball of all the Gitcoin round repos if anyone wants to play with the
data.

------
gdb
Very fair feedback on moving the deadline. We'd debated doing it or not, and
ended up moving it in the spirit of distributed systems education. A lot of
people in the chat appreciated it, but that's obviously a selected segment.

So we know for next time, how strong are other people's feelings on this?

~~~
bct
I'm glad you did (but I wouldn't have finished if you hadn't).

More people learning & having fun is better. IMO people only complain about
the extended deadline because of its effect on the rankings.

The scoring system was an interesting addition over CTF2, but it also created
these complaints and silliness like people submitting the same code repeatedly
to get the best score possible.

It may have been better to only tell players what broad percentile group
they're in. There would still be an incentive to write faster code, without
the frustration of being edged out of position #20 because someone else's
submission was (arbitrarily) scored a few points higher.

~~~
henrikm85
I actually think there is a middle ground; just bump the deadline, allow
people to still solve it and get a T-shirt but freeze the scores and
everybody's happy.

Btw, I agree that the scoring needs to be consistent or less coarse grained
(percentiles) so that people don't submit thousands of attempts to get the max
out of their solution...

------
mathias
Added a link to your write-up and solutions to the collection here:
[https://github.com/ctfs/write-ups/tree/master/stripe-
ctf3](https://github.com/ctfs/write-ups/tree/master/stripe-ctf3)

------
kilink
I love his hash table solution to level0!

I would never have spent much time optimizing such a trivial challenge as
level0, but after seeing the leaderboards with such ridiculous scores,
curiosity got the best of me, and I had to figure out how people were
achieving such speedups.

I used a precomputed bloom filter to get 2641 points on level0, because I
couldn't think of another way to avoid doing the expensive processing of the
dictionary ahead of time. That solution simply read in the filter from a file
at run-time. I thought of embedding the data in the source code to avoid the
file I/O, but instead moved on to work on some of the later levels.

~~~
henrikm85
Op here. I have done a lot of work involving hash tables in main-memory
database research and therefore really like applying those optimizations. You
are right though that it might be overkill :-)

------
hansy
I read through this and was amazed by the OP's approach to the problems even
though I could barely understand 10% of the reading. How do I learn to think
and code like this? I honestly have no idea where to begin.

~~~
henrikm85
Thank you. What you have to keep in mind is that I do databases research for a
living so many of the problems involved in each challenge are very familiar to
me. My suggestion to you is to get a stronger systems programming background,
I figure that's what helped me the most.

------
jstanley0
I'm glad I participated in this challenge. It was fun to micro-optimize stuff,
and I'm really happy to have been introduced to Go. The last level was a
challenge, mostly due to Unix sockets and Goraft bugs and lack of
documentation, but I enjoyed the language enough that I went back and redid
level3 in Go, taking a top-30 score right at the buzzer. (Or where the buzzer
would have been had they ended on time, which they should have.)

My family doesn't appreciate my long hours at the computer, and I haven't
caught up on sleep yet, but I'll definitely do this again.

------
adwf
One recommendation I'd make, is to reveal all problems at the very beginning.

I logged in, saw that the first problem was nothing to do with distributed
computing. Also realised it would take me longer to submit the code than it
had taken to work out the answer... then I quit and didn't look back.

It's a shame, because levels 2 & 4 possibly look quite interesting, I might
have persevered if I'd known. Sadly, limited time on my hands means I
dismissed it too quickly!

------
femto113
I've blogged my extremely lazy solutions for the first 3 rounds. TL;DR is
level0 used a hash table, level1 found an existing utility, level2 used a hash
table. Grand total of about 15 lines of new code.

[http://www.codesuck.com/2014/01/stripe-ctf-minimalist-
soluti...](http://www.codesuck.com/2014/01/stripe-ctf-minimalist-solution-
edition.html)

------
carise
Nice article. I got stuck on level3 and perhaps got close to finishing it, but
there were a few unforeseen things that came up. I'm glad I did what I could
for this CTF, though. This is the first CTF I've done, and pretty much all the
languages were new to me. Definitely a great experience!

------
darklajid
I've got mixed feelings this time.

Disclaimer out of the way: No sense of entitlement here, every time I see a
challenge like this I expect to fail. I'm really, really happy that people
(Stripe, here) offer their time and resources to entertain us.

I'm a proud owner of the CTF2 shirt.

This time I couldn't finish level4. My takeaway is: I invested too much time
and shouldn't have done it. I guess I'll pass next time when a challenge w/ a
deadline starts.

Level0 was a nice warm-up.

Level1 was immensely satisfying and interesting, but the pvp challenge was ..
well.. dead. By the time real life let me join others were throwing GPU based
miners at the problem and traffic was constantly heavy anyway. So amazing
level, the arena type thing was imho not a good idea.

Level2: Here the problems began. Local tests and remote tests diverged, plus
this level was pushing you to use node.js. Will come back to this later. The
actual challenge was interesting, so my issues here were infrastructure and ..
language

Level3: This was absurd. Again, the challenge sounded interesting. The sample
solution you're supposed to fix came in Scala and you had some constraints
(time/memory). I had a couple of (locally) working solutions that never made
it remotely. After hours and hours I gave up and solved this challenge - by
executing grep for each request. Yes, I called grep and parsed the output,
didn't index a thing. That felt .. bad.

Level4: Didn't solve it. Utterly frustrating. The consensus (hah.. hah.. get
it?) seemed to be that raft (goraft) would be the way to go, but I invested
again hours (8? Probably a lot more) and didn't make it. Problems with the
deployment were mentioned on the channel again and again, goraft has even open
pull requests and recent tickets from the stripe guys, as far as I can tell.
Dependencies were a mess. Someone built the reference solution to this level
and found bugs in goraft (see Github, goraft issues)? I .. don't think this
was a good idea. Plus, forcing unix-sockets into the game (where everything
around it doesn't support those and hey - if you want to have a 'curl for unix
sockets' you end up finding another go project that just didn't work for me)
was really messy.

So, putting aside the fact that I wasn't smart enough to clear level4 (it was
obviously possible!): The templates came in nodejs/Scala/go. I knew a little
about nodejs, nothing about the other two. Ripping the provided framework out
was kinda possible, but a pain in the ass to debug (you could run a script
that installed stuff remotely - I tried solving level3 in Clojure and it was a
miserable experience). So challenging tasks plus new languages, plus deadline,
plus 'servers are overloaded and cannot answer': This was just way too much
for me. I still tried to solve level4 until the server fell over again and the
CTF was finally closed.

My take away: All the ideas, the problems, were cool and interesting. But I
guess even the Stripe guys were surprised to see the load/spikes/outages -
probably a consequence of the 'distributed'/'load handling' type of problems,
versus the 'try to break in' challenges last time. It hooked me with the
leaderboard, nice website, good problems. But it was an exercise in
frustration for me (and Scala/Go are _really_ not for me..).

That aside: Thanks a lot, Stripe. I cannot even begin to imagine how much work
you guys invested in the preparation and administration/maintenance. I do
consider the extension fair - and didn't gain a thing from that (was already
at l4 at that point).

~~~
cespare
> Ripping the provided framework out was kinda possible, but a pain in the ass
> to debug

This is very different from my experience. They give you a build.sh script
that does anything you want and a container with open access to the internet.
Anything you print to stderr is shown to you. I didn't use the provided code
for any of the challenges, and thought it was awesome that they didn't force
you into a particular implementation. It's just a reference to help you get
started.

And hackers staying up all night to write openCL gitcoin miners? That's just
awesome. I don't know how you concluded that pvp "wasn't a good idea". I
thought that challenge was just fantastic.

~~~
darklajid
To clarify:

\- Ripping out the framework caused lots of issues for me. Maybe my choice was
wrong? I actually (in spite of my complaints) used two 'interesting' languages
for myself. I polished my ocaml for the miner and switched to Clojure for
level3. The latter was hard, because the infrastructure was down a lot and you
already had a hard time getting past 'Kicking of your trial' messages - only
to fail because leiningen complains about the inability to verify ssl certs
for example. Try to fix it (doesn't happen locally...), push, fail to kick of
the execution of the build script.. Loop.

\- The miner: As I said, I loved the idea of that challenge. The problem was
highly interesting. My problem with pvp was mostly "There's no way to
compete". When I joined it felt already like installing a bitcoin miner on my
home machine. openCL based miners? That's changing the field of course and the
only way to participate at that point is to follow - and I'm running a laptop
here with an integrated Intel chipset for everything else. That totally
ignores the fact that I don't know a thing about openCL or CUDA and would've
started from scratch (as w/ Scala, as w/ go).

I certainly understand the fascination, but .. how many people _could_ compete
against each other? The implementation (gpu based) and the network (git pull
was sooo slow, same for push) made this inaccessible. At least for me.

------
ZenDan
I definitely learned a lot. I was able to pass all levels but unfortunately
didn't have enough time to optimize them, even though I would have loved to.
Thank you Stripe for putting this on! Looking forward to the next one!

------
Bootvis
I really had a blast doing level 1. I wrote my first real C program and it was
as blazingly fast as I hoped. After that level 2 was easy but like many others
I got stuck in level 3.

------
icanhasfay
Anyone else seeing links to empty repos?

~~~
philk10
"Note: I’ll add the source once submissions are off."

~~~
icanhasfay
Didn't catch that, thank you.

------
grey-area
Here are my thoughts: I managed to scrape through level 4 so saw them all -
this CTF was fun and educational. Level 4 in particular felt like a real
education (for me at least), and a great way to learn. Unlike many others I
was fond of level 0 because it brought up questions that even beginner
programmers should address - how to organise your data so that you can search
through it easily, which was in some ways a preparation for the other levels.
For the more expert than me it probably felt tedious or inconsequential, but
it was quick to solve, and I thought inviting for beginners and open ended.

 _Lessons learned for me_

Distributed consensus was really interesting; I'd never heard of it and
enjoyed reading about Paxos and Raft, even if I don't fully understand them
yet. Distributed searches can actually be faster than searches in one process
- I didn't believe this but as resources were limited on the server it was
true in some cases (depends how fast you can make your search, mine wasn't as
fast as Henrik's :).

Big scala projects take an aaage to compile - is it normally this bad in real
projects? Anyone care to comment? The remote was pretty slow, so that might be
why.

Golang needs package versioning! On level 4 there was a versioning bug with
differing versions of the go-raft library being fetched locally and on the
server which led to problems setting the heartbeat timeout of raft. That was
confusing and annoying.

 _A few notes for the CTF authors_

The dramatic variation between resources on remote and locally was a problem
in tests - I felt like that needed to be adjusted for as the local test
results were meaningless compared to the server. This particularly made level
3 and 4 difficult.

Code examples are great to get you started, but it'd be nice if there was also
a clearly specified API for each one which you were required to conform to, to
make it easier to use another language, and if the build scripts simply
launched one binary at all times - level 3 had two different styles of
starting the servers for remote and local for example, that was frustrating.

The varying test cases were sometimes frustrating, and were too varied and
randomly chosen. I completely agree with Henrik that you need small tests.
Each run should have say 5 tests which can be run individually (for speed), or
as a suite. The downloading of tests broke a few times for me and left me
stranded and constantly re-downloading new test data/tests seemed a bit
pointless. Proper logging output when tests fail is really important - level3
was particularly bad for this as the diffs were useless; I had to edit the
test harness to get decent output.

The use of the JVM on level 3 was pretty painful for me - compile times on
remote were in the minutes, and I couldn't even get it to compile locally even
after installing another JVM, nor on remote without compiling all the sbt
stuff - this one was probably my fault, and I just rewrote with golang in the
end. Explicit language independence for all the problems would be great.

Unix sockets felt like an unnecessary distraction which caused pain on level
4, it would have been nice if it used IPs, but perhaps for reasons to do with
Octopus that wasn't possible, and on a deeper level, being given broken code
to fix just felt a bit wrong and distracting - having slow code you have to
speed up is one thing, but having code which feels like it has been sabotaged
is another.

Don't bother changing the deadline even if people whine - the contest isn't
remotely fair anyway or an objective test, so it's better to have a deadline
and stick to it.

Oh, and thanks again Stripe, it was a blast.

The original code and some more writups:

[https://github.com/ctfs/write-ups/tree/master/stripe-
ctf3](https://github.com/ctfs/write-ups/tree/master/stripe-ctf3)

[http://blog.joneisen.me/post/75008410654](http://blog.joneisen.me/post/75008410654)

~~~
necubi
The reason the build times on level 3 were so bad was due to time spent
downloading libraries. Stripe seems to have set things up to use a different
ivy cache for each user, so every time you deployed code it had to redownload
all of the dependencies. For me, this tended to take a couple of minutes.
Actually compiling the Scala was pretty fast (< 10s for me).

So yes, Scala is generally slow to compile, but don't blame scalac for this
one :).

Personally, I didn't quite manage to finish 4. I had a working solution
locally, but didn't have the hours to debug why their remote environment
differed so much from my local environment. My submissions also failed
randomly about 4/5 times as the deadline got closer, presumably due to
overload on their servers. And the unix domain sockets indeed felt
unnecessary; I probably spent more than an hour just dealing with issues
related to that.

The format (particularly on 3 and 4) of being given pretty broken code also
sort of messed with my normal approach to code reading, which assumes that the
code is more or less ok. This was particularly problematic for me on 4, since
I had never before used Go and had a harder time figuring out what was wrong
and what was just Go weirdness.

All around, though, I had a pretty fun time. Even if it did suck up way more
time this week than I intended.

Note to Stripe: I would gladly pay $10 next time around if it could buy
more/better ec2 instances.

~~~
grey-area
_Stripe seems to have set things up to use a different ivy cache for each
user, so every time you deployed code it had to redownload all of the
dependencies._

Thank you. So normally you'd have the dependencies prebuilt, and it wouldn't
take long at all to deploy? That sounds more sane. The build time for golang
was < 1 second for comparison, and at first I couldn't get rid of the sbt
build due to confusion over start_server/start_servers being used to launch,
which was a bit annoying. I couldn't get their solution to build locally so I
just gave up on Scala, which is a shame, perhaps I'll try again sometime.

l4 was pretty hard, even just integrating raft in the time available, I
started adapting what they had, then switched to inserting raft - I was still
left with some issues, and it was humbling and awesome to see people scoring
so highly on it for me with things like in-memory dbs, but also great just to
read about the problem space, which isn't one I'd normally touch at all. So an
education even if you didn't pass, which was pretty chancy anyway in my
experience.

 _Note to Stripe: I would gladly pay $10 next time around if it could buy more
/better ec2 instances._

I wondered if that was part of the challenge, particularly on l3, it meant it
was harder to solve with just one instance, but yes it did cause frustration,
particularly because a working solution locally would then not work remotely
and scores were all over the place, even for the same code, due to contention
and the random tests loading remotely. It's probably really hard to set this
stuff up, and I look forward to the infrastructure posts to see how they did
it.

