
Stockfighter is live - jzig
https://www.stockfighter.io/
======
patio11
Hey guys -- the most resource-intensive part of the infrastructure (the game
master boxes) is currently overwhelmed by your enthusiasm. We're spinning up
more boxes for you, but it will take a few minutes.

~~~
tptacek
I BLAME RAILS.

~~~
patio11
Thomas and I have a running gag about this. Rails is holding up admirably. You
can still actually see the website. Our problem is presently that one player
implies 1+ ruby (not Rails) processes running bots and doing orchestration for
their levels, and that has pegged the CPU on all of our GM boxes and also,
possibly, saturated their network links as they hit the venue with... I don't
actually know how many orders have been placed already but I'm going to wager
a guess that it's "a lot."

~~~
joeyspn
> Rails is holding up admirably. You can still actually see the website.

Site is down now... (at least in Europe)

~~~
danielvf
Site has been mostly down for me in the US over the last ten minutes.

------
a-dub
This is part of the game. To beat this level you sweet talk the founders into
letting you help them fix their scalability problems. If you get the site
running in 3 hours, you get a bonus score of ∞.

~~~
zatkin
∞ score doesn't sound bad. I should start up an LLC and give away 99% of my
score to that LLC, for charity purposes.

------
netcraft
I can't get the first level to play, seems things are still overloaded. Really
looking forward to this though, hope you will post or email again once things
settle to remind us about it.

edit: this is the message im getting trying to play first steps, if it helps,
dashboard says "Trades.Exec() (Status: playable!)" (3:08 UTC)

> We couldn't start the level: This level has been locked by a different
> server. (Sorry if this happened accidentally as a result of a server crash
> -- it will clear up in an hour.) Refresh page to check if server is down?

~~~
patio11
I'm going to try to build a self-help tool for unlocking level instances.

Briefly: you get round-robined to a (healthy, importantly) GM machine the
first time you try to start a level and that association needs to be fairly
sticky. Once you've started the level, a piece of that box's memory is all
your own until the level ends. To prevent a single player from getting memory
on all the boxes we "lock" a level to a particular box. While the level is
running, no other box will open the same level for the same player.

The problem is when the GM with your box becomes unhealthy. Nginx will put you
in touch with a different GM, which could possibly mean your level is _still
running on a box you can 't contact._ At that point, you have to wait for
either that box to hurry up and finish dying, for me to do a redeploy, or for
one of the other GMs to figure "OK, that level appears locked but it's a very
old lock so it's unlikely it is legit. I'll unlock it for you and grab the
lock."

Levels generally die ~40 minutes after starting (but who knows with a failing
GM). The lock ages open after ~1 hour.

~~~
xfour
Perhaps it could be as simple as keeping a hash of levels tied to a box. When
your monitoring alerts that your box is unhealthy, run a process to remove all
locks associated with the dead box allowing the player to refresh and restart
the session?

Also, maybe a new error, is getting mixed in, response from /gm/levels
{"ok":false,"error":"exit status 1"}

~~~
Rapzid
I was thinking you could treat is as a distributed hash ring with key
shuffling, healing and the like ala Riak. Or a virtual actor cluster ala
.NET's Orleans.

You could use container migration to move people off dying nodes, or to
redistribute the cluster.

Ultimately though, it's probably a pretty lean startup and there is only so
much experience and effort you can apply to each area.

------
tptacek
Well, sort of. Erin and I are scrambling on last-minute emulator/compiler
bugs, but Patrick is letting people play with his trading exchange levels.

~~~
S4M
Raah, that was the part I was looking for, I wanted to use it to learn a bit
about assembly and code generation - I couldn't care less about HFT (sorry
Patrick).

Good luck!

~~~
tptacek
Things are looking good, but we're decorating the family Christmas tree
tonight, and so I took a break about 30 minutes ago. My current guess is that
our part will be up tomorrow early.

Just for anyone interested:

We're pushing a "trainer" tomorrow, which is essentially "one level" of the
CTF, so that people who have never looked at low-level embedded systems code
can get the hang of it, and so we can, you know, not blow up when the actual
CTF runs.

The trainer can do some things that the real CTF UI won't be able to do
(because they would make levels too easy to clear), so I'm just fine with
that. It's fun in a different way from the CTF.

The full CTF we'll post in a week or two. Ironically: it was pruning all the
real CTF levels out that screwed me this week. You go into thinking "all we're
doing is removing functionality, this will be easy", and by the end of the
week you're setting your computer on fire.

~~~
gknoy
Additionally, if you're especially anxious to be introduced to an assembly-
themed CTF, you can also go look at Microcorruption. I would wager that Thomas
and Erin's current project is "better", but Microcorruption still seems rather
good.

------
sergiotapia
It's seems the site is extremely broken, I can't navigate anywhere and
clicking links that have "#" href's don't do anything.

["handleListAllowedLevels", Object] blotter-bundle-dd01d8c9.js:1549 No level
specified in URL. blotter-bundle-dd01d8c9.js:1605 ["enterLevel does not think
it can reenter level: ", "first_steps", NaN, Object] blotter-bundle-
dd01d8c9.js:2602 ["handleEnterLevel -> startLevel", Object] blotter-bundle-
dd01d8c9.js:2556 ["Error starting level:", Object]

~~~
bm1362
It seems the launch was a bit premature, there are tons of errors. I think
this is a symptom of a single developer writing everything from scratch. I
respect both Thomas and Patrick for their opinions, but I think HN falsely
equates that to engineering expertise.

------
edanm
Congrats on launching!

I'm getting this error when trying to play the first level: "We couldn't start
the level: couldn't connect to the GM server to start the level. Refresh page
to check if server is down?".

I assume it's just too many people hitting the server at the same time, but
fyi.

------
shreyas056
Just out of curiosity, can you post the stats about load on your site and what
kind of issues did you face, would appreciate it.

~~~
patio11
Definitely something we'll do; later.

------
gort
Unable to validate my email. Earlier it wasn't working at all, now it tells me
I need to be logged in (but I am, I think).

Edit: Fixed for me now.

~~~
patio11
We had a bug in email validation (worked on staging _grumble grumble_ ).
Separately, our deploy script blows away memcached. Guess where email
validation tokens were kept.

I temporarily turned off bust-cache-on-deploy and the rapidly rising number of
validated emails suggests things are working now. Now back to more DevOps
craziness.

~~~
laarc
I wanted to give you a status update of what new users currently experience.
Or at least what I'm experiencing.

After creating an account, I was redirected to the front page with no
notification about whether it succeeded or failed. No emails have arrived.
When I try to create it again, I get "username already taken" and "email
already taken", so something worked. But when I try to log in, I'm redirected
to the front page with no notifications. In other words, login is failing.

To clarify, this isn't a complaint. Congratulations on launching, and
apologies if you're already aware of these issues.

Edit: This is similar to
[https://news.ycombinator.com/item?id=10724699](https://news.ycombinator.com/item?id=10724699)
but I'm unable to log in after clearing cookies. I get "Couldn't sign in.
Check username/password" for a correct password.

------
xfour
Seems to be pretty stressed out at the moment, tons of 500s and timeouts. 2-3
refreshes per page, got to the Level controls dialog, nothing doing.

We couldn't start the level: This level has been locked by a different server.
(Sorry if this happened accidentally as a result of a server crash -- it will
clear up in an hour.) Refresh page to check if server is down?

------
digitalnalogika
[https://discuss.starfighters.io/session/sso?return_path=/](https://discuss.starfighters.io/session/sso?return_path=/)
redirects to
[https://staging.stockfighter.io/](https://staging.stockfighter.io/)

~~~
patio11
Thanks; fixed.

------
dfc
Usernames must start with a letter? That is an odd requirement. I usually pad
my username with zeroes when sites dont let me choose dfc. Out of curiosity
why the constraint?

~~~
laarc
Databases are usually sharded by the first letter of the username. Devs like
to start their usernames with a number, which causes problems.

Also, usernames are injected into their code. Therefore they must be valid
identifiers. Don't name yourself exit() or ret, or you'll crash their servers.
Naming yourself nop will give you a distinct advantage relative to other
players, however.

I don't know.

~~~
artursapek
It took me too long to realize you weren't serious

------
lackbeard
UPDATE: Can log in now. Cleared browser cookies, but have no idea if that's a
coincidence or not.

I seem to have hit some kind of bug signing up:

1\. Filled out signup form, hit enter.

    
    
      > Page reloaded, sign-up form empty. No other feedback.
    

2\. Filled out signup form again, hit enter.

    
    
      > Told username and email address taken.
    

3\. Check email.

    
    
      > no new email.
    

4\. Filled out login form using info entered in step 1.

    
    
      > Page reloaded, not logged in.

~~~
tedmiston
Update 5 min later: Maybe not. I logged in and while playing First Steps is
listed in "Stuff You Can Do", clicking a level from Level Controls is instead
throwing errors in the console.

I'm assuming this is not part of the game.

\--

It looks like email invites went out about the same time as the HN post, but
it seems to be good now.

------
tedmiston
From the welcome email: "You can play with any tech stack which speaks HTTP,
or try doing everything by hand (or curl) if you like playing life on hard
mode."

Love this.

------
ohquu
Trying to start the first level:

"Couldn't resume the level because Couldn't connect to the GM server to resume
the level."

------
mason55
I'm on the instructions for level 2 and it points to the API docs
([http://starfighters.readme.io/](http://starfighters.readme.io/)) but that
URL is giving me a 404.

~~~
ShaneWilton
There's an extra 's', try
[https://starfighter.readme.io/](https://starfighter.readme.io/)

~~~
mason55
Yup, I got it now from the API docs link in the level. Seems like a typo in
the level 2 instructions.

------
tux3
Really awesome to see it launch!

It looks like the GM server is down at the moment, so I have yet to start the
first level, but I'm more than happy to mash the F5 key till it works.

~~~
ohquu
If we all mash F5, maybe it will start up faster!

~~~
tux3
I know what will help, writing a script to mash F5 faster and notify me when
it works.

------
yoodenvranx
Is there any deadline or is this a long-running project? I'd love to play with
this but I don't have any time at the moment.

~~~
patio11
We intend this CTF to be the flagship product of our company for the next
several years. If you can't play it today, no worries.

~~~
yoodenvranx
Thx for the info and all the best for your new company :)

------
thenomad
Congratulations on having the same sort of initial launch problems that other
very successful games have!

Seriously. There's nothing worse than preparing for scale and then discovering
you don't need it because there's not enough interest. Wheras I'm sure you'll
get these initial load problems sorted soon enough!

~~~
eterm
While you are correct that it's not good to over-engineer without knowing what
is needed, reading this mornings' update shows that while you shouldn't over-
prepare for scale and have it unused, this isn't mega-scale:

[https://discuss.starfighters.io/t/state-of-the-game-last-
upd...](https://discuss.starfighters.io/t/state-of-the-game-last-updated-
sunday-morning-cst/1568)

tldr: It took ~3000 game instances from < 5000 sign-ups (and therefore I
assume << 5000 concurrent players) to bring the game down.

------
davnils
You didn't do any load testing before releasing this on HN?

------
rcsorensen
The handshake agreement presented here makes me very happy.

Were many revisions done on it, or was it pretty clear to write?

~~~
patio11
I wrote it in one take 15 minutes before hitting Go.

------
ertemplin
It seems like your nginx proxy is having trouble... I keep getting the same ip
from DNS and nginx error messages. Maybe you should bring up some more and at
the very least use DNS to send requests round-robin to the proxies.

~~~
patio11
We only have one Nginx box at the moment. You might have caught it during one
of the Rails reboots I've been making in the last few minutes during
redeploys.

------
ShaneWilton
Congrats on launching!

If you're looking to dive into the problems more immediately, there's a wealth
of API clients and other resources available in the forums:
[https://discuss.starfighters.io/t/helpful-external-
tools/136](https://discuss.starfighters.io/t/helpful-external-tools/136)

Don't be afraid to try the game if you don't have much programming experience.
It's seriously a great way to pick up some valuable skills.

------
fchollet
Confusing, still pretty buggy. Maybe it's a bit early to release?

I'll try to compile some constructive feedback in a bit.

------
2oi4j3
How does the "finding jobs" part work? Are you aiming to help people get a
foot in the door at companies which would be looking to interview them, or
trying to skip the "technical interview" part completely? (generate interview
opportunities vs job offers)

~~~
zrail
[http://starfighters.io/](http://starfighters.io/) (click "how?" in the last
paragraph)

~~~
2oi4j3
hmm yeah I get that but still, what does an "introduction" mean? :)

~~~
zrail
Sequence of events:

1\. You play stockfighter and get the highest score this year.

2\. patio11 emails you and says "hey, you're awesome. are you looking for a
job?"

3\. You say "yes!"

4\. patio11 emails pc (CEO of Stripe) and says "hey 2oi4j3 is awesome, here is
proof (i.e. your stockfighter score)"

5\. pc emails you and says "hey let's talk about what job you would like at
Stripe, and let's bypass most or all of that technical interview crap because
we already know you're awesome"

6\. you and pc decide that you would be best on the X team, you start next
week.

~~~
notatoad
7\. pc sends stockfighter a bunch of money

~~~
zrail
Yes, quite right. I forgot that part.

------
blondie9x
I wasn't able to access the levels yet unfortunately. Is there still some
testing going on?

------
yblu
Does anyone know what tool is used to create the stockfighter API doc [1]?
Looks really nice.

[1]
[https://starfighter.readme.io/v1.0/docs](https://starfighter.readme.io/v1.0/docs)

~~~
vandot
It looks like the chose the three column option from
[https://readme.io/](https://readme.io/) .

------
paulpauper
We couldn't start the level: This level has been locked by a different server.
(Sorry if this happened accidentally as a result of a server crash -- it will
clear up in an hour.) Refresh page to check if server is down?

------
chenster
So much reading about what it does. Don't assume people already know how to
play a programming game. Most of us don't. Is there an intro video that at
least we can watch while you address the server issues?

------
div
[https://starfighter.readme.io/](https://starfighter.readme.io/) top
navigation "Stockfighter" link links to staging.

------
onslauth
Well done on getting it out there. Also well done on ensuring you get no time
off during the holiday season :D

Will be jamming it in the morning. Hope you guys get some sleep at least.

------
kuzmin
Wow, I never think I've had a confirmation email arrive so fast.

FWIW, I'm also hitting the rate limit exceeded error when trying to access the
"first_steps" level.

~~~
insamniac
same and same

------
airza
can't login with my migrated account. Here's what i get on pw recovery:

"The change you wanted was rejected.

Maybe you tried to change something you didn't have access to."

~~~
ShaneWilton
Try clearing your cookies. They flipped on the secure flag, and it borked a
few things.

------
SBCRec
Woohoo, Congrats on going live. I am excited to play around!

------
fastball
Are you aware that the API docs have the subdomain "starfighter" on readme.io
rather than "stockfighter"? Seems... strange.

~~~
omash
That's hilarious. Now they have "stockfighter" too ;)

[https://stockfighter.readme.io/docs/getting-
started](https://stockfighter.readme.io/docs/getting-started)

------
iso8859-1
> Our servers are under continuing heavy load due to having launched recently
> and the site being on the front page of HN. This is causing poor performance
> and undefined behavior.

How is that possible? As I understand, undefined behaviour is either there
when you compile, or it isn't (it is a function of the source code). So if the
undefined behaviour is a function of traffic intensity, I assume you compile
different C code base when you have high load. I never heard about something
like this before.

~~~
cube00
It could be around the high load causing errors that are causing unexpected
flow on effects. For example, an earlier thread talked about how you're locked
to a specific server in the pool and if your server goes down, you can't
continue because no other server in the pool will talk to you until the lock
expires. Having said that I'm getting around 4 different errors randomly so it
could be a number of things playing out in undefined ways.

------
samavirya
Awesome, I've been looking forward to this! Signup went smoothly for me and
I'll jump into the first level as soon as I can.

------
ufmace
Cool! Sounds like there's some teething problems right now, so I'll put this
on the check it out later list.

------
DGAP
I'm getting "Level Instance is not running" on the first level.

------
xfour
Getting a lot of 500 NGINX errors trying to confirm email or login.

------
ctmkpp
Congratulations!! Really happy for you Patrick

------
omash
Level 1 complete!

------
purpleidea
Nice try, HFT company that wants free algorithms.

~~~
daveguy
Except the trading interface is an API. You can keep all your precious
algorithms on your own computer.

------
dbancajas
I am not a devops but seriously curious, wouldn't this scalability problem
they are experiencing been solved already by using Amazon EC2 and stuff?

~~~
airza
not sure if you are joking, since they are using ec2 afaik

~~~
dbancajas
okay. so isn't the purpose of that is to auto scale based on system load? Am I
missing something?

~~~
freyr
Yes and no. According to Amazon, _" EC2 is a web service that provides
resizable compute capacity in the cloud."_ Really, an EC2 instance is just a
virtual computer. You can, with the click of a mouse, create a new EC2
instance and load a pre-defined system image onto it. This allows you to,
rather quickly, manually increase or decrease the number of EC2 instances
based on load. Automatic scaling of EC2 instances can also be done, although
it would be provided at a higher layer.

However, a more critical factor is whether the architects designed their
_systems and software_ to scale effectively when the number of EC2 instances
is adjusted. If there is a bottleneck that is not properly addressed by adding
EC2 instances, then adding EC2 instances will be pointless.

------
markbnj
Suggestion: deploy your GM server as Docker containers on kubernetes and auto-
scale while you enjoy some fine wine in a casual atmosphere. :).

Seriously, good luck. Success hurts sometimes.

------
rifung
Is this meant for people interested in security? I ask because the main site
says you guys are making CTFs to replace interviews, but I'm not sure if
that's interviews for security roles or all developers.

I'd be interested in either, but for the moment rather not add yet another
thing for me to learn

