

I Broke Justin.tv - mjdipietro
http://blog.socialcam.com/i-broke-justintv

======
jrockway
_Finally, a few months into the job, I wrote “break” where I meant “continue,”
and caused our payments servers to shutdown one-by-one over the course of
several weeks._

So, they don't write tests at justin.tv, and they don't do automatic
deployment? Sounds like a great place to work at...

~~~
abstractbill
There seems to be a general assumption that writing [unit?] tests is always
the "right thing" to do, even at a startup, and no matter what the code in
question does. In my experience that's absolutely not the case.

~~~
timr
Just to be clear: Bill's opinion on this isn't universally shared at JTV.

When you're two guys in a garage, unit tests probably slow you down. But at
some point, they begin to dramatically speed things up -- working on someone
other person's code becomes orders of magnitude easier when there's a
comprehensive test suite to validate your changes.

The problem is that at some point between first employee and a dozen (or more)
engineers, you start to wish you'd spent a bit more time on testing code.
Writing those tests from scratch becomes "a project", and it only gets done
with herculean effort.

~~~
jdeseno
Having been at JTV, looking back it almost seems insane to try to do as much
work as we we're without good coverage. If it matters, it should get a test.

I think a lot of people also see tests as slowing things down but, they also
provide a lot of value in documenting intentions, make upgrades easier, etc.

~~~
timr
It also matters a lot more when you're hiring relatively new/inexperienced
people, and throwing them in the deep end. I disagree with Bill mostly because
I see the purpose of testing differently than he: the point is not to catch
All Of The Bugs; the point is to make it _easier_ to confidently work on code
that you didn't write.

There's a lot of counter-intuitive stuff that happens in production code for a
big website, and it's not totally fair to throw inexperienced people at it
without at least _some_ safety belts in place.

~~~
abstractbill
Yeah, this is a fair point.

I think, as much as I dislike pair-programming overall, that's something I
would prefer to use as a safety belt here (because it's only expensive for the
period during which a new employee is still learning). Pair programming hasn't
ever been tried at justin.tv either though, as far as I recall. I'm probably
going to give it a try at ZeroCater for new engineers actually (we're
hiring!).

------
rglover
Cool to learn that a bigger site tosses programmers into the ring from the
start. What's even better is that they don't have meltdowns over mistakes and
make sure every event is a chance to learn and improve rather than be
concerned about losing one's job. There should be more of this in the dev
community. I think I'd be much more comfortable in an environment like this
where I'm being challenged as opposed to spending 3-6 months working under
someone else and only half getting it. It may have _some_ implications on
their business, but they're confident enough to figure out a fix so more power
to them.

~~~
dfxm12
There is a middle ground where you work with some controls in place that don't
bring the whole production system down. Fat fingering something (as described
in the article) can happen no matter what your experience level is. More than
showing that they treat new hires well, I think this reflects poorly on their
dev environment...

------
freebsd
I followed Dreamhack's streams on Justin.tv and it was really an awful
experience for me, with several downtime right into the live performances and
such.

I do remember someone from the staff coming into the overcrowded channels and
expressing their astonishment feelings toward the fact that the chat was
actually standing and surviving the load.

Besides that, some phrases they translated into my language are just
preposterous, usually it's just better off to do not translate a site at all
if you manage to do it in such a way.

I did read all the comments where they've pointed out their many improvements,
i hope it will get better and better since a platform like that has great
potential.

~~~
wildmXranat
I subscribed to the NASL stream and paid $25 and suffered a similar awful
experience related to video quality. Any questions on how to fix it on their
chat was met with anger from mods, with answers not to spam even though I
asked once or twice the most, and now my comments are getting downvoted in
this thread. I really liked Justin.tv and had intended on paying for more
streams, but it seems like a big joke to me as far I can see.

~~~
emmett
We are always working really hard on quality of service for video. For
example, note the button we have added to report video problems.

The biggest problem here is measuring quality - none of the obvious metrics
really correlate.

That said, sorry you had a bad experience and we will strive to make sure in
the future you have a good one.

~~~
dsl
Go to YouTube and play a video. Right click and hit "Show Video Info." Right
click again and do "Copy Debug Info."

The top left of the video and your clipboard should give you a pretty good
idea of the metrics you should be tracking.

~~~
abstractbill
I would be very surprised if the most appropriate metrics were the same for
static and live video.

------
websymphony
Really good article. Purpose of article is to motivate talented people to join
Justin.tv and I think it does that very well. First thing I did after reading,
was to look for the available positions and actually found one that excited
me. Didn't apply though. Why you might ask. Although article promotes the
"doing" more than "doing absolutely right and best way". Job description is
asking for more of "doing right and best way" than "getting it done". And I
don't blame them for it, as they have successful product and they would like
to keep it that way. What better of doing that than hiring best of the best.
For now, I am just going to wait and work on improving my skills to that
point.

------
wildmXranat
I would like to see how many happy OSX users are on Justin tv. It feels like
they didn't test for usability at all. I asked them how it could be possible
to code a player that is capable of playing HD flash at 1fps. Yes - 1 f p s,
whereas other sites can play it fluidly(1). It's as if breaking stuff is their
motto and answering user questions on how to solve it or explain why it
happens is not.

Overall, their product is good enough to make me pay for a stream or two. But,
as a user used to free video, that eventually pays for yours and you can't
deliver, - ouch.

 _edit: (1)_

Downvotes, pfft . What a joke.

~~~
emmett
We use almost 100% OSX at the office (with smatterings of Ubuntu) so I have to
say: yes, we have a lot of happy OSX users. If anything it's the IE users on
Windows who should be annoyed.

I watch from home on my macbook air constantly and from work on an old mac
mini, so I don't think it's a general OSX issue. What kind of hardware are you
on? Which OS version?

~~~
wildmXranat
Explain why my streams at 720p and up are unplayable on my top end macbook air
10.6 . I have asked how to fix it via Justin.tv support and had no response.
Honestly, I would not give a shit, but since I paid $25 bucks, I would expect
to see what I pay for... uh ?

~~~
emmett
That's very strange. Maybe it's a flash version issue? Because I watch NASL
all the time on my air (which sounds like the same hardware as yours but maybe
slightly worse) and it runs great.

Well, flash drains my battery like nothing else, but that's just flash video.
Same thing happens to me on YouTube.

~~~
wildmXranat
Well, me too. I paid for the NASL stream and couldn't watch it. Surely, you
can understand that throwing $25 away without getting the service kind of
sucks. To contrast it against an HD stream from gomtv, these GSL games
stream/re-play archives beautifully without any stutters.

So I want to be a fan, a paying fan, but it's kind of a bummer that I can't.

------
TechnoFou
You will learn from the biggest mistakes, but you will damage a lot from them.
It's all about balance !

