
A Developer's Worst Nightmare (The TinyGrab Story) - nam3d
http://cocoacoding.com/2011/04/03/developers-worst-nightmare/
======
gm
If anything this is a cautionary tale about how to screw up your startup.

I have no idea what TinyGrab's architecture looks like, but I can say with
full confidence that backups were an afterthought, and they find themselves in
deep shit because they did not have a proper backup in place. It is
inexcusable to have the current situation and still claim they have good
backups.

"The issues lays with the server being compromised and being forced to shut
down. The old API and apps relied on a fixed IP address. It would have taken
us months to get TinyGrab v1 back up to scratch"

Ok, computer scientists reading this: To which of you does this make sense?
It's hard to keep counts of all the fails in here:

\- If your problem is a corrupted server, replace it and assign the same IP
address.

\- Why would anyone hard-code IP addresses for an API to work? Were you born
yesterday?

\- It should take at the very most hours to recover from a truly catastrophic
failure. "Months" you say? You do not know what you're doing, I say.

\- They have a bunch of backup and not one works, specially when there is no
data loss? Smells to me like no source control; no off site backup, and
absolutely no testing of the backup. If you have lived a single day in system
admin work, you know that it is not enough to do a backup. You need to test
the damned restore.

Maybe this could be a story about when _not_ to self-finance your startup. Get
outside funding and hire people that know more than you do about the
technology involved.

Sorry for being an asshole in my comments, but this situation is full of fail
at every step.

~~~
samengland
You said it - you are indeed being an arsehole in your comments.

Firstly:

"...this could be a story about when _not_ to self-finance your startup"

That's easier said than done. There is absolutely no point in over analysing
the situation and then concluding that TinyGrab should have got "outside
funding" and "hired people that know more than they do" about the technology
involved - because what is done, is done. You are seriously insulting the
intelligence of everyone behind TinyGrab if you think they don't already know
that version one was problematic. The point here is that the popularity of
TinyGrab was never predicted or planned and that it grew from strength to
strength entirely organically. At no point did they think about scalability
from the outset - so they acknowledged the problem and were rectifying this
with a full rewrite in TinyGrab 2. They we're simply unlucky in that they were
unable to roll this update out quickly enough - they were hacked before they
could complete TinyGrab 2, and therefore did the right thing and released
TinyGrab 2 which was very nearly complete anyway. With the lack of income and
voluntary manpower they had, they couldn't have changed anything in hindsight
at all. So in that respect this was not "full of fail" as you so crudely put
it.

Secondly:

What you need to do is put yourself in the shoes of those battling to actually
keep TinyGrab online, and realise that they are actually trying their best to
respond to every support request and resume the excellent level of service
their users have come to expect - and frankly they've almost achieved that
now.

What you also need to realise is that those behind TinyGrab are not in it for
commercial gain - they are in it because they love the product and their
users.

Since this is a completely free service they could have very easily - and
might I add completely legally closed TinyGrab as of today, with no risk of
litigation against them. They would also most probably refund existing premium
users, but would not be legally obliged to.

What they have instead chosen to do, is keep this service online, do their
best to get TinyGrab 2 to the bare minimum in order to roll it out completely,
and work day and night as a team to respond to every support request and
@mention on Twitter to keep users informed and solve their issues
individually. I think this is absolutely admirable given their 300,000+ user
base and could never be described as "full of fail" especially since the vast
proportion of their users do not pay them a penny - and especially since those
behind TinyGrab work in their own free time on the project whilst receiving no
commercial gain whatsoever.

TinyGrab will easily recover from this over the coming days - all they need at
this moment in time is some moral support and some understanding - a bit of
slack if you like - not a bunch of anonymous, cynical, patronising and moronic
nobodies trying to tell them how it should have been done in hindsight. As I
said above, it's frankly just insulting the intelligence of those behind it -
any reasonable person can see that they were fully aware of the pitfalls of
TinyGrab version one and were trying their best to roll out version two. Give
them a break.

~~~
gm
Maybe the problem is that I _have_ been in these guys shoes (anyone that has
been awakened at 3am with your system completely off line and bleeding money
raise your hands).

The fact that it is a volunteer effort is pretty much irrelevant. One does not
make worse decisions when one volunteers his/her time.

Rather than nitpick your points against my "full of fail" comment I will say
this: The people involved had several choices to make in all of this, before
and after the hack. Pretty much the key choices were mistaken: Not having a
good backup in place. Not giving a second's thought to how to restore. "Taking
the opportunity" to perform a major version change (to an unfinished version,
no less) while the system was completely down. And on and on and on.

I really don't get why it matters if it is a volunteer effort or not.
Volunteering does not mean "do low quality work", nor does it mean "not
mission critical".

Had this all been handled differently, it would have been a "why we were doen
for a couple hours yesterday" blog post instead of "why were barely working,
and will be for the next few weeks".

------
michaelcampbell
"TinyGrab was attacked and the attacker was able to gain access to one of
their servers. Although the details are scarce, the attacker corrupted the
codebase for TinyGrab 1. The TinyGrab team was forced to migrate their entire
userbase to the newest version, a version which wasn’t completely ready for
yet."

I didn't find any mention of it in TFA, but do people just not do backups
anymore? Or rolling, dated backups? My wife owns a small retail/web store, and
I backup her Quickbooks, the database, and the source code nightly, and keep
at least a few weeks of those backups on the drive where it's stored, 2
different backup drives, and and offsite place.

Is this overkill? Perhaps, but it would have saved us from _that_.

~~~
maukdaddy
Apparently part of Web 2.0 is deploy and forget? How could no one have at
least a snapshot of the code and database?

------
rll
I must be missing something. How is it possible that they don't have umpteen
copies of the sources checked out of whatever source control system they are
using? Even if the central repository was destroyed, every developer should
have a more or less recent version checked out somewhere.

~~~
zacharypinter
Looks like a PHP site. I'm guessing they weren't using version control and
were instead just editing the files over FTP.

~~~
Androsynth
so php == ignorance? are we really going to make that assumption? thats just
foolish.

~~~
MichaelGG
No, but PHP easily allows a "edit source on server" as a deployment model.

~~~
random42
Seriously, You dont even need a security attack for losing the changes with
this model. A nice little "harddisk" failure would do the trick for you. :)

------
chrisleydon
We have more backups than you can shake a stick at. The issues lays with the
server being compromised and being forced to shut down. The old API and apps
relied on a fixed IP address. It would have taken us months to get TinyGrab v1
back up to scratch, or a few weeks to perfect 2.0. No data was lost, what
would you choose?

Chris Leydon TinyGrab Founder and Project Manager chris@tinygrab.com

~~~
enko
Now I'm even more confused. What was stopping you redeploying the server side
app onto a new box and changing the hardcoded IP over to the new machine?

~~~
magic_haze
Exactly. As a lot of other comments have mentioned, using URLs would have been
a better idea, but its irrelevant now. Assigning the same IP to a different
machine would have been a perfectly acceptable (and most straightforward)
solution. I fail to see why they didn't choose to do that.

We're all interested here in learning from tinygrab's mistakes, not criticize
them after the fact. Why patronize us with that "would have taken _months_"
quote instead of giving some technical details about what really the problem
was?

------
acangiano
They did an extremely poor PR job. My account is no longer working. I would
expect an email of apologies that explains the problem and buys them time.
Instead, I got an account that mysteriously doesn't work anymore, and a
password reset link that claims my account doesn't exist. I had to learn about
what happened from a random HN post.

~~~
chrisleydon
If you can shoot me an email, chris@tinygrab.com, I'll deal with your account
and help get you set back up again.

~~~
acangiano
Thanks Chris. Just sent an email.

------
philjackson
"Like all premature babies, TinyGrab 2.0 just wasn’t ready for mainstream
usage"

What the fuck?

~~~
markkanof
Yes, this seemed like a particularly offensive metaphor. I suppose it works at
a logical level, but to compare a human life to a piece of software seems kind
of extreme.

------
randrews
I used to use TinyGrab all the time, it was a pretty indispensable part of my
workflow, which is why, after nearly a week of it being dead, I got fed up and
wrote my own:

<http://dl.dropbox.com/u/651972/ScreenDuck-0.1a.zip>

It's obviously very rough right now, but it actually works. Later on I'll
write a website for it and put it up on screenduck.com.

~~~
pauldino
This is the first I've heard of TinyGrab, but for Mac users looking for
something else Cloud seems to do the same thing (that is, it can automatically
upload screenshots and put the URL in the clipboard for you).
<http://getcloudapp.com/>

------
cheez
Why wouldn't they just put the old code base and fix the security hole? Why go
to a new code base? Now you have two problems!

------
aneth
This situation makes no sense at all based on the explanations here by
chrisleydon.

You have backups of the code - can you explain why it isn't a pretty
straightforward process to set up a new server? The fact that you have a fixed
IP address has no importance as far as I can tell, and that's the only
explanation offered.

> It would have taken us months to get TinyGrab v1 back up to scratch

Really?

