
Charm shutting down because of unfixable kernel panics with Rails on Ubuntu - bittersweet
https://raw.github.com/gist/020171fce9ffca2b77d4/0b194a4bc566e2395833f57f8c11453ddbeb43a5/gistfile1.txt
======
whalesalad
What??? How is this even remotely possible?

I call BS. It's either a smoke screen for some other issue, or their app was
built back asswards. Either way, this post and the circumstances are
indicative that Amy and/or the Charm leadership has no idea what they are
doing.

And they were not even launched? Beta or something? The comment about opening
to the general public implies that they weren't getting that much usage. These
days it's hard to build something with a modern framework and decent hardware
that does not at least work for a closed-beta period. Add 100k to the mix and
honestly what the hell.

Amy/Tom run Freckle and do some other cool shit so why are they dropping the
ball on this? It's like getting a paper cut on your finger and going home for
the day because it's just too much to bear. Cry me a fucking river.

Why does this keep happening?! People are throwing cash into the dumbest shit
these days.

~~~
gizzlon
Who gave these guys & gals $100k?

Reading this must make founders striving to get investments furious..

~~~
ahoyhere
Yes, the other gentlemen are correct. This was all our own money. Which we
earned from scratch from our other products (all doing very well, thank you).
And, perhaps more galling, VCs approach us every month to try to invest (but
we're against it).

I'm sure shutting down a perfectly good and potentially very profitable
product will make lots of wannabes and aspiring entrepreneurs jealous. Luckily
I don't make my life decisions based on what other people would do in my
situation.

When you run a real business, with real customers, you quickly discover that
what sounds good, isn't, and what looks bad, is often the absolutely right
thing to do. Luckily, our early access customers for Charm heard & understood
& appreciated my email.

~~~
gizzlon
I'm sorry I jumped to the wrong conclusion. Not very sorry, but a little bit.

FWIW, it deserve respect to shut down something you built with your own money.

In hindsight, I think you would have gotten an entirely different response had
you said _"We can't run Charm with the team we currently have, there are
technical challenges we can't solve"_. Understandable and believable. But for
all the people dealing with problems like the one you blamed on a regular
basis, saying what you did just sounds like:

 _"We spent or life-savings building this great house! But it keeps running
out of toilet-paper, so we burned that s"#¤ down!!"_

~~~
ahoyhere
When running out of toilet paper causes your house to fall down and people to
lose money, your metaphor will actually make sense.

------
jtdowney
They seem to be over their limit with Github, pasted below as a mirror:

You probably noticed Charm had some nasty downtime a couple weeks ago.

Service quality is very important to us. If we didn't think we could do
better, we wouldn't do it at all.

We've spent very generously on sysadmin services and infrastructure (nearly
$100k of investment on sysadmin services/infrastructure alone). We hired the
best possible, and we splurged on a redundant, powerful, and expensive server
configuration from the beginning.

Now we've discovered that there's some kind of base incompatibility with
Ubuntu, which is giving us kernel panics which nobody can track down. Charm
has been plagued by mystery technical problems from the beginning, when we had
to backport from Rails 3.x to 2.x because of massive performance slowdowns
which even Rails Core members couldn't identify.

What this has really shown us is that, if we open Charm to the general public,
we won't be able to provide you with the kind of service you deserve. We are a
tiny team, and so far, we've had zero luck in our attempts to grow by hiring
developers. Problems which are small now will only get bigger.

There are a lot of things I'm willing to take risks with, but not with _your_
ability to provide support to customers for _your_ business.

And so it is with a very heavy heart that we will cease operating Charm from
Dec 15, for the foreseeable future.

You won't be billed again, and we'll refund your last payments.

We will gladly help you migrate your data out of Charm. Please contact us
directly (support@charmhq.com) for help.

Thank you so much for taking a chance on us, and sharing our dream for a
superior email interface.

I'm truly sorry to disappoint you.

Best wishes,

Amy

~~~
atomical
Paste the link in another tab.

------
sudhirj
For a 100k, they could have

1) Bought a ton of Heroku dynos with every single addon enabled. That would
have given them any version of Rails they wanted with Ruby 1.8, 1.9 or even
2.0; Postgres, MySQL, Redis, MongoDB, two different kinds of Memcache and
three different kinds of asynchronous processing.

2) Gotten fully managed servers at Rackspace with pro support for pretty much
any setup they wanted.

How is this even possible? The same application happens to hit bugs in the
Ubuntu kernel and Rails core that haven't been fixed all the way to 3.2.9? And
they have 100k to spend but couldn't be bothered to try different kernels and
distros? Really? Have they seen the AMI launch screen on EC2?

~~~
bibinou
3) hired two $5000/mo sysadmins for a year.

~~~
ahoyhere
I shared your comment with people who actually hire for technical positions,
and we all had a good laugh.

------
cheald
Wow, this is unprofessional.

Lots and lots of people run Rails apps on Ubuntu. Many more run them on non-
Ubuntu distros. There is not an epidemic of "welp, Rails magically crashes the
kernel, time to pack up and go home", and citing that as the reason to close
down after spending over $100k on infrastructure smells really, really funny.

"A poor workman blames his tools."

~~~
npsimons
_"A poor workman blames his tools."_

. . . but even a master carpenter can't build a house out of rotten wood. Of
course, to me, rotten wood is OSX and Windows (why anyone would build servers
on those instead of one of the BSDs or Linux, I'll never know; to me,
development is also incredibly painful on Windows and OSX), but Ubuntu is not
without issues. The rallying cry of "usability first!" is all well and good,
until things like taking shortcuts by way of binary drivers causes crashes.
Still, one has to wonder why they didn't just try another distro. If you think
your tools (or your "lumber") are the problem, try different ones.

~~~
cheald
Lots of people successfully run Rails on Ubuntu in production. I don't think
it's fair to cry "rotten wood" here, unless the Rails ecosystem is flush with
amazing systems wizards who can work around crippling flaws in the OS to run
their software.

~~~
npsimons
_I don't think it's fair to cry "rotten wood" here, unless the Rails ecosystem
is flush with amazing systems wizards who can work around crippling flaws in
the OS to run their software._

Agreed; however, if they were having problems with Ubuntu (or indeed Linux in
general), why didn't they try something else? That's the quick and easy way to
see if it's really the platform or your application that's the problem (quick
tip from my own userland developer experience: it's almost always not the
platform; the times I've had Linux crash on me I could always trace to one of
three things: 1) flaky hardware, 2) binary drivers, or 3) I was mucking about
in kernel space ;)

~~~
cheald
What really blows my mind is that they did a regressive rewrite to Rails 2
rather than learn to use profiling tools.

The whole situation just makes my brain stare blankly and wonder what on earth
was happening over there.

~~~
ahoyhere
Yes, because it's not like Thomas (my partner/husband) is a performance expert
or anything. And it's not like 3-4 Rails Core team members who tried very hard
to help us have ever heard of profiling tools.

My mind just exploded, too, due to your willful inability to read the details.

------
dazzawazza
If rails causes a kernel panic then this is really a bug in the kernel. No
user land app should ever do this. Why not try to scale on *BSD or Windows all
of whom have different kernel and scale just fine?

$100,000 and no one said "hey what about running it on <insert other os>"?

~~~
meaty
This is normal for Ubuntu kernels from experience. They are shitty. Various
server releases have given us hell. They're fudging them somehow.

We moved to debian after numerous problems and have had no issues since.

~~~
tedivm
Yeah, when I see people running Ubuntu for their production servers I always
assume it's developers with minimal system admin experience who just want to
use the same distro for their server as their desktop. I've got nothing
against Ubuntu, and have even used it to convert a few windows users over, but
if you're going to run a production server then install Debian on it.

~~~
davidw
I have been using Debian since 1996, and became a package maintainer in 1997,
a position I held for nearly a decade until I began to not have enough time
for it. While continuing to think that Debian is a first-rate effort, I got
tired of certain aspects of it, and moved to Ubuntu, lock stock and barrel. I
actually like to have my desktop system marching in sync with my server, so
that the packages are pretty much exactly the same. And you know what? Ubuntu
works pretty well on the server, and comes out with nice regular releases,
allowing me to decide, and gauge, time-wise, when to upgrade. Certainly, like
everything else, it has its defects, but by and large, I've had good luck with
it, as both a desktop and server distribution.

------
davidw
My business runs on Ubuntu/Rails just fine.

Maybe if they were a bit more familiar with Linux they would have done better,
or at least have saved some money figuring out what went wrong:

<https://twitter.com/amyhoy/statuses/189072763590418432>

~~~
viseztrance
That link was really insightful. The fact is that there are tens of thousands
of servers running Rails on Ubuntu without any issues.

Sounds to me they would rather blame Ubuntu than their inability to fix the
issue.

Later edit: to elaborate that last point. I'm not saying Ubuntu is not at
fault, but rather that problem solving goes with the territory. Given the wide
array of alternatives, this was not the sort of thing you could openly
complain about.

~~~
davidw
<https://twitter.com/amyhoy/status/94166175247908864>

Ouch.

~~~
npsimons
Wow. Hmm, how can I put this politely? As a desktop Linux user, I don't feel
particularly left out in being ineligible for the hiring pool for a company
that wouldn't let me use Linux. Maybe they scared away all the competent Linux
admins with their negativity towards Linux?

~~~
ahoyhere
Baseless speculation is one thing, when the original post actually contains
information which contradicts you, it's just a sign of laziness.

~~~
beatgammit
What do you mean baseless? There were two provided links showing your distaste
for Linux. One could be a fluke (had a bad day), but two shows a pattern.
This, added with the mention of using Ubuntu (and not trying another distro)
shows that Linux is not your forte.

I, for one, would not like to work for you. I'm not trying to be offensive; in
fact, I respect the decision to cut bait when you realized you were over your
head. I just don't think I, as a Linux user, would fit in your organization.
Many of the good sysadmins also have this same viewpoint, so the above
certainly is well founded.

------
facorreia
Does this make sense? They don't mention trying to get support from Engine
Yard on hosting their app on Rails without kernel panics, or trying a managed
solution such as Heroku, or trying other OSs other than Ubuntu such as CentOS
or something from Joyent. Just trying with the best possible syadmin they
could hire and "Rails Core members".

Anyway, kudos for not submitting their users to a service they're not
confident they would be able to provide with adequate quality and better luck
next time!

~~~
npsimons
_or trying other OSs other than Ubuntu such as CentOS or something from
Joyent._

Or even just Debian. I ran into issues (years ago) with Ubuntu stability due
to closed source drivers on a laptop; sure, I have the skills to debug that
sort of thing, but why waste my time? I had been running Debian just fine on
desktops and servers for years, and after switching the laptop to it, the
lockups stopped.

I'm more than willing to admit there may be real technical issues they ran
into, but it sounds to me like they gave up too easy, and we may never know
what the problems are. Would've been nice to at least have seen a bug report
(honestly don't know if they submitted any, but I'm willing to bet they
didn't).

~~~
beatgammit
Or Arch Linux. Arch packages are pretty vanilla, so it would be a pretty good
way to reduce the attack surface.

Also, with Arch, you get less by default, so you'll end up with a cleaner
system. I didn't realize how much stuff I didn't need on a default install of
Fedora until I set up an Arch system.

------
ivix
What on earth?

Blaming this on Linux or Rails is ridiculous. And as for 'mystery technical
problems'...

I think the reality is more like they got in over their head, decided it was
all a bit too difficult, and gave up.

------
electic
We run a lot of webservers, our own hardware, on Ubuntu and we do not have
this issue at all. Rails doesn't cause Kernel panics. It sounds like they have
a bad install or honestly, just don't know what their are doing.

Probably a bad idea to advertise you can't keep a simple webserver up.
Probably won't get funded ever.

~~~
jbigelow76

      >Probably won't get funded ever.
    

Amy Hoy is in the same bootstrapper camp that DHH and Jason Fried are in.
Regardless of the situation, I imagine "won't get funded ever" is ranks pretty
close to the top of her "don't give a shit list".

~~~
electic
Sure. But it also means a lot of people who are looking at her future projects
will pause and think twice.

~~~
ahoyhere
Think twice, like what? "Gee, there's a woman with integrity who shuts down a
service while it still works 99% of the time, and refunds my money, instead of
growing it like a nutjob until it has weekly downtime like oh I don't know
Desk.com"?

I do so hope you're right.

~~~
Anderkent
A service with a weekly downtime is still more useful than no service at all.

------
blrgeek
Let me see

1\. Open Source OS

2\. Open Source web server

3\. Open Source programming language

4\. Open Source web framework

5\. Some (relatively) simple customer support ticketing app [not likely to be
a system pounding Gorilla]

And they claim that they've not been able to root-cause a Kernel panic and a
framework slowdown?

And they've spent $100K on that with no results to show for it?

That sounds really strange to me.

[I've debugged Linux kernel panics on custom stacks, with no disk/logs, only
console, 32MB of RAM, PPC cross-compiled - and never run into a dead-end like
this]

~~~
facorreia
"When Qurve got involved with the Charm project the site was being alpha
tested by a handful of early adopters, but a months away from being ready for
public consumption. Qurve was brought on in a Ruby on Rails development role
to help fix bugs, complete the necessary functionality and ensure the
necessary code quality for a public launch.

Over the next few months our role expanded to include a complete rewrite of
the user interface, a reimplementation of the credit card billing processor
and a few more major changes."

<http://www.qurve.com/clients/charm/>

~~~
ahoyhere
Yes, Daniel helped us backport to Rails 2.x and make a lot of improvements
while we were at it, after everybody had given up on finding our Rails 3.x
performance problems.

------
pixelbeat
This seems like a bit of a cop out? Why not try a new kernel or separate stack
like Fedora for example?

~~~
jfb
They spent over $100k on sysadm/devops and couldn't try to deploy a different
kernel? What?

------
axlerunner
This is a bogus excuse. There must be some other reason they are shutting down
as this appears to be a rectifiable problem.

------
nateberkopec
What was Charm? What did it do?

100k for "infrastructure alone" seems like a massive investment for a startup.
How far along were they?

~~~
arctangent
It was a customer support / issue tracking service.

~~~
j2labs
This type of service has been written many times before without kernel panics.

I didn't get to see Charm before they shut down, so I have no idea what they
were doing.

Could anyone speak towards how their experience was unique enough to cause
kernel panics?

~~~
cheald
Anyone outside of the company would just be guessing. If anyone inside of the
company could answer that question, they wouldn't be shutting down.

~~~
j2labs
Pointless response.

------
rlpb
Canonical provide commercial support for Ubuntu, don't they? For "$100k of
investment on sysadmin services/infrastructure alone", did they ask Canonical
for support?

~~~
facorreia
Actually they spent like $20k in what I assume was the salary of their
sysadmin:

<https://twitter.com/amyhoy/status/271128780058267649>

The rest was infrastructure costs ($40k) and porting from Rails 3 to 2.

------
twp
Any links to bug reports or mailing list discussions where Charm reported
their problems?

Googling for "rails charmhq" doesn't reveal much, just that one of the Rails
contractors was a company called Queve: <http://www.qurve.com/clients/charm/>

------
raverbashing
Yes, kernel bugs exists

You can open a bug with distros and try to work around it. It is usually
doable

"unfixable kernel panics" don't seem something a person familiar with Linux
would say. Also, there are several distros, kernel versions, and Ubuntu isn't
my first choice for a server.

And the gist is over the rate already.

------
runjake
If you trust a random gist link as little as I do, here's a more reputable
confirmation:

<https://twitter.com/amyhoy/status/271012712253911041>

------
zdw
It sounds like they were on the bleeding edge, and got cut too deep. Premature
optimization, yadda yadda...

It's better to go conservative on your infrastructure, get it stable, then
experiment if required.

~~~
ahoyhere
Not sure why you got downvoted, because we agree with you.

------
jondot
Sounds a bit strange.

I'd love to see an Ask HN with _proper_ technical details. Who knows, maybe
they'll get better ways to solve this rather than close shop.

------
citricsquid
Does anyone have any links to information on charm, or demos? All I can find
is their website, which has no information: <http://charmhq.com/>

Found this so far: <http://charmhq.com/Charm_Bootcamp.pdf>

------
northisup
Does rails not run on Centos, Debian, or SuSE?

~~~
drbawb
I can _personally_ attest that it runs on SuSE (SLES), Debian, Arch, Fedora,
[and Ubuntu].

I have no idea what this team was trying to do, but I imagine there _had_ to
be more than what they wrote in this gist. That, or they simply ran out of
money to troubleshoot the issue further.

There are also many different ways you can run Rails apps. They could've tried
to run them on varoius Rack compatible application servers (Passenger on
Apache or Nginx, Thin or Unicorn, et al.) -- If they migrated backwards to
Rails 2, they could've used an [obsolete] version of Passenger with Ruby
1.8.7-Enterprise, which had many speed and stability improvements. (Assuming
their app wasn't 1.9.x compatible.)

tl;dr: There has to be more to the story than this, they had so many options.
For a $100,000 investment in administration, I'd have trouble believing they
_didn't_ explore other options.

------
jbigelow76
I wonder how many users Charm had and I wonder how many would have been needed
to make porting the code from Rails to something like Java worth it.

------
dantiberian
The reason that everyone is so incredulous at the shut down is because so few
details were given about what they tried. I'd be really interested to know
what they tried e.g. different kernels, different OS's.

------
krsgoss
The message makes it sound like it's all over, but this alludes to a reboot in
2013:

<https://twitter.com/amyhoy/status/271279292741201920>

------
cpeterso
charmhq.com's current landing page does not render correctly in Firefix. The
Charm logo is positioned off screen in Firefox, but centered on the screen in
Chrome.

~~~
cpeterso
The Firefox layout issue is because Firefox's HTML "quirks mode" is less
quirky than WebKit's. With a proper DOCTYPE to disable quirks mode, Chrome
will render the page like Firefox.

------
krobertson
Anyone have a link to their website or anything? Talk about a horrible name
for a startup, simply cannot even find it.

~~~
jbigelow76
You can find (previous) references to it on Amy Hoy's blog unicornfree.com.

------
sdrgalvis
Did they test other linux distros?

~~~
sudhirj
I don't think they tested other browsers. Distros might be pushing it a bit.

------
ahoyhere
What I find interesting about the reaction to this email, from you bench
spectators, is that you completely missed the point of the email.

The email isn't "Wah we hit a server problem, bye bye."

The point is:

"What this has really shown us is that, if we open Charm to the general
public, we won't be able to provide you with the kind of service you deserve.
We are a tiny team, and so far, we've had zero luck in our attempts to grow by
hiring developers. Problems which are small now will only get bigger."

If you've never run a serious product, or a real business, or tried to hire
for technical positions, I can understand why you'd zero in on the "facts"
about the technical situation and ignore all the "foofy personal window
dressing," and write things like "I call BS! It's a smoke screen!" or "Why
didn't they just try BSD?"

And yet I addressed the actual problem in very clear terms in a paragraph you
can't possibly miss.

Next time somebody makes a hard business decision and you hear about it on HN
and come out, irony guns blazing, may I humbly suggest you read more than the
subject line written by the unrelated HN submitter?

As for any poor silent, lurkers who wonder if this is how they will be treated
if they -- gasp! -- ever find themselves in over their heads, or in a business
they realize they _don't actually want to be in_ … our customer reactions have
been uniformly:

"Aw, I'm so sorry... Charm is such a nice piece of software... your email was
so touching."

Why? Because we've always shown our customers respect by creating great
software, and we're showing them even more respect by ensuring we do not make
promises we can't keep.

Our friends and technical acquaintances have been full of nothing but
sympathy, understanding, and for those closest to the situation, praise for
making the right, hard decision.

Yep, it sucked. Yep, we poured something like $200k into development,
redevelopment, and infrastructure all told. Yep, it is a fucking amazing piece
of software and the best thing I've ever designed.

But is it worth the constant heartache of the impossible task of finding
people equipped to work on it? Of having it stuck in some kind of product
half-life because of that? Of feeling responsible for, but incapable of, being
"on call" in the middle of the night?

Of feeling guilty because, unless we can somehow suddenly be great at those
things, we're taking money for a service which might let our customers down?

Nope. It's not worth it.

And boy do I feel lucky and privileged that because we spent nothing but our
own money on it, we are free to decide to do whatever we think is right.

See also my principles 10 and 11: [http://unicornfree.com/2011/lessons-
learned-from-16-years-of...](http://unicornfree.com/2011/lessons-learned-
from-16-years-of-hustling/)

