
Joyent: 54+ hours of downtime - enduser
I am a customer of Joyent.<p>For the past two days, the server on which I host some customer websites and mail servers has been down. I did not receive any warning or explanation by email or telephone. When I filed a support ticket, I was given a link to their help site with some status updates.<p>Status updates (paraphrased):<p>2011-02-26 09:22:19 GMT: server offline<p>2011-02-26 14:38:38 GMT: no ETA<p>2011-02-26 17:09:42 GMT: estimated downtime 12+ hours<p>2011-02-27 22:23:08 GMT: estimated downtime 20+ hours<p>That final estimate of 20+ hours is in addition to what is now 37 hours of downtime. Joyent has offered no reparation or backup service (e.g. backup MX service to keep mail from bouncing).<p>For your consideration when choosing hosting providers.
======
jasonhoffman
I get the frustration. I really do and I apologize for any bump like this. I'm
the founder of those old textdrive "share hosting" servers (they really have
nothing to do with Joyent's current product line). These servers are ~6 years
old and they're largely populated by "lifetime" customers who have tricked out
various bits of their accounts enough that an automated migration has been
basically impossible. We've been telling everyone for 3 years to really get
off of them, we've been offering free upgrades for 4 years, including you btw,
you got a free upgrade back in Nov 2007, and there's even support tickets
detailing it (which you've participated on). So now we're piecing together
everything that was left but those good ole FreeBSD UFS file systems take
awhile.

~~~
enduser
Offering a transition from FreeBSD to Solaris that requires the effort of the
customer is not free. Asking a customer who knows FreeBSD to take on
administration of a Solaris virtual server for your convenience is not
reasonable.

You are offering an unacceptable level of service to customers who have
supported and recommended you from your early days. You were paid in advance
for a lifetime of service. I took you on your word and offered lifetime
hosting to friends with small websites for small businesses. If you can't make
good on your agreements, I will eat the cost of moving these sites to
Rackspace Cloud. I can't risk another 54 hours of downtime.

With proper backups and a fresh server it should not take 54 hours to get back
online.

~~~
jasonhoffman
To even call this "Joyent" is misleading.

The packing system is the same, it's BSD userland. The paths are the same.
Everything is the same. The kernel shouldn't matter. And if you had told
support that you wanted us to just migrate you because nothing was out of the
ordinary, what would they have done? They would have migrated you ....

And this has been quite a good deal for you. The logic of moving from
something that costs you nothing to something that does costs you something is
interesting.

~~~
officemonkey
>And this has been quite a good deal for you. The logic of moving from
something that costs you nothing to something that does costs you something is
interesting.

Downtime costs him. So much so that he's willing to pay for uptime.

Also, it didn't cost him "nothing". He paid for service. It wasn't free.

Your reply sounds like you don't value him as a customer anymore because he's
not giving you money anymore. Nobody forced textdrive into offering the
"lifetime" package. You should treat him like a valued old customer, rather
than calling his logic "interesting".

~~~
jedsmith
> Your reply sounds like you don't value him as a customer anymore because
> he's not giving you money anymore.

I think you mean: _replies_.

------
dasil003
Frankly, I'm surprised you are still on there. I too was an original VC and I
left _years_ ago due to the support failings.

Look, we were investors in the truest sense. TextDrive was this revolutionary
new shared hosting platform with a lot of promise (flexibility, no
overselling, etc). Problem was that shared hosting is a commodity with no
profit margins. Since TextDrive was started by people who want to build cool
shit, rather than people who knew anything about operations in a customer
service industry, the result is a massive series of pivots that left us out in
the cold.

I'm not making excuses for anyone here, personally I think Jason Hoffman spent
way too much time in the forum sucking up the love, and not enough time
figuring out how to run a support operation. So he burnt his bridge with me as
a customer. I won't touch Joyent with a ten foot pole no matter how good it
is.

But here's the thing: _that was their bridge to burn_.

It doesn't matter what 200 early customers think, because those 200 one-time
payments were not going to build a successful company. The bottom line is that
the market for a "premium" shared hosting service got squeezed by commodity
hosting on one side, and VPSes on the other. Even if they had managed
reasonable uptime (which they didn't because they totally underestimated the
technical cost of giving people the freedom they did), it still would have
been a doomed market, because VPSes got so cheap anyway. We can complain about
what a bad deal we got, but it doesn't change the fact that the last salary we
paid was in 2005, and the hosting world has moved on. They did what they had
to.

------
kylecordes
I generally like Joyent, and use them. So I offer this tip with a tinge of
sadness:

It looks like these message are from a problem with their Howe server. That is
one of their old servers, which came over from the Textdrive merger years ago.
I used to have some sites hosted on that same server, and was likewise not
very happy with things. It appears to me that Joyent as a company doesn't care
very much about that legacy equipment and line of business.

The way to make the pain stop, is to stop using Joyent's old TextDrive
servers. You can do that by leaving Joyent entirely, or by moving to their
newer stuff. Either way it's a fair amount of work, depending on the
complexity of what you are hosting.

I took the path of moving to their newer stuff, and have been very happy with
Joyent ever since.

~~~
iskander
I had a lifetime account with TextDrive. I don't know if it was official
policy at Joyent to neglect the old customers, but I certainly felt like I was
being pressured to repurchase services I had already paid for. After several
outages, data loss and extremely slow responses from tech support I switched
to linode. No problems since.

~~~
chuckmcknight
Hmm, Joyent migrated my TextDrive lifetime account to their newer stuff at no
charge. Did you ask Tech Support if there was a migration path for you?

~~~
jasonhoffman
This is correct. We've been offering upgrades to newer stuff. You just need to
contact support.

~~~
boredzo
Do we actually need to contact support to be upgraded? My understanding from
the old Shared Accelerator documentation[1] (which you seem to have deleted)
was that we were supposed to wait to be offered a “golden ticket”, unless we
absolutely needed to be upgraded ASAP.

Are you even still offering regular (not “cloud”) web hosting? I can't find it
on Joyent's site anymore; it seems as if the Shared Accelerator plans are
discontinued, too.

[1]: <http://wiki.joyent.com/shared:kb:start>

Edit: Found it. This is the page that implies that TextDrive users have needed
to wait for a “golden ticket” to be offered to them:
[http://replay.waybackmachine.org/20090305040858/http://wiki....](http://replay.waybackmachine.org/20090305040858/http://wiki.joyent.com/shared:kb:migrating-
from-textdrive)

~~~
boredzo
Ah-ha! Looking at the front page of their (apparently new) wiki, I found a
link to the old wiki:

<http://oldwiki.joyent.com/>

And _that_ wiki has up-to-date information on migrating from TextDrive to what
are apparently now called “Shared SmartMachines” (formerly Shared
Accelerators). And that page finally does resolve the question of whether we
need to contact them or wait to be contacted: We do need to ask for a golden
ticket, not just wait for it.

That's pretty well buried. Joyent really should send out a mass email to all
TextDrive users asking us to upgrade, telling us exactly what effect it would
have on our billing, and telling us exactly what we need to do.

It's still unclear whether, in the special case of Howe, we still need to ask
for golden tickets, or all Howe users are being migrated to Shared
SmartMachines without having to ask. The support post:

[http://help.joyent.com/index.php?pg=forums.posts&id=949](http://help.joyent.com/index.php?pg=forums.posts&id=949)

implies that they are rebuilding Howe, which implies that we will not simply
be migrated to a Shared SmartMachine as a result of the failure (we still
would need to ask), but they have not directly said.

The one remaining worry I have is that there's nothing on the main Joyent site
about Shared SmartMachines. It seems like they're inviting users of one
deprecated service to migrate to another deprecated service.

------
jarin
There seems to be some confusion in this thread about the difference between
"free" and "paid for in advance".

Cutco will re-sharpen your knives for free, no matter how old they are, even
if you bought them on eBay, because they made a 'lifetime resharpening'
promise with the original purchase. And they're basically a multi-level
marketing scam.

------
frankwiles
I have one single client who refuses to move off Joyent. My experience with
them has been less than stellar. Frequent problems with no explanations, ask a
support question like "How does one do X on your system?" and often get "oh
you want to do Y I see, here is a link to the support forum for that." and the
link isn't at all related to X or even the Y.

They aren't horrible, but I would never use or recommend them to anyone.

~~~
jasonhoffman
If you're on an old textdrive server, then you need to move, like we've been
saying.

~~~
amock
Have you considered offering to buy back the lifetime offers from people who
won't migrate? It seems like that might be cheaper in the long run than having
to support the old BSD servers forever.

------
itsnotvalid
[http://www.joyentcloud.com/about/policies/cloud-service-
leve...](http://www.joyentcloud.com/about/policies/cloud-service-level-
agreement/)

For clause 2:

    
    
      2. Service Level
    
      Goal: Joyent’s goal is to achieve 100% Availability for all customers.
    
      Remedy: Subject to Sections 3 and 4 below, if the Availability of customer’s Grid Container is less than 100%, Joyent will credit the customer 5% of the monthly fee for each 30 minutes of downtime (up to 100% of customer’s monthly fee for the affected server).
    

So probably you can get this month for very little (Takes only ten hours for
getting full refund if SLA is signed)

But at the same time their exceptions list are pretty long, things like
_emergency_ maintenance and upgrades are considered as exceptions.

~~~
swombat
Sounds like the OP is getting the service for free anyway...

~~~
officemonkey
It wasn't "free". We bought a "lifetime" package. I don't recall the details,
but I ponied up a couple hundred dollars at a time when we didn't know
textdrive was going to make it. I don't expect a lot of support, but I do
expect uptime and courtesy.

~~~
itsnotvalid
Oh I see... then, I just wonder how they could eat up for life-time
subscriptions for so long. After some Google searches, I've found that those
plans are one off, and seems to cost $799 to sign up
(<http://hostingfu.com/article/joyent-lifetime-accelerator>). If the terms was
not changed, it now cost $25 a month on JoyentCloud to have the same thing
with the SLA I mentioned. So it takes 31 years to compensate the cost of the
$799 lifetime subscription to pay off. But the fact is that now they have "10
TB Monthly Bandwidth" (or unlimited depending on which web page you are
looking at), I am just blindly guessing that lifetime users would get that
too, right?

------
fingerprinter
FOR JASONHOFFMAN

I see you posting quite a bit here...you keep saying the same thing: "take the
free upgrade that is being offered".

There also seems to be quite a few old textdrive lifetime members here.

Perhaps the best approach would be for you to outline the steps for those
textdrive folks to migrate off of those servers and take the free upgrade you
are offering. I personally have no idea how...so if you could explain it, give
a link or some concrete information it would be helpful.

I for one would be happy to migrate if it means better servers, I just need to
know how...I'm sure I'm not alone.

~~~
tylerritchie
From one of the very many forum posts on the exact topic:

"If you have not sent in a request and would like to get a jump on the
migration, please send an email to support [at] joyent [dot] com with the
following information:

\- Subject line: migration request + plan type if known (startup, plus,
premier) \- Full Name \- User Name(s) / Primary Domain(s) (please note new
info if you want to change it at this time) \- Existing Server(s)

If you wish your username or domain name to be changed, please include this in
the email and we will process when we send your golden ticket. Waiting until
after your account is set up will incur a $50 fee and you will receive a brand
new account (old one will be blasted).

Please do not send follow up requests as it only slows the process down. If
you asked questions when you sent in your migration request, we will answer
those when we send your golden ticket out.

Once you receive your 'golden ticket', you will have 60 days to migrate to the
OpenSolaris environment. If you need the second account for more than 60 days,
regular monthly charges will apply."

~~~
fingerprinter
Thank you for that.

I find it a tad bit odd that I received NO emails on migration. Frankly, there
is no conceivable way that announcing in a Joyent forum and expecting users to
become aware of it is a sound strategy. Pull vs Push and all that.

Even so, I'm willing to migrate if I can find some additional information as
to what is involved.

Again, thanks for posting...I'll go digging through Joyent's forums now....

------
soulclap
Haven't read _all_ the comments but this really seems like a 'special case'
being an exception and does definitely not apply to the experience and quality
you'll usually get on Joyent.

I have been with them for a couple of years and the only reason why I left was
because the project I used to host there was put on hiatus and I don't need
that kind of 'enterprise' hosting for my other stuff.

I agree there shouldn't be downtimes on any of their machines but just going
by the headline, you'd think Joyent is unreliable when indeed their own server
plans are rock-solid.

You make it sound like they are Dreamhost. That said, hope your problems will
get solved soon.

~~~
enduser
I will edit my original post as soon as they solve the problem. No response
yet, other than excuses and denigration from Jason Hoffman (Joyent founder) in
this thread.

Edit: I can no longer edit the original post, but I will post a comment.

------
asuth
We're Joyent customers as well, and their support has always been great, so
this is definitely surprising. You should get on the phone with them (they
provide an emergency number). What kinds of machines are you running? Is it
just one, or many? It shouldn't be necessary, but message me if you're still
not getting through and I can ping some contacts.

~~~
jasonhoffman
Thank you, we actually take our support very seriously and we're pretty
accessible. He's talking about a different product than what you have.

~~~
payingcust
No you don't, I asked via a support ticket two days ago for an estimate, I
can't even get a response from the support team. S I am frustrated enough to
post here

------
kenmck
I have a lifetime account as well. Joyent's tech support has always been
excellent. I take this as a wakeup call to get my stuff off of the old severs
beyond that I can see little fault with Joyent.

~~~
kenmck
btw. Elapsed time between opening the support request and an answer from
someone at Joyent - 9 minutes... Elapsed time from opening support request to
receiving account details for an account on a new machine about 26 minutes.
That includes the time it took me to get back to them about some details of
the account.

------
nphase
I'm sure I'll get down-modded for saying this, but on WHT people will tell you
that you get what you pay for, which is in stark contrast with what most
people here are saying (lack of gratitude for being a charter customer, etc).

Sure, you paid $200. But if you've been a customer since 2007 then you've
basically paid the same price as any cheap shared hosting provider. Contrast
this experience with Layered Technologies that ran three price hikes and then
when I still didn't leave the server I had with them essentially told me to
fuck off once their harddrive failed.

At least jasonhoffman had the courtesy to tell people their best option is to
switch servers to avoid trouble.

But, like they say, you get what you pay for. If lifetime support and
stability is what you'll come to expect, don't pay bottom dollar prices up
front.

Disclaimer: I am not a Joyent customer in any way.

~~~
Klinky
The better option would be for Jason Hoffman to say that he stands by his
offer & will ensure that those who took advantage of the TextDrive deal will
continue to see the same level of service or better that they received back
when they bought it. Advertise your new services fine, but don't try to upsell
people anytime there is downtime & don't go around telling everyone that these
people are whiners getting their service for "free".

~~~
jaaron
Okay, I get that migrating costs time and that's definitely not free, but
saying that Jason upselling is false. It's been repeated by a number of
commenters here and it's starting to bother me.

I'm a lifetime Joyent (TextDrive) member. When Joyent started offering newer
shared hosting accounts, they allowed me and all other lifetime members the
opportunity to migrate. No new fees, no hassle. Just a nice, shiny new space.
To me, it's a testament of their support that they didn't try and keep all the
lifetime members stuck on the old FreeBSD system.

I also find it unrealistic to expect lifetime customers to never have to
migrate from one server to another. Do you really expect the server to run for
20 years without ever having to do anything yourself? Seriously, take some
responsibility for your site and move to the latest supported service.

~~~
Klinky
So there are no monthly fees after migrating from the old TextDrive servers?

It sounds like the offers to migrate have either not been sent to all or the
messaging has been inconsistent as to costs/ease with which this can be
performed.

------
jsdalton
I had some issues with a Textdrive server recently as well and migrated over
to one of their smart machines (at a lower price than what I had been paying).
I was a bit frustrated like you, but Joyent was extremely, extremely helpful
in assisting me with the migration and now everything is running smooth.

I would try to touch base with them again and see if they can help you out.

~~~
jasonhoffman
And there's no reason that you needed to pay additional unless you bought a
different product.

------
andrewvc
I remember trying to get a quote / get a few details about buying large
quantities of bandwidth from Joyent a couple years back.

They consistently took days to answer simple questions via email. I generally
take it that if a company can't even get its sales people responding quickly
there's a snowballs chance in hell it's techs will.

~~~
jasonhoffman
If it was a couple of years ago and for large quantities of bandwidth, then
you would have been talking to me. We didn't have "sales people" a couple of
years back. And I would have said "No" because we weren't in the business of
selling "large quantities of bandwidth"

------
pygorex1
You get what you pay for.

This isn't a glib statement. A business transaction is about more than coming
to agreement about the price of a thing or service. It's also about creating
and maintaining incentives for both sides.

I pay slicehost for a VPS. They get $50/month from me in return for disk
space, CPU cycles, uptime and support. Our incentives are aligned.

With Joyent/Textdrive, every month you get hosting and every month they get
nothing. The incentives are not aligned. In fact, from Joyent's perspective,
the incentive to provide any meaningful service diminishes over time. No about
of cajoling or complaining or threatening Joyent will change this imbalance.
Sure, it's 54 hours of downtime today - but I can guarantee that a year from
now the outages will be far longer and far more frequent.

"But we had a deal!" Yes, you did. And you were a fool to enter into it.

Here's the bottom line:

Your customers don't care what's causing the downtime. It's still your fault.

Website users don't care that the deal has soured. It's still your
responsibility.

Joyent doesn't care that they're providing cr@ppy service to Textdrive users,
despite the postings on HN by Jason Hoffman (hey Jason, instead of wasting
time posting on some internet forum, why not spend that time fixing the actual
problem?)

Seriously - Joyent doesn't give a sh?t about the lifetime accounts. If they
did, then why are you experiencing 54 hours of downtime? If they do care, and
are applying resources to solve the problem, then why is it taking so long? 54
HOURS TO FIX A SERVER. That, to me, is evidence of a shocking level of
incompetence. In either case I would get the hell out of there.

------
oziumjinx
T minus 10 minutes before this hits TechCrunch

------
grandalf
I've had a very bad experience with Joyent. Abysmal performance, tech support
in denial in spite of obvious tests. Just leave, your life will be better.

------
sid_g
I remember that chunkhost has really aggressive pricing.. And I remember there
service being pretty good...

~~~
jasonhoffman
He's currently at "free", so get any more aggressive than that.

~~~
enduser
Paid for lifetime service in advance to help bootstrap your company, not
"free".

------
desigooner
Is there any time that Mr. Hoffman's going to call it quits on this thread and
actually take it offline in an attempt to resolve the OP's issues?!

Frankly, the tone of most of his comments have been borderline denigrating and
it's doing nothing but adding nuisance value to the thread.

~~~
ericd
He's probably getting exasperated by the virtual lynch mob which seems to be
downvoting everything he says and upvoting every angry comment, which is
ridiculous. He's actually saying quite a lot of reasonable and useful stuff
which is sitting at 0 or below.

------
netik
joyent had this sort of failure regularly about 4 years ago and continues to
do so. Avoid.

~~~
jasonhoffman
Also incorrect. "About 4 years ago" we always did >99.99% uptimes even on the
old textdrive stuff. I'd be happy to show all the old stats and graphs, we
still have them.

~~~
iskander
Jason, you're experiencing a PR hiccup; it's not a big deal but it's also not
clear that responding to every message in this thread is going to help you.
Instead, why not (1) make things right for the OP and everyone else on Howe
and (2) put a response on the blog?

You Joyent folks will never get a subset of your customers to migrate off
textdrive hardware (so it's no use to keep yelling "just migrate!"). However,
if you let those older servers fall apart you'll only get more negative PR.

~~~
jedsmith
I get having a passion for your product -- I really do. Some of the responses
here from Jason are on the border line, though, and I think they are making
the situation worse overall. I wish, for his sake, that he'd have quit while
ahead.

Without a doubt, Jason has the best intentions here, but I'm getting a
_really_ bad vibe from reading the entire thread -- particularly the _by the
way, you contributed to a ticket and I can look it up so I'm going to remind
you so that you'll look stupid for bitching about us_ bit.

The hostile you-are-wrong attitude, while maybe technically correct, is a
disaster for outside perception.

------
payingcust
I will be moving from joyent. I am paying 15usd per month and have not been
told to migrate. now I read the founder saying it's my fault that I am
suffering so much downtime because I should have searched the forums and
migrated years ago. Well I am going to migrate now.

------
snapplez
You must be on HOWE too. I am in the same boat - unhappy. send a tweet to CEO
David Young @davidpaulyoung. (from @snapplez)

------
dp111
joyent is enterprise grade and doesnt have this sort of downtime. its probably
a break in your app.

~~~
prodigal_erik
[http://help.joyent.com/index.php?pg=forums.posts&id=949&...](http://help.joyent.com/index.php?pg=forums.posts&id=949&pc=6)
acknowledges it's a real outage and talks of rebuilding the server after a
significant hardware failure from two days ago, which implies they didn't have
a disaster recovery plan.

~~~
patrickgzill
Good catch; I just wonder how it can take more than 2 days, when you already
have a bunch of employees as well as equipment and presumably spares, to build
a replacement FreeBSD server and transfer data?

------
shareme
The lesson jasonhoffman could learn:

If you have a small set of users who bought lifetime accounts to bet on a
company you founded than even with new company I would bend over backwards to
make sure their needs are met as a thank you.

Otherwise some gets the bright idea to put up a joynet sucks web page with
proper Google approved SEO and makes this little PR bump a big EFFING
MOUNTAIN!'

jasonhoffman, I imagine you have less than 12 hours before that happens..as
TechCrunch may do it for 'free'..:)

~~~
tedjdziuba
Parse Error.

------
visakhcr
Digging Techcrunch got me this story: Twitter and Joyent splits (Circa Jan 31,
2008)

[http://techcrunch.com/2008/01/31/twitter-and-joyent-split-
am...](http://techcrunch.com/2008/01/31/twitter-and-joyent-split-amidst-
downtime-travails/)

Twitter moved to Verio and here's Biz Stone's reply to Techcrunch:

We’re still very much engaged in our efforts to bring solid reliability to
Twitter. Achieving our goals is a sustained effort, not an overnight fix.
Performance is our most important measure of success and we appreciate both
the patience and frustration folks are sharing with us.

With regard to discussing technical specifics about last night’s efforts,
we’ll be more keen to do that once we have a chance to come up for air and
cover it with some perspective.

------
moe
Ouch.

------
milod
It sucks that your hosting is down, but this is a pretty malicious use of HN
considering you paid $200 over 6 years ago for lifetime hosting. Less than $3
per month by now. What SLA were you guaranteed?

The lesson for service hosts, NEVER offer lifetime anything.

