
The SalesForce Outage: A Nail In The Coffin For SaaS? - gk1
http://www.zerto.com/blog/dr/salesforce-outage-one-more-nail-in-the-coffin-of-i-can-trust-my-saas-provider/
======
darawk
While events like this are not great, isn't it a little myopic to think the
solution is to run it yourself? 20 hours of downtime sucks to be sure, but is
your own uptime rate really going to be better than theirs? Not to mention all
the other issues with operating your own service like this.

~~~
Scramblejams
That comparison isn't apples to apples, though. Internal solutions are
frequently less complex than cloud systems, and have fewer and more
understandable failure modes as a result. I could throw a rock at my last job
and hit mail servers that had better uptime than Gmail, to use one prosaic
example.

~~~
tomc1985
I have to agree with this. We recently replaced a homegrown webapp with its MS
Dyanmics GP equivalent and headaches have increased exponentially. It's as if
the system was designed to be so complex as to require a team of people to
operate, plus a manager to supervise, plus a support person to field phone
calls... when one guy churned out the original in a day, and ran for a year-
and-a-half with only minor technical (and no major organizational) hiccups.

~~~
jasonjei
When you write of Microsoft Dynamics GP, this FizzBuzz Enterprise Edition repo
comes to mind:
[https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...](https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition)

------
krschultz
No surprise an article like this comes from an IT vendor that sells services
to people that still own their own hardware.

Which brings up the real question. Can you afford to _not_ run on someone
else's infrastructure? Yes, Salesforce had an issue. How much would you need
to spend on your own company's infrastructure to do better? Is it worth it for
your business?

You need to own the gear, hire the internal people, pay for a host of
enterprise software that is probably more expensive than services, etc.

~~~
toomuchtodo
> Can you afford to not run on someone else's infrastructure?

Most businesses do, so I would say "Yes".

We'll go through this whole decade long period of how wonderful the "cloud"
is, and then we can go back through the next phase where everyone is getting
paid to move services back locally because of how "easy it is to manage with
Docker/Kubernetes/$flavor_of_the_week_cloud_orchestration".

I'm old. I've seen this before.

~~~
bduerst
Where did you see this before?

I think the economies of scale will eventually draw most to using hosted
services, even if it's as simple as better bottom line for the IT supply
chain.

It's a little like managing your own email or website server on site - the
overhead ends up being more resource consuming than farming it out, and we
haven't really seen people switching back to their own servers.

~~~
toomuchtodo
> Where did you see this before?

Mainframes->commodity off the shelf x86->"cloud" (virtualized) computing.

~~~
bduerst
I think that's more of a trend in fragmentation.

Super Computer -> Personal Computer -> Virtual Computer -> ???

~~~
toomuchtodo
I agree its fungible; TL;DR is everyone changes gears every decade as to what
"The Truth" or "The Way" is.

------
sysexit
Cheap shot by a company doing backup/DR.

Zerto, you have great technology, but chasing ambulances is a poor marketing
strategy.

------
kraig911
I know this is not a productive comment but I hate editorial or commentary
from companies that chase crisis for more business. I used to work in hosting
and these things happened and then all our competitors would jump in on the
bashwagon... it got to a point where everyone was chasing each others tails
and not productive.

Plus I don't care who you are or how big you are. If something goes wrong
there's a possibility your app will go down. H.A. notwithstanding is a best
practice but some of the most robust systems in the world still blow up. They
were invented by people ergo they are inherently flawed. That's the beautiful
thing about them.

------
tacos
S3 guarantees three nines of uptime. Thus, you should be prepared for 8+ hours
of downtime _at any moment_.

If Salesforce gives you 99.8% vs. Amazon's 99.9% and this shocks you into
writing a reactive web post, it's a good time to review some basic systems
theory.

[https://en.wikipedia.org/wiki/High_availability](https://en.wikipedia.org/wiki/High_availability)

~~~
aianus
> S3 guarantees three nines of uptime. Thus, you should be prepared for 8+
> hours of downtime at any moment.

It's 99.9% per month, so that's ~40 minutes of downtime not 8+ hours.

~~~
loa
40min x 12months = 8h / year

~~~
colanderman
GP said "at any moment". If the SLA is per-month, 8h won't happen all at once.

------
raverbashing
"This was where the real problems began – the file discrepancies responsible
for the database failure had been replicated back to the Washington data
center while at the same time, the backup in Chicago had not completed in time
and, as such, could not be used as a restore point either."

Ah, another case of "To make error is human. To propagate error to all server
in automatic way is #devops."

------
gxs
Throughtout my career the interesting thing about the level of scrutiny the
"cloud" receives is that people rarely ever applied that scrutiny to
themselves.

Is it secure? Hm..I'm sure Salesforce does more work around security than the
vast majority of their customers would even be capable on their own.

Similarly, these types of catastrophic events will happen and I am just as
confident in saying that Salesforce was able to diagnose and fix this problem
better than the vast majority of their customers would have been able to do on
their own systems. I've seen JIRA on prem go down several times.

I've worked building enterprise applications my entire career. Unless you have
an extremely lean and engineering oriented organization, it's almost always
better to buy than to build in house. There are exceptions, but for the most
part this holds.

Having said all that, every single place that I've worked at has had it's own
data lake/warehouse and having your own backup of your data is critical.

~~~
ProAm
> I'm sure Salesforce does more work around security than the vast majority of
> their customers would even be capable on their own

This is a fallacy that people like to believe because they are not doing the
work themselves. The fact is SalesForce hires the same type of developer/admin
that any company would hire and that person is trying to do their best like
everyone else. Because it's behind a curtain people feel its more secure, when
in reality it's probably the same level of competence/incompetence that anyone
can do hosted on-prem.

~~~
kodfodrasz
But giving this service is the main product of salesforce, therefore intrinsic
quality measures (like security) are more important to their management (for
their well understood need for customer trust), than if you were developing an
in-house product.

In the latter case intrinsic quality of a biproduct of your operation would
not matter to your management, because they primarily think in order to give
the best quality to the customers of the company's product, and the intrinsic
quality of that product matters to them (eg. to use BPA free materials for the
rubber ducks they manufacture). This is why they will feel spending on
security of the in-house development unjustified, until a major security
breach happens.

~~~
ProAm
> therefore intrinsic quality measures (like security) are more important to
> their management

Right, I agree, but just because it is important doesn't mean they are good at
it. And there is the fact that there is now likely 1 or 2 people attending to
1000 customers, than 1 or 2 people in house tending to 1 customer.

~~~
phxrsg
Except that security engineer : customer ratio doesn't make sense as a metric.
Those 1000 customers all need the same thing - for their data in the cloud to
be secure. They don't have differing security requirements that pull the
engineers apart in different directions.

So it really is a comparison of 1-2 security engineers vs 100-200 security
engineers.

~~~
ProAm
> Those 1000 customers all need the same thing

This is completely not true, many customers often need specific custom
tailored security for business specific needs (This is something most startup
SaaS developers dont realize or care about because they are usually just
starting development and care about growth and base product). This is where
the problem lies, you paint with a broad brush and get the 80%, the other 20%
is a nightmare usually. Most the time the 80% has holes in it too, like we saw
here with SalesForce.

Whenever Im architecting a system, I always tell the execs you can either pay
for it now (going on-prem) or pay for it later (with Saas) because at the end
of 5 years you always end up paying just about the same, its just a matter of
where. With payroll, downtime, customer losses, hardware, DR, wasted time with
project managers from your SaaS providers.

Id always rather have control and know my system than depend on someone else's
hire who behind a curtain of three, four or five nines of marketing jargon.
It's always a pay me now or pay me later situation.

------
DenisM
Would a whole-day power outage constitute "one more nail in the coffin of
centralized power distribution"? I doubt it.

The article's contents is still worth reading though.

~~~
talmand
>> Would a whole-day power outage constitute "one more nail in the coffin of
centralized power distribution"?

Yes.

------
wakwanza
Id rather think its a failure in planning.From AWSs playbook the conventional
wisdom is that things fail all the time and you have to plan for these
eventualities.See the Netflix models and Googles multihoming approach to
spreading the service offering.Its always going to be the case that a cloud
provider can fall over however if you are simultaneously running on different
providers , all you have to worry about is the speed of rerouting traffic to
your still operational site.Short of the entire internet keeling over from a
backhoe incident you should be sitting pretty.

------
gautamdivgi
Zerto makes DR solutions for VMWare based private clouds. IMHO - the title is
sensationalist and probably an attempt to drive interest in private clouds
over public clouds.

That being said the firmware bug could have happened in a private cloud as
well - impacts would be localized to a company though.

Either way - public clouds are here to stay unless someone needs extreme scale
in which case they'll use a private cloud (e.g. Dropbox moving out of AWS).
But then they most likely may not use VMWare for their virtualization
solution.

------
mysticlabs
The main issue is with how Salesforce is running its multi-tenanted cloud
infrastructure.

Modern techniques and solutions already exist to minimize these kinds of
events from happening using a multi-cloud approach, containers, geo failover,
etc.

So while this sucks for Salesforce, it doesn't mean every SaaS platform or
cloud infrastructure provider suffers from the same legacy issues as
Salesforce.

If anything the author should be arguing the flaws of multi-tenancy, not that
SaaS is dead because of a power outage of a single SaaS provider data center.

------
gtrubetskoy
A single circuit breaker shouldn't cause an outage if your servers are powered
using dual power supplies from separate circuits or you are load-balancing
across servers powered by separate circuits. This appears like mere
incompetence.

------
kentt
Cloud we have a less linky-bait headline. The death of SaaS is completely
nonsensical.

------
FollowSteph3
How many smaller businesses that use salesforce would have lost all their data
with hardware failures over the years? Most smaller businesses have absolutely
no backups. Remember HN is the exception as were a technical crowd ;)

------
mtehonica
Not a nail in the coffin by any means. Simply a poorly architected
infrastructure.

~~~
rhojo25
I agree. DR is a must.

------
newman314
Remember: "There is no cloud, there is just someone else's computer"

