
Xero reveals the impact of mammoth Amazon Web Services migration - King-Aaron
https://www.crn.com.au/news/xero-reveals-the-impact-of-mammoth-amazon-web-services-migration-490674
======
devonkim
Most of the value of AWS in my experience isn’t really the infrastructure
services as much as the higher level managed service offerings in databases,
rudimentary machine learning frameworks, and a decently documented (better
than 90%+ of enterprise software) set of APIs for things in general. RDS has
its faults but hiring DBAs or deputizing existing engineers to setup backups
and upgrades on DBs is not very high business value in itself compared to
rolling out new features. Heck, even an API call to simulate failovers is
hugely invaluable for enabling developers to not suck at using databases in
production.

------
abskarora
Definitely a good investment of resources.

AWS offers the convenience of managed hardware and services on, for which it
is charging its premium. As companies grow, they can think about investing in
rented servers and managing the software on top of it themselves.

Beyond a scale, owning and managing the servers may also make sense for many.

------
ericcholis
I don't see it mentioned, but it's entirely possible that Rackspace actually
consulted on the migration. They do offer AWS services, realizing that they
can't compete and hoping to capture some dollars.

------
yakuza00000
Can someone put up a link for comparing the costs of azure, aws, and google
cloud? it seems like there are some big efforts underway to migrate away from
AWS.

------
damm
and in 6 months they can turn around and claim they will save 30% of their
hosting costs by migrating to bare metal.

It's just a game to keep VC's and other people happy.

\----

Hey ma! look I cut our costs by 25%

~~~
sitharus
You are correct, if you have no need to add additional servers or scale the
platform, have predictable bandwidth and storage requirements, and your
sysadmins don't mind patching all the base OS instead of just rolling new VMs.

However AWS allows you to accomodate uneven growth and load. You can create
new machines in minutes (or half an hour for Windows). The virtual nature
allows you to quickly ditch a broken host for a good one. You can also save
money on development infrastructure by switching it off, or run background
jobs on spot machines. Then there's the other services available such as S3
and Lambda.

Of course there are other clouds worth considering, but the advantage of cloud
infrastructure is the ability to call it up without months of lead time.

~~~
brazzledazzle
I think your points are sound but it’s worth pointing out that you can roll
new host OS’s and VMs without patching on-prem too. You just have to have the
discipline to run your internal data center the way cloud providers do. You
can even take advantage of one of the few benefits of on-prem and roll a fresh
host OS without downtime for the VMs running on it by using live migrations.

~~~
NathanKP
> You just have to have the discipline to run your internal data center the
> way cloud providers do.

AWS employee here, so I am obviously cloud provider biased, but in my
viewpoint having that discipline only comes from having very good, expensive
engineers. It's hard to get people who really know how to run a datacenter
well, especially when most of the good ones would probably prefer to work for
a larger cloud provider since than is more interesting and impactful than
running a smaller datacenter for a single company.

The forgotten / underestimated cost is the salaries of these people to set
things up well in the first place, be on call to fix things when they break,
and perform those upgrades in the same way that cloud providers do.

A single engineer salary can get you a LOT of cloud compute power on AWS. If a
cloud provider allows you to build the same thing with two fewer engineers
then that's probably an extra $400k or so for the infrastructure budget after
you factor in these engineer's salaries, taxes, benefits, etc. And $400k can
get you an incredible amount of AWS cloud compute.

~~~
e_d_e_v
> The forgotten / underestimated cost is the salaries of these people to set
> things up well in the first place, be on call to fix things when they break,
> and perform those upgrades in the same way that cloud providers do.

The thing is, you need this equivalently for any cloud provider as well.
People who know how to operate and run a cloud account effectively are not a
dime a dozen, and given the migration going on now are actually in higher
demand than some qualified datacenter operators.

~~~
jopsen
IMO the best examæel is spinning up a DB with nightly snapshots. This is
trivial in AWS. Doing it in your datacenter takes planning, decispline, skills
and time.

Same goes for backups to S3 vs. setting up a SAN on the datacenter.

Managing AWS infrastructure isn't trivial, it's easy to forget what that
instance does, if everything was provisioned with point and click.
Cloudformation or terraform makes it doable. But technical debt is also
something you can defer payment on :)

~~~
NathanKP
Yeah the DB example is particularly relevant. Amazon Aurora now has continuous
point in time backups so in the event of something going wrong you can roll
the database back to an arbitrary second:
[https://aws.amazon.com/blogs/aws/amazon-aurora-backtrack-
tur...](https://aws.amazon.com/blogs/aws/amazon-aurora-backtrack-turn-back-
time/)

Additionally Aurora has robust zero downtime self recovery from a variety of
error conditions including disk failures, and you can test that out by running
a SQL query that simulates disk failures for example:
[https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Auror...](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/AuroraMySQL.Managing.FaultInjectionQueries.html)

This level of operational excellence is incredibly hard to build yourself
either on premise or even in the cloud if you are only using VM's. Time and
time again you see companies that thought they had backups but then discover
the backups don't work right. It's not because these companies are stupid, its
because it is hard to do right, and building and testing this requires a lot
of engineer time. When you do take the time to hire the right people who can
do it right you end up finding it is way more expensive to do it yourself than
using a cloud managed service. AWS is able to offer the service much cheaper
because the cost of development and maintenance, etc is spread across a huge
number of customers.

------
PostOnce
All this to "close the gap to profitability", which, after the migration, they
still will not have done, they'll still be in the red, after more than a
decade.

It makes at least some bizarre level of sense that a social company like
Twitter, with no clear customer other than "maybe advertisers" to not turn a
profit for some time, but for an accounting software company, whose product is
paid, to burn cash for over a decade, it just doesn't seem right. It seems
like a scam to bilk investors.

Let's raise a glass to the inevitable bankruptcy, in which the founders and
VCs don't suffer because the IPO already happened. Here's to you, Rod Drury,
Peter Thiel, and Chris Liddell (now working for the Trump administration).
Great job, guys. If you end up in jail, I'll send postcards.

~~~
nikanj
Not sure how New Zealand does things, but in the EU, it's relatively common to
have zombie companies stagger on for decades on government life support.

There's a _ton_ of funding going from the states to various agencies for
various pro-software, next-silicon-valley initiatives. Most of this money is
handled by career bureaucrats with little personal experience in running a
business.

It's easy to waste someone else's money by giving CPR to a large domestic IT
firm, when the alternative is being held responsible for the lost jobs when
the company goes under.

~~~
PostOnce
I wonder if this is in fact even worse, keeping people "occupied" in money-
losing companies, where if the company went under they might (especially in
tech) find something productive (in the literal sense of producing something
of value) to do.

That seems more likely especially in a case like this, where the employees are
skilled, and where the company isn't doing anything unique (novel, charitable,
beneficial to society in a nonprofit way, etc) that would merit rating it by
some other metric than profit.

It just seems tremendously wasteful to be losing tons of money in a market
where there's already a lot of competition and you're not doing anything new.
This isn't an electric car, and AI, a space company, where the outlay might be
a lot and the time to develop the tech is great, this is a long-established
market full of already-solved-profitably problems.

I would guess most of Xero's employees are outside of NZ, but since NZ's
primary export is dairy, the government might be foolish enough to try
propping it up with some nebulous hope that "keeping the tech industry afloat"
will somehow be beneficial, rather than letting the few technologists in their
employ go to work for another, more useful company, or to go and start a
startup of their own (perhaps with government funding, in lieu of it going to
Xero, in that hypothetical situation.)

~~~
pc86
I use Xero and the interface and functionality is great, but what makes you
think someone writing JavaScript for Xero has the skill to work on AI or
spacecraft?

~~~
PostOnce
I worked on js and php once, and now I work on AI. The tech market in NZ is
very limited so I imagine anyone working for Xero in NZ has better-than-
average odds of being there just because its a job, rather than because thats
all they know how to do (or all they want to know how to do).

Maybe without Xero, they'd be out looking for and learning new things,
starting new companies, etc. That's what I would do if I worked for Xero
(which was conceivable, at one point) and they closed down.

