
Bank of America's CEO says it's saved $2B per year by building its own cloud - sogen
https://www.businessinsider.com/bank-of-americas-350-million-internal-cloud-bet-striking-payoff-2019-10
======
Someone1234
I think a strong argument can be made either way.

There's definitely financial savings to be had by doing it in-house (either
literally or utilizing existing data centers as here). But what I've
experienced is that companies are often attracted to Big Cloud™ not just for
financial reasons but to flatten/simplify their corporate reporting structure.
It is the ultimate delegation.

This isn't simply about having fewer staff (although that can reduce internal
politics and inefficiencies), but making fewer decisions and more importantly
reducing the potential for making the wrong decisions (which can be career-
costly, regardless of right decisions made previously).

These "soft" costs/savings are rarely discussed because they cannot be
measured, unlike a balance sheet. But there's definitely a whole other
dimension to infrastructure outsourcing that is worthy of note.

In this case BoA are saving money but in exchange retain additional layers of
staffing, that require oversight, and ultimately decision makers willing to
take risks.

~~~
trabant00
> This isn't simply about having fewer staff (although that can reduce
> internal politics and inefficiencies), but making fewer decisions and more
> importantly reducing the potential for making the wrong decisions

This is a myth directly from cloud providers marketing campaigns: that using
the cloud simplifies infrastructure to the point that you don't need as much
and/or as qualified staff.

In reality it is quite the trap: without experienced system administrators AWS
and the like are free to dig their hands deeper and deeper into the pockets of
their unsuspecting customers.

~~~
jedberg
Yeah but you don’t need data center engineers, real estate specialists. Power
specialists. Etc.

~~~
ddorian43
You can rent dedicated server or onprem ? people always go from cloud to
building your own cpu in these threads, never in the middle

~~~
pentae
Yeah we literally turned our $7k a month AWS bill into $1k a month by changing
to 3 dedicated servers (Fully managed!) and cloudflare. It's better in every
conceivable way.

~~~
WrtCdEvrydy
For anyone doing this... look around.

Tooling has only improved.

We were able to cram a large set of applications into a group of instances
using CapRover.

Instead of 2-3 instances, and a whole bunch of RDS, we ended just recreating
the databases there along with minio for storage (we write a small amount of
data to S3 storage).

------
KaiserPro
I used to work at a very large financial publishing company.

I joined just at the point where they were moving from VMware to AWS.

It was never a cost saving exercise. It was a way to completely change the
culture of the IT department. As they were one of the very first news
websites, they had at the time ~18 years of cruft to deal with.

The drive to the cloud was about flexibility, and if we are honest, shaking
the staffing tree to get rid of the ossified staff.

They went from three data centres (mixture of SUN and Intel blades all backed
by FC) to pure AWS/heroku. The running costs went up significantly, the opex
was close to £3mil in AWS bills alone, all to put text on a web page.

It will continue to rise because it offers flexibility, and as each product
now has its own account, standards are difficult to enforce. This means lots
of snowflake installs of X that are now critical to Y.

The cloud is more expensive for large companies, for most workloads. For SMEs
its the total opposite.

~~~
weberc2
> It will continue to rise because it offers flexibility ... The cloud is more
> expensive for large companies, for most workloads.

The way I read this is that the operating costs are higher in the cloud, but
presumably dwarfed by recouped opportunity costs. This makes sense to me,
given my limited experience working in large companies where every interaction
with IT involved a 2-6 week turnaround time. Need a VM? Fill out a bunch of
paperwork and then wait a month. We missed a lot of opportunities that way.

~~~
TheCraiggers
A private cloud can fix that turnaround time just as easily.

The true opportunity cost in large corporations is from, IMHO, been the
general attitude of "we can't do foo new thing because that's [not best
practice, not (barely) supported by some other large corporation, too cheap,
too big of a change, etc]."

The ability to create extremely flexible workloads in minutes means nothing
when security has to approve sneezes and containers are only something you'd
find in a fridge.

------
shakil
Looks like $2B of savings are from reducing the number of servers (200,000
servers earlier to 70,000 now). That has nothing to do with the advantages of
a private cloud over public, etc. They were just over-deployed and by reducing
the number of servers they would have enjoyed similar savings no matter where
they were running them.

In fact they could have saved even more, and earlier, if they had already been
in the public cloud, where you can just spin down the VMs and stop paying for
them. In your own data center, a server is already sunk cost that you cannot
entirely eliminate just by shutting it down.

~~~
Skunkleton
Is $15,000 per server normal?

~~~
nwallin
Yes.

"Server" implies redundant power supplies, 4-?? disks, an excess of high
quality capacitors, a heavy steel case, a motherboard probably 4x the area of
a typical ATX motherboard, ECC RAM, far more RAM slot capacity, etc.

Furthermore, extended service contacts are the norm. Five year same day on
site service isn't unusual. Five year 24/7 on call/next day replacement is
typical. One year warehouse service is unheard of.

That's just the device itself. Servers also imply a rack to put them in, real
estate for said racks, 24/7 HVAC for said real estate set to substantially
below room temperature, and an expectation to run it (and consume electricity)
continuously. You also need network architecture to support them.

This isn't even getting into the realm of stupid specced servers. 512GB RAM?
Easy. 128 cores? Sure. Disk space, network capacity are cost limited: if your
server is limited by network speed or disk capacity you can simply punt a
briefcase full of money at your supplier and get more. You want all of that
arbitrary capacity disk space to be SSD that's just an accounting problem.

------
gumby
I'm not really sure why this should be a surprise. Just as big companies use a
mixture of owned and rented real estate for various reasons, the same is true
of other large expenses. If you have a large, predictable, core workload it
makes sense to bring it in house, and use the elastic (rented) resources for
unpredictable stuff (e.g. new efforts).

It's not like Amazon doesn't know this too (e.g. they try to make the
transition harder by offering lots of proprietary services that help you ramp
up faster, but can't be used elsewhere)

~~~
davnicwil
Agreed - past a certain (probably fairly massive) threshold of predictable
long term load, it seems clear renting resources is going to be more expensive
than the all-in cost of owning and operating your own.

Were the opposite true, you'd wonder about the economics of the cloud business
model.

I think a more interesting way to look at this is, are the other advantages of
cloud services (e.g. lots more flexibility and bundled proprietary
technologies that you can't economically build yourself) actually worth the
extra money?

Yes, by not using a cloud service you might save $2B a year, but does that
cost you the opportunity to make even more than that, given you're probably
moving slower or at least less efficiently than you otherwise could?

~~~
bonesss
> but does that cost you the opportunity to make even more than that, given
> you're probably moving slower or at least less efficiently than you
> otherwise could?

The potential opportunity of fast-moving cloud features needs to be weighed
against the opportunity costs of slow-moving cloud features. Where bespoke
solutions can immediately provide tailored performance and maximize technical
capabilities, a missing feature in any of your cloud providers services can be
a showstopper or unmitigatable roadblock. And while lots of the technology is
past the bounds of reasonable economical replacement, some of the technologies
being shared through the cloud are nigh unfathomable to recreate.

Which is to say that the black ju-ju behind Windows update probably takes
making a new MS to build up to present maturity, and unless you're a certified
"big boy" letting small teams somewhere else fully dictate what you can and
can't do at service boundaries probably impacts you in the long run.

Based on that, and IMO/IME: the answer isn't a binary choice but a constantly
shifting point on a spectrum between the two, where on-premise/local-cloud and
remote-cloud services are aware of one another and maximize capabilities while
minimizing costs. Hybrid installations are just stronger, and are easier to
reshape according to costs.

------
bpchaps
I used to work at Bank of America as a level 2 app analyst back when they
first started building Quartz. At the time, it was advertised internally as a
system to be used for reporting, and so it had lots of built-in functionality
to connect to databases, etc. Pretty neat.

That said.

The method of encoding _production_ database credentials was _rot-13_. No
joke. In the Quartz interface, you could double click on a starred-out set of
credentials, and it would run rot-13 on it and display the password. This was
for FX, rates, credit card, mortgage, etc etc etc. Having access to this cloud
system gave effective access into _all_ of Bank of America and Merrill Lynch.

They probably save a lot of their money by using very, very bad practices.

Still only the second worst security fail I've seen.

~~~
pouta
Could you share the winner?

------
julienfr112
I work for banks, and I can tell you that when you are doing devops with their
"own cloud", you are miles away from a real cloud experience : no os choice,
no hardware choice, slow provisioning, no access to repo, low and inconstant
virtual disk (EBS) speed... I guess that you get what your paied for, and
maybe the 2B saved on cloud are spend on IT service that suffer from such a
poor own cloud experience. If they have 50 000 IT employees payed 100k, that's
5b a year. Including professional services and you are maybe at 10b. Just
increase the productivity of this 10b by 20% and the money "saved" is not such
a good deal...

~~~
sundbry
50,000 engineers just to run a bank? I don't think so.

~~~
doovd
250,000 engineers at JPMorgan.

~~~
kevstev
There are ~250k employees at JP Morgan, not engineers.

[https://en.wikipedia.org/wiki/JPMorgan_Chase](https://en.wikipedia.org/wiki/JPMorgan_Chase)

~~~
mherrmann
s/employee/engineer/2

(Note the 2.)

~~~
tekno45
What tool is this for? Sed?

I've copy pasted it before, but i think you just made the syntax click

~~~
simpsond
Yep, sed supports that syntax (substitute). echo 'There are ~250k employees at
JP Morgan, not employees.' | sed 's/employees/engineers/2'

------
pmoriarty
Many large companies save money by having their own cloud, but those clouds
sometimes really suck compared to AWS.

I know of one very large company where any request for a change in their cloud
infrastructure always required a minimum two week advanced notice. In AWS such
a change is just a mouse click away, and could be done in seconds.

AWS also has a really amazing integration of a large variety of services which
is really hard for in-house clouds to match. I wonder how many AWS services
the BoA cloud has, and how their own integration of those services matches
that of AWS.

~~~
jjav
That's not a comparison of external cloud vs. self hosted though. It is a
comparison of different operating models with respect to change control.
Either one can exist in both external vs. self-hosted infrastructure.

I've been in places where the engineering team plugs in an old PC under a desk
somewhere, gives it a public IP address of there's the production server.
That's the self-hosted equivalent of the engineering team having full AWS
access and any change is a mouse click away.

I've also experienced places where any change does take a week or two of
approvals even though it is hosted on public cloud.

There is a time and place for all approaches. What works best with a three
person startup putting up a MVP is quite different from what is best in a very
large corporation operating in a regulated environment.

~~~
jeremyjh
Change control and service delivery time-frames are two different things. When
I left BOA in 2017, you could get most production changes implemented with
only two days of lead time from a formal change control perspective. But,
requesting a new Virtual IP on a load-balancer could easily take two weeks,
just for every layer of bureaucracy to wet its beak. And it was impossible to
request something so basic without an online service request, and then follow-
up emails because the standard service offerings left all kinds of details
undetermined and no structured way to provide the information.

~~~
closeparen
Most production changes require a second set of eyes, sometimes from a
particular team, but it's all "just" code review. You put your change in the
team's queue, their oncall engineer reviews it the same day, you land the
change and it gets executed automatically. Most have implemented namespacing
so that changes that only affect your own team's stuff can be approved within
your team.

This is all on owned hardware. The difference is that we're a SWE driven
company (corporate IT is off in its own world, run in the more traditional
way, but they don't touch engineering's production datacenters).
Infrastructure teams provide APIs, not JIRA forms.

------
neonate
[https://web.archive.org/web/20191021160628/https://www.busin...](https://web.archive.org/web/20191021160628/https://www.businessinsider.com/bank-
of-americas-350-million-internal-cloud-bet-striking-payoff-2019-10)

~~~
qorrect
Hey cool trick thank you.

------
kfk
The appeal of the cloud is less IT politics which in big company is a serious
problem. Business Units can sidestep IT and go “self service” and spare a lot
of pain and time in dealing with IT. I don’t think any serious manager going
into the cloud is doing it for the savings. But of course this is a bank and
technology is their future so it makes sense that they want to keep a tight
loop on the cloud activities.

------
hnthrowawaybofa
I used to work for BofA as a quant in their Charlotte HQ. Their cloud decision
is the least surprising. You really have to see it from their pov. My boss
used to say - "Charlotte is a 2-horse town. You either work for Bank of
America, or you walk across the street to Wells Fargo"! Compared to these 2,
the other companies are much smaller along most axes (market cap,employee
count, or just sheer heft). There's an uptown walkway( like a private overhead
glass tunnel) that safely escort bank workers from downtown to bank without
coming into contact with riffraff :) Its just a whole new level of planned
design. Imagine if all the FAANGs had an interconnected private glass tunnel
walkway that looked down upon the unfortunate denizens of MV/PA/SF,while the
chosen ones swiftly segwayed from FAANG to FAANG whilst checking in their
latest git commit into kubernetes or whatever it is you guys do :) That's what
it was like. Quite unreal.

But all this clout leads to a rather head in sand mindset on most strategic
items, like technology choice, programming language choice, cloud choice (or
non-choice in this case), version control choice etc. Everything was done in-
house, in the most boring safest way possible tech ( mostly Java about 3
versions behind, some strange python lib where any function call was
automatically logged!, and Excel & Matlab all over the quant land. I mostly
sftp-ed financial data from some Quartz cloud...felt very quaint to do these
sort of things in 201X. All laptops were locked down windows dell boxes on
which you couldn't install anything, & ran some strange norton antivirus which
hogged all the memory. My interview itself was so old fashioned. I thought
since it was a Quant job, I'd get questions on math & finance. They trotted
out their "chief developer" who wanted to know how to model a chair with 4
legs using Java OO. You know the 1990s Grady Booch garbage full of UML, with
parent Table class & child Table & Leg Class & friend function & all that
jazz. I was like Jesus this regressive inheritance based OO shit is still
alive! Its a deeply old fashioned slow moving place. Very large IT budget with
pretty much half of Charlotte working in some capacity for the bank. So yeah,
if you had all the personnel & all the money, why wouldn't you build your own
cloud. You are paying for all these people anyway, might as well give them
something to do. Of all the employers I've worked for in my lifetime, this was
the one place where I was personally asked NOT to work so hard, because I
stayed at my desk after 5:30 pm.

(Sorry I have a regular 4 digit HN account but the bank doesn't like it if you
talk about them. One of their lawyers once tracked me down because I mentioned
some harmless datapoint about a technical problem I had worked on.)

~~~
svd4anything
> One of their lawyers once tracked me down because I mentioned some harmless
> datapoint about a technical problem I had worked on.

Ok so now you said that won’t it be pretty trivial to identify you again? How
many “4 digit HN” users have they really tracked down before .. 1?

------
vshastry
You’re also neglecting TTM as a consideration. As an example, I’m stuck having
to wait 6 weeks to get new hardware provisioned into an available slot in one
of my data centers to spin up a new Hadoop cluster, something my team could
probably do in a couple of hours in the cloud.

That being said, we control the HW and SW stacks end to end so I don’t have to
worry as much about the nightmare scenarios the NordVPN folks went public
about today. Critical given we’re in fintech ...

~~~
tbyehl
Friend of mine at a payroll processor you've all heard of was excited to
migrate to AWS because it took 4-6 weeks for IT to provision a new VM. On
their app team's dedicated hardware!

So much of the Fortune 500 lust for public cloud seems to come down to working
around inefficient IT procurement and provisioning processes. They haven't
automated, they don't maintain enough excess capacity, they haven't managed
vendor relationships to assure fast order turn-around, financial controls are
too onerous, etc.

Everything's virtualized but otherwise they're still operating like it's 1995.

------
bloody-crow
> The bank, which has a $10 billion annual tech budget

Am I the only one completely befuddled by this number? What the fuck are they
doing with these money and 200k servers? These are facebook numbers. For a
bank. What?

~~~
gpm
I'm not sure why you find this surprising, a bank has a much more complicated
problem space than Facebook.

\- They have a reasonably similar number of users (a fraction, but a large
one).

\- Mistakes cost a lot, so they have to be a lot more careful. It's a lot
easier to make money hacking a bank than hacking facebook.

\- They have to comply with all sorts of regulations.

\- They probably don't trust their own employees to not be trying to commit
fraud.

\- They have to parse data on a scale that is likely similar or greater than
facebook's. To detect fraud/lost credit cards/.... To decide who to give loans
to. To price insurance. To decide how to trade stocks. ...

\- They have to run a physical fleet of devices in the field, outside of their
control, that have to give people the right amount of money ~100% of the time.

At a glance I see that Facebook has something like 300 petabytes of data [0].
I've worked at a bank, my team had something more like 10, but I don't think
much of it was things like video that are just naturally huge. BOA is also
approximately an order of magnitude bigger than the bank I was at.

One rumor I heard while there was that there had been a bug in one of our
mobile apps that had been costing us a million dollars a day in server time.

[0] [https://www.brandwatch.com/blog/facebook-
statistics/](https://www.brandwatch.com/blog/facebook-statistics/)

~~~
SomeOtherThrow
How do you spend that much on software and have such an abysmal usability and
security story? I don't _think_ there is anything technically difficult about
the consumer software they offer, namely
[https://www.bankofamerica.com/](https://www.bankofamerica.com/).

~~~
gpm
I think you would find that their security is better than you think. Otherwise
they'd be hemorrhaging money left right and center to North Korea and the
likes.

As for usability, probably a degree of incompetence, mixed with design-by-
committee and legacy. Edit: It's worth pointing out that banks usually don't
gain or lose customers based on their UX, so it's not something that the
business optimizes much.

I don't think there is anything technically difficult about almost any of the
consumer software facebook offers, except scale. The same applies here but
exchange "scale" for "scale, reliability, security, and regulatory
compliance".

~~~
SomeOtherThrow
> I think you would find that their security is better than you think.

You can get in to my account by verbally relaying my grandfather's first name
over the phone. You can open a bank account with a SSN and no photo id. What
security? Their "security" is a fraud department, much like our credit card
industry.

> Otherwise they'd be hemorrhaging money left right and center to North Korea
> and the likes.

This is not how transactions work.

> Edit: It's worth pointing out that banks usually don't gain or lose
> customers based on their UX, so it's not something that the business
> optimizes much.

All banks offer the same shitty experience. What does differentiate them if
not their software? They offer literally nothing my local credit union doesn't
offer.

> I don't think there is anything technically difficult about almost any of
> the consumer software facebook offers, except scale.

No argument here, but facebook at least manages to hire designers and not
impose weird non-sensical patterns of auth, like "Look for this image when you
log in".

------
sairahul82
If you know your traffic well, private cloud saves a lot of money. Few years
back i bet hybrid clouds will take over and for some reason it did not happen.
I still believe hybrid cloud is the solution for mid size and up companies.
You definitely need cloud provides for handling traffic spikes.

~~~
api
Nobody ever got fired for using AWS.

~~~
shrubble
I am not at liberty to say the company name, but i do directly know of one
person at a director level who was fired for using AWS.

They turned on encryption for an option and did not realize the huge price
difference between the encrypted and unencrypted option meant $9000 per day in
additional charges. After 1 month (30 days) this came to light, and he was
fired.

~~~
brokensegue
that's not what people mean by the phrase "Nobody ever got fired for using
AWS."
[http://wiki.c2.com/?NobodyEverGotFiredForBuyingMicrosoft](http://wiki.c2.com/?NobodyEverGotFiredForBuyingMicrosoft)

~~~
shrubble
Before that it was 'IBM' ...

------
robocat
If the 2 billion were saved over two years, that is an increase in profit of
~4% over those 8 quarters ((75-56)÷56÷8)§. That's a pretty good outcome.

§ = Q3: "The bank said Wednesday that net income excluding an impairment
charge rose 4% to $7.5 billion, or an adjusted 75 cents a share. When
including the $2.1 billion charge tied to the end of a partnership with First
Data, net income fell to 56 cents a share"

------
cbsmith
That implies their business would have data center costs beyond most tech
companies (Lyft for example is spending $300 million with AWS over three
years, Snap is spending $3 billion over five years).

Something doesn't add up here.

~~~
cortesoft
Tech companies don't necessarily spend more on computing resources than non-
tech companies. The article says they had 200,000 servers in 60 datacenters.
That is certainly more than most tech companies, and is not surprising. A
large bank probably has more computing needs than someone like Lyft.

~~~
cbsmith
200,000 servers isn't actually that many. 200,000 c5.xlarge's at on demand
rates (which nobody with 200,000 servers pays) will cost you about $300
million a year. So they are realizing savings at about an order of magnitude
higher than you'd expect cost wise.

I used to work in banking... their compute needs are actually quite basic.

I think what we're really seeing here is some creative accounting around how
operations and costs are accounted for. They've got to be talking about costs
of managing and operating their cloud as well, and most of those costs would
be in terms of training & workforce retooling.

------
mfer
> Right now, the bank estimates its private cloud is 25% to 30% cheaper than
> public providers, though it also recognizes that probably won't last
> forever.

I am glad they shared this situation. We often here marketing telling us that
public cloud is always better. It's nice to see different perspectives being
shared and some details around them.

~~~
vinay_ys
If you have $50million+ you can deploy on capex in one shot (economies of
scale negotiation doesn't work well at lower numbers), and you can hire
systems engineers (C/C++/Go/Rust 5+ years systems experience x 20 engineers),
you can put together a _single tenant_ compute cloud needed to run a typical
Internet facing web/mobile application with all the bells and whistles (lots
of app server clusters, db clusters, big data clusters, GPU clusters etc with
HA, scalability, security, DR etc). At this point, capex/opex cost will be
easily 30% lower than the best negotiated bulk price from public cloud.

It is worth doing this if your in-house engineering team is at least 100+ app
service developers and have high feature churn rate.

But be warned, your systems engineering / shared technology team should be
level-headed and mature and rest of your app service engineering should be
good too to pull this off well. If not, you will be in serious developer
productivity pain and there won't be quick and easy fixes once you have put
down the capex.

------
blunderkid
Public cloud is definitely a great value for startups. When you are small and
want the flexibility and speed and the sophistication that cloud providers
make possible. The premium they charge is well worth up to a certain point at
least. When you are an established business with mature tech and of the size
of BoA, it is a whole different math. Your investment in your own cloud if
done right can be a huge saving that can give more mileage (on Wall Street)
than your competitors speed in execution, esp if you are a bank, where time to
market isn’t exactly a winner.

------
adventured
Outline: [https://outline.com/nDEM3d](https://outline.com/nDEM3d)

~~~
downandout
People seem to be posting hard-paywalled links more and more around here. They
were supposed to be banned, but I guess the rules are selectively enforced by
@dang here nowadays. The link to the article should be changed to this Outline
link.

------
cm2187
All the flexibility that the cloud offers is wasted on a large bank. I
remember one of the selling points of cloud computing for developpers is that
it takes months to provision a physical server in a large organisation. With
[enter cloud name here] a developper can create a VM in only a few minutes!

Well, enters a typical bank bureaucracy, cost controls, approvals, etc. And
now we are back to taking months to provision a VM!

------
reilly3000
Is this a story about virtualization? I fail to see the distinction between
data center and private cloud in this context.

~~~
rolltiide
I was going to explain how there's no difference but then I thought about the
more facetious terminology

cloud - someone else's computers

private cloud - someone else's computers that you own???? or that nobody else
uses, but in this context you also own???

~~~
rlpb
Cloud also means access on-demand by API.

Private cloud usually refers to that same abstraction layer. Hardware
provisioning is abstracted away from use and available on-demand from a pool,
and likewise deployment works automatically via the same or an equivalent API.

That's very different from the operation of a typical company-owned datacenter
from a couple of decades ago.

------
gowld
ITT: people who didn't read the article and believed the false headline that
Business Insider made up.

The 3rd sentence of the article, in bold, explains that the savings were from
consolidating on-prem hardware, not from eschewing external cloud.

The article cites BofA itself at sayignthat savigns vs external cloud is
tenuous at best:

> Right now, the bank estimates its private cloud is 25 to 30% cheaper than
> public providers, though it also recognizes that probably won't last
> forever. Still, the company believes the architecture it has built will give
> it leverage in negotiating contracts with these companies

------
ivanjaros
cloud will always be more expensive than bare metal. period. cloud is great to
start out when you have no clue about how much power you will need to satisfy
your customers and to be flexible enough to cover some unexpected spikes. but
once you're out there earning, the only reason to stay on cloud is comfort.
business-wise it is a black hole for money. many will argue that cost of
personnel to manage your own infra does not favour it over cloud. but if you
do the math you will see that savings when renting only three bare metal
servers each month will pay a full-time salary for a sysadmin. you don't even
need to own your own hardware. the bare metal rentals these days are insanely
low and if you truly want to save each penny, you buy your own because in a
matter of months you get yor investment back on saved fees alone. the thing is
that these big tech companies convinced so many people these days that cloud
is the only way that they cannot even fathom how anybody can run on their own
hardware. the new sysadmins are already "brainwashed" and inexperienced this
way and the old school guys that have experienced managing their own hardware
are slowly "dying" out. soon, just a mention of managing your own hardware
will become a joke to these newcommers when in reality the cloud was never-
ever a cheaper option to go with in the first place. not to mention that these
days we have so much technology to make things easy(kubernetes, rancher,
proxmox, docker, lxc..) that it is quite laugable to fear bare metal and
religiously praise the cloud like some kind of savior.

------
commandlinefan
I continue to be amazed that any business of more than a few hundred employees
would ever consider using off-prem hosting. Cloud services are there to
bootstrap a business, not maintain it.

~~~
sjg007
Well the issue is that you bootstrap on AWS and then have a hard time going on
premise without a major rewrite or initiative. That and you want to keep
adding features and driving the product forward... it is easy to see how this
happens. Plus you can have 200 people focused on things other than running ESX
boxes or docker or whatever you need.

------
jariel
Folks are generally misinterpreting the 'cost' of the cloud by mostly looking
at data centre costs, margins etc.. The 'TCO' or 'Total Cost of Ownership' of
cloud services is generally much lower than otherwise.

The cloud is valuable because of the variety of features, immediately
available, at very large scale, run by very credible people with $0 in capital
expenditures.

In most cases, it's worth _far_ more than the price, which is why it's so
popular.

In very, very few cases is it 'cheaper' to run one's own cloud given the
massive operational overhead of hiring people with very specialised skills,
risks all over the board, the enormous planning, extensive investments.

Wallmart must have 1000's of trucks but they don't 'roll their own' delivery
vehicles, because there's an inherent degree of specialisation in making
vehicles.

So yes - in some specific cases it makes sense to 'host': say Dropbox, just
wants massive cheap storage, or Google, has special needs & scale & ability,
or some entity has very specific privacy requirements - and of course I wish
there were more variety in providers, especially localised ... but the cloud
is essentially now the default hosting option and that's not going away
because it makes a lot of sense.

The surpluses generated by such services are vast - it's a big step forward
for everyone. Companies using AWS are probably profiting more in surplus than
the margins in AWS.

------
crawdog
"Building its own cloud" = leasing data centers from Equinix or other large
data center providers. Financial services have challenges using public cloud
due to regulatory and compliance requirements. Most of these challenges are
self made - artifacts of moving their teams from "This is how we currently do
it" to a shared services model.

Curious if they are all in on providers like Redhat Openshift or Pivotal Cloud
Foundry as their PaaS layer.

~~~
coredog64
Given how little money Pivotal was making from Cloud Foundry and how expensive
each license was, I’d bet a large sum that the answer is no.

(Some context: My last employer had an AWS bill of about $3m/year that was
mostly EC2. Running PCF on top of that would have been another $2m/year in
licensing. And that was after the volume discount.)

------
petarb
“Right now, the bank estimates its private cloud is 25 to 30% cheaper than
public providers, though it also recognizes that probably won't last forever.”

~~~
hinkley
After a meetup this spring, someone talked my ear off about how Dell is highly
motivated to have a private cloud solution that works for people.

Having Dell and Amazon in a bidding war over your next project is probably the
best world you can be in.

For my money, you should run one data center in the same location with most of
your tech talent, and a second one geographically distant, and regionally load
balanced.

But we typically don’t write our software for this, and you can’t get the
business to do a rewrite until they see how stupid expensive a lift and shift
ends up being.

I think these narratives about saving money are typically covering up a story
of how much was squandered starting five quarter ago...

~~~
coredog64
Unfortunately, Dell’s private cloud solution exists today, and it’s an 18
layer shit sandwich of all the disparate companies that Michael Dell has
stitched together into his holding empire.

------
graycat
I have a question: I'm doing an _ambitious_ Web site startup; current status,
rushing to alpha test; and from this thread and more I wonder about servers
and our _server farm_ and in-house network, that is, for the options here, in-
house, co-location, AWS, Azure, etc. I'm using Microsoft's software, Windows,
.NET, SQL Server, etc.

Q. For many of the reasons given in this thread, etc., I'm leaning to having
our own in-house servers, server farm, etc. For that, is there a source of
information I will be able to use, say, as consulting, depending on the issue,
an hour, day, week, month at a time, to get us past the chuckholes in the road
on the usual work -- system and network planning, installation, configuration,
monitoring, diagnosis, correction, etc.? E.g., can I just call Microsoft for
such issues, CloudFlare, VMWare, Cisco, etc.? Assume that money to pay for the
products and consulting will not be a problem.

If the broad answer is "Yes", then that will take a lot of entries off my TODO
list and let me sleep better.

Thanks.

------
beardedman
For what it's worth, I really don't like AWS - aside from their
S3/Route53/CloudFront products. There is a layer of abstraction that is so
minimal that you may as well roll out the services yourself & save a ton of
money. Of course, if you like that layer, then that's fine too.

~~~
Aperocky
If you look beyond that, AWS offer layers of services that you might not
realize existed.

Cloud DB with no worry about physical hardware (in addition to S3 buckets
which is mostly for files). lambdas when you don't want to manage a server,
cloudformation when you don't want to manually start your stack by clicking,
API gateway for quick ways to create APIs without worrying about stuff. And
powerful orchestration layer that connects all of these together.

In fact, a lot of these services are bootstrapped on AWS itself. If you know
your way around AWS, you can really find a sweet spot where it's cheap
scalable and mostly worry free. The problem is that AWS domain knowledge in
itself is _deep_ and not many people know enough about AWS to avoid large
amount of pitfall and inefficiencies that a knowledgeable people can quickly
spot and fix/alternate.

------
xrd
My experience working at an aging e-commerce company that has their own cloud:
totally shitty.

They used off the shelf management software that they misconfigured. Getting
VMs was so painful and they never worked correctly.

When I came back from paternity leave, they had gotten no response from their
automated system and deleted all my VMs even though they could have seen heavy
usage. If they had manually followed up they would have gotten an out of
office email but they didn't.

Those are just two of the examples of how terrible it was to use their
internal cloud.

You are always going to be five years behind if you do it internally. That
might be a good cost calculation and I wasn't in a position to argue.

But, it meant innovation was terrible there.

------
gamesbrainiac
Something does not make sense here. The main cost savings seem to come from
using fewer servers. Why could you simply not do this on Azure or GCP or AWS?
Are the advantages coming from leveraging bare metal servers? I'm a little
confused.

------
spicyramen
For BoA we need to consider that is a Bank and many of the information they
store may be too sensitive to pass it to a third party which as we know always
have security issues. Better to deal with those internally. Many factors to
consider in this case: real estate, expertise in-house to build data centers,
budgeting, technology required. In our case without Cloud would have been very
difficult to bootstrap our business and manage it, as most of the budget was
to hire engineers to work in our product and pay bill.

------
vkaku
In my opinion, a company can save money on a cloud if it has these things:

\- Access to cheap real estate

\- Access to enough human capital, especially a team of 80+ to operate all
aspects of a cloud: hardware, switches, openstack / kubernetes

\- Requires scale of 20k+ servers = 320k+ VMs and a strong inclination to not
host on AWS/Azure/Others.

\- Lease to servers costing about $250 / server / mo. With AMD, you could go
half of that, with VMs costing $10 a month. Add network costs of $15 / 2 ports
/ server / month.

------
manigandham
> _" Now, it's pared that down to 70,000 servers, of which 8,000 of those are
> handling the bulk of the load."_

I wonder what the other 62000 servers are doing.

~~~
eitally
Lots of dev/test, storage, R&D, BI/reporting, and miscellaneous internal
business apps most likely.

------
SomeOtherThrow
Bank of America should really not brag about the quality of the software it
makes based on its consumer website. Maybe they should consider spending more.

------
narcindin
JPM's Jamie Dimon talks about his cloud strategy on the most recent
shareholders letter, page 34.
([https://www.jpmorganchase.com/corporate/investor-
relations/d...](https://www.jpmorganchase.com/corporate/investor-
relations/document/ceo-letter-to-shareholders-2018.pdf))

Note that he leaves open the option for both public and private cloud.

------
safog
So one thing people seem to be ignoring is that cloud != automated
provisioning of VMs by engineers.

There’s so much more AWS, GCP and Azure offer these days that trading away all
the stuff like metrics, monitoring, dynamo-dB, s3, lambda, scalable RDS, Spark
pipelines and generally a whole suite of products that go into building a
modern web app for some scripts that can provision on demand seems to be a
poor choice.

------
gowld
Counteractual: Suppose BofA wasted $2B per year by building its own cloud. Or
is BofA decided to migrate from woncloud to AWS, and wasted $2B in the
process? Would a CEO make a public statement about that? Or would they fiddle
the numbers (which are impossible for an outsider to verify) to make their
decision look like a winner?

------
vinay_ys
70000 servers in 23 datacenters – sounds awful. Why have it spread out so
much? 350 million capex for 70k high end server DC setup may make sense.

$2B = 40% savings means their opex was $5B and now it's $3B. That is high
expense. Guessing most of it is licensed software costs? Like Oracle on every
single core?!?

------
grantseltzer
I'm not so convinced how great of a job they did.Bank of America has by far
the worst user experience I've ever seen. I had an experience recently where I
changed my password, and after 2 days my new password stopped working and my
old one did.

~~~
mcv
That's not good, but it's also only one of the many, many things a bank does,
though. The most important one is keeping the money safe of course. It's
entirely possible for a bank to do some things right and other things wrong.

------
formercoder
This is simply a fixed cost vs variable cost decision. Fixed costs give you
operating leverage which amplified your earnings in good times and bad.
Variable costs don’t have the downside, but impact your margins.

------
AlexCoventry
Do banks using third-party cloud services just trust that the provider won't
abuse their data? Do they actually store financial data there, or just use
them for front-end stuff like websites?

~~~
lmkg
The trust will come from very stringent contracts (beyond the regular
contracts that other customers would use), third-party certifications that the
provider achieves, and the provider being covered by certain aspects of the
regulations that cover banks (or healthcare or whatever).

------
gok
That BofA had >$2B/year of IT spending is the real story here.

------
sam0x17
That's almost as much as they make on overdraft fees!

~~~
mindslight
That's unfair - they were only fined $67 million for fraudulent overdraft
fees! Though I have to wonder how much they're currently making by inducing
every one of their customers to write fraudulent checks.

Though there is something comforting about a predictably evil company though.
Much more straightforward than "do no evil" while building a surveillance
machine that would make a Stasi officer blush, or "this helps our users" while
hindering Hong Kong's protests.

------
jtlienwis
To quote Steve Oberlin of Nvidia ( quoting House M.D.) about cloud
customers... "Everybody lies."

------
swedish_mafia
Can anyone who has worked on Bank of America comment on if any of the things
they build actually worked?

~~~
theincredulousk
This must be a jab, but obviously yes.

Last year they had $28 billion in net income. With that kind of money you can
hire the best in the world to build literally whatever you want. They could go
build a space shuttle if they felt like it - probably without much trouble.

In the unlikely situation that it didn't work, they'd pay someone else to fix
it or re-do it. The least likely (albeit not unprecedented) end situation is
that 'none of the things actually work'.

~~~
brokensegue
Boeing can't even make an airplane without trouble and you think BoA could
churn out a space shuttle? The space shuttle program was estimated to have
cost $200B.

~~~
theincredulousk
Boeing has made literally tens of thousands of airplanes "without trouble".
One debacle doesn't mean they "can't even make an airplane".

"Space Shuttle" was rhetorical, but yeah, they could churn out a space
shuttle. They could buy SpaceX and use their engineers to do it. BofA has $2.3
TRILLION in assets.

The point is you can buy expertise. It's not like a bunch of bank tellers
coded up a cloud platform. Nor would you have a regional branch manager
running an aerospace program. You'd hire people that know how to do those
things the same as any other expert at any other company.

------
kerkeslager
Obviously small businesses don't have the option of building their own cloud,
but there is a middle ground: use public clouds, but avoid coupling yourself
to them. Most cloud providers provide servers that can be treated like a
barebones Linux server, and that's all you really need to be productive with
most server applications.

~~~
bcrosby95
Then just use a colo center. It's worked for us for close to 20 years.

Last time someone had to perform hardware maintenance was at least 3 months
ago. If you're small enough (20-30 servers for us) the things mostly run
themselves. Any other overhead we have would be present if we were just using
something like EC2.

The bulk of our hardware is 10 years old. We have some servers that are 17
years old and still chugging along.

~~~
kerkeslager
> Last time someone had to perform hardware maintenance was at least 3 months
> ago. If you're small enough (20-30 servers for us) the things mostly run
> themselves.

If those servers are production servers, I'd say that's not small--you're
already scaled to the point where you have hired a whole person (or two) to do
server work, no?

I've worked at places where I'm the only technical person. I know enough to
set up a Django server, but the reality is I'm not an IT guy, I'm a developer.
Paying for a few cloud servers means we don't have to hire anyone to do server
maintenance, which gives me an entire person's salary as a server budget
before we have to consider whether hiring a person would be better. A lot of
companies never reach the size where that becomes a concern.

Of course everybody wants to get big enough that they need a server farm, but
as long as you don't vendor-lock yourself, moving from a virtual Linux server
on the cloud to a physical server in a colo center is fairly trivial.

My personal website has been running on a single $10/mo Digital Ocean server
for 5 years. I think it would be very difficult to beat that price with a colo
center and physical hardware. Most businesses are obviously larger than that,
but I think it's a bit of an overreach to claim that it's always more
expensive to run on the cloud. It just isn't, for many real-life situations.

------
trident1000
"the cloud" = your data is now our data on our computers (I think we all know
this). When companies like Microsoft start saving my files to their personal
servers as default, I hate you. Stop.

~~~
cltsang
When did that happen? Assuming we are not talking about Azure China.

------
Account123481-x
why do I not believe this?

~~~
lsllc
Serious question: Why is it that everyone believes that going to the cloud is
cheaper? How is it that AWS can buy the same servers, same switches same
everything, make a fat profit and somehow it's sell it cheaper than you can do
it? I get that there are economies of scale, but not 2-3x right?

Sure, you don't need those admins to rack up new servers, but don't you just
trade those costs for costs around AWS "devops"? (IAM / authentication
anyone?).

I keep hearing execs telling me that moving from self-hosted to AWS is going
to save a ton of money ... but I just don't believe it. I would assume if
you're that bad at running an IT business in-house, you'll be just as bad at
it in the cloud.

------
panarky
Bad title.

They didn't switch from AWS to private cloud.

They consolidated 60 data centers with 200k servers to 23 data centers and 70k
servers.

~~~
valbaca
The title isn't bad: "ignoring" implies they were never on AWS and just went
their own way.

~~~
jhall1468
The savings implies they saved $2B over AWS. I can't read the article since I
don't have a BI account, but unless they saved $2B over AWS (rather than $2B
over their previous infrastructure) it's definitely misleading.

------
not_a_cop75
Get outta my dreams. Get into my cloud.

------
m0zg
200K servers to serve ~50M households? Seems rather inefficient. What were
they doing with all this hardware?

------
nartz
This is completely false - I imagine they didn't take into account how much
the migration and upkeep actually cost them. AWS's promise is not that its
infrastructure itself is cheaper in terms of number of servers, or otherwise.
Its cheaper in that it helps automate a lot of manual things away.

~~~
pacofvf
I had experience in Quartz (BofA cloud), and deploying is 100 times easier
than AWS, everything is automated. Imagine building a cloud service where you
trust all your clients, and all must share the same information if they have
the correct auth, what I trying to say is that their use-case actually made it
simpler, that's where the savings in software had came from.

~~~
Mathnerd314
So we'll hear about a BoA hack in a few years where the attacker got into the
cloud somehow and then had unlimited access to all the other servers? Great.
Exactly what I want from a bank. /s

~~~
cnlwsu
Or you can look at it as a single correct auth and encryption mechanism is
shared company wide vs each individual teams intern inventing new was to
base64 encode your password. Glass half full or empty

------
vijucat
I have a bullet proof plan to earn a $10m bonus in my two years as CTO at
<Large Firm>:

1\. Identify retarded firm that uses AWS or Azure like money grows on trees
AND without too much AWS lock-in

2\. Join as CTO and move infra to OVH

(2.1 Convince OVH to not irritate international customers by making my browser
show "Translate this page?" for EVERY SINGLE page, even on us.ovhcloud.com,
because they HAVE to show their French lineage at all times instead of just
having English pages in English! They've already improved their Customer
Service a lot, so I have great hopes)

3\. Reduce server costs by 4X

4\. Profit!

Seriously. Unless you really need auto-scaling or one of the other unique
features of AWS / Azure products, you don't need to pretend you're the next
Facebook or Netflix.

Also seriously: de-cloudification is a great business idea.

