
Tell HN: Wells Fargo completely offline - p3nt3ll3r
Wells Fargo is completely offline. How does this happen? Been this way since 6am PST!
======
jedberg
For everyone complaining about "how could they not have failover" let me ask
you this: Would you take a job with Wells Fargo to fix their infrastructure?

And if you did take such a job, how long do you think it would take you to get
the budget and approvals from all the auditors necessary to fix everything?

How much would they have to pay you to take that job, knowing how frustrating
it would be to get anything done?

And now you see why banks have such terrible IT.

(One of my mentors actually works at BofA, and says he only does it because he
gets to work 6 hours a day, gets a VP title and a ton of money, but nothing
ever gets done)

~~~
benoror
Nice mentor you got there

~~~
jedberg
Heh, well he's also a brilliant technologist. He just has other priorities
now.

------
nemothekid
Apparently there was a fire in a Datacenter - theres a thread on /r/sysadmin
by an insider.

[https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...](https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_down_declining_transactions_and_no/)

~~~
mabbo
Incredible. They had all their mission-critical infrastructure in a single
data center. How many billions of dollars do they make per year? And they
can't afford even a tiny bit of redundancy?

If I were a customer, I'd use this as a sign that this company is not
technically competent enough to manage my money.

~~~
surge
They had redundancy, the redundancy failed.

I mean truthfully when do you get to test your redundancy against a true
disaster. It was a mess. WF is 20 companies rolled into one so the fact the
disparate systems from 10 different banks works at all is kind of a miracle.

~~~
pushtheenvelope
Facebook regularly takes down multiple data-centers at a time to stress test
this. Its users rarely notice (which I think is the point).

~~~
shagie
There is a different level of criticality for "my post didn't go through, hit
refresh" and "my transaction didn't go through - the restaurant said my card
isn't working."

Would you honestly want to go to a bank and say "if we unplug this, we can
find out what fails."

~~~
computerex
Facebook handles money too, though. Also I think the parent was making the
point that Facebook builds software with resiliency in mind so when a failure
does happen, the software deals with it gracefully.

~~~
krageon
They can have (and did have, at least the long time ago when I still used it)
weird cache persistency errors and "please just refresh to fix that" type of
workflows if you have bad luck. That sort of behaviour is simply not
acceptable for a bank.

~~~
computerex
Are you talking about Messenger? That was a front-end issue, and they created
React/flux to fix that.

------
Talyen42
3rd largest bank in the world offline for an entire day because a smoke alarm
went off

great job guys

~~~
qaq
pretty interesting attack vector

~~~
drdeadringer
I feel like I've seen this in 'Mr Robot' or similar.

------
clairity
for those who haven't taken the plunge, now is a good time to move to a credit
union, regional bank, or online bank:

\- for instance, in cali: [https://www.golden1.com/](https://www.golden1.com/)

\- high interest accounts: [https://www.bankrate.com/banking/checking/best-
checking-acco...](https://www.bankrate.com/banking/checking/best-checking-
account-rates/)

\- good rate online bank: [https://empower.me/](https://empower.me/)

get better service, lower/no fees, and good convenience without all the
economy-damaging selfishness and hubris.

~~~
legohead
I imagine a credit union would be susceptible to the same type of thing
(single location/point of failure) due to their smaller sizes.

~~~
tatersolid
Credit unions and small banks outsource almost all of their core systems to
companies like Fiserv. Literally thousands of banks and CUs all ride their
infrastructure

~~~
themodelplumber
I ran a website for a local CU for about 10 years. It was pretty fun doing
things for their website that made their Fiserv web tools look like they were
from 1970.

There was no interaction with user or account data on my end, just CMS-
building, letting them go crazy with coupon promotions or pet shelter PSA's or
warnings about ATMs being down, etc. They'd pitch me on a new project, I'd
send them a timeline, and we'd add new features or tools to the website. I
even hired a local college student to create some super-basic but useful
financial tools for the site.

I loved how small-town-feel the whole thing was. At the beginning of the
relationship, they gave me a list of broad requirements like "SAS-70", I found
a DC to match, I sent them a contract, and I couldn't believe I actually had a
banking institution as a client, 4 years out of school and a brand-new
business owner.

Eventually they merged with another CU and went away, but apart from the very
occasional "server down" notifications while I was on vacation, those are some
really fun memories.

------
bridanp
Regional bank checking in. We've been told to reroute certain workloads
through other means until 6am central Friday. Regardless of that, the story
coming out of this over the next few days is really going to help Business
Continuity departments at other institutions plan for this scenario. My teams
are running mock scenarios this afternoon based on the auto power shutdown
causing a data center to be unable to fail over. Might end up being something
else entirely, but it's still a good scenario to investigate.

~~~
a3n
I get paid by direct deposit into my WF acct between midnight tonight and 6am
tomorrow. I suppose I can eventually rely on my employer keeping records and
replaying everything.

~~~
bridanp
If ACH areas are involved, hopefully they already had those queued a day early
per normal routines and you'll (hopefully) be fine. There are exceptions
allowing companies to provide ACH files late. That's the exception and not the
rule.

------
habosa
For those who have worked in bank IT, I have some questions.

So these days there's very little physical money. Most of my "wealth" is just
entries in various digital ledgers. My bank says I have $XXX and my brokerage
says I have $YYY and my retirement account says I have $ZZZ.

Let's go with the bank account case. Is it possible that a catastrophic
accident or attack could wipe my balance down to $0 with no way to recover?
What if a data center was nuked? What if two data centers were nuked?

How much redundancy is in the system? Are there third-party agencies that
track private bank ledgers? How hard is it to take them out too?

Ever since I read "The End of Alchemy" (a great book, btw) this thought has
haunted me.

~~~
40acres
Generally I think here is where the FDIC and SPIC would come in.

~~~
gdubs
I’d love someone who knows the details to explain how that claim process would
work, because while the FDIC would insure in the event a bank goes under, in
that scenario there’s usually records.

What would happen if there were no records? Surely there’d be lots of people
making significant deposits in between snapshots. If you’re “paperless” I’m
not even sure how you’d reconstruct your balance.

Would certainly be time consuming, and in this case time would certainly be
money.

~~~
Aloha
You presume the live system is the only record - its not, there are backups of
those, then historical records to reconstruct what might have been there.

Bear in mind the live records are only one part of what a bank keeps.

------
rubbingalcohol
You can still access Wells Fargo Online via the direct link:
[https://connect.secure.wellsfargo.com/auth/login/present?ori...](https://connect.secure.wellsfargo.com/auth/login/present?origin=cob&LOB=CONS)

~~~
sxates
Confirmed - was able to log in. Some account info is still there, though my
mortgage is unavailable. Maybe they'll just lose it?

~~~
howard941
Your deposits, sure, your trust, of course. The mortgage loan? Fat chance.

~~~
propogandist
Unless evil corp was using a blockchain to have an immutable ledger, there's a
chance (although extremely small)

~~~
tastroder
If evil corp stores said buzzword stack exclusively on that large rack that's
currently on fire because of compliance nothing is gained. Not every problem
is a nail.

~~~
mulmen
I don’t follow, are you saying this outage was caused by compliance or that
the blockchain can’t help here because of compliance?

~~~
tastroder
Neither, I was trying to suggest that offering Blockchain and related
technologies as a solution to every problem in existence is kind of an
annoying trend. I don't really see the outage discussed here as a technical
problem. Quite literally any failover mechanism would have been preferrable
here.

------
webdestroya
From their API team:

Please be advised that the Wells Fargo Gateway is currently unavailable.

We're experiencing system issues due to a power shutdown at one of our
facilities, initiated after smoke was detected following routine maintenance.
We're working to restore services as soon as possible.

We apologize for the inconvenience as we continue to work on a resolution.

We will update you no later than 3:00 pm, Eastern Time.

~~~
rococode
Amazing that one of the largest banks in the US is somehow still (apparently)
reliant on a single location.

------
williamstein
Wow, functionality is still significantly degraded over 24 hours later. I just
tried to log in to check on an account balance, and the site was extremely
slow. It showed my balance, but failed every time to load the transaction
history with "Error in external system" or something like that. The sign in
page still shows: "Alert: Some customers may be experiencing issues accessing
online and mobile banking. We apologize for any inconvenience."

------
rolph
OK, Hanlon's razor applies here:

[https://en.wikipedia.org/wiki/Hanlon%27s_razor](https://en.wikipedia.org/wiki/Hanlon%27s_razor)

recall Stuxnet, its possible this was an attack, somesort of malicious mod to
firmware, but_ Hanlon's razor.

from the reddit:

"throwawayfordays75 1399 points 8 hours ago*2

Throwaway since I have first hand knowledge. Fire suppression went off in one
of their main Data Centres from some utility work this morning. No power to
any of the network or compute equipment and some failovers did not work as
expected. "

At this point im wondering what "utility work" was happening.

~~~
hinkley
That user posted an update:

"everything minus core network gear was manually being unplugged from any PDUs
to help the control the initial power-on."

Can't we... Why don't we have rack hardware that can handle this situation? I
thought some HDD RAID solutions had circuitry to keep them from browning
themselves out while spinning up the disks. I guess I'm surprised this isn't a
solved problem at the rack level now.

Or have we been so focused on never cold booting a rack of servers that we
haven't spent any effort on foolproofing of cold booting a rack of servers?

[Edit: answering my own question] apparently these exist and are called
Managed PDUs. Can we deduce WF doesn't have them?

~~~
mustardo
Some servers "stage" power on to disks. as spinning up disks is often the
largest source of inrush current for e.g. a 10 disk server might only apply
power to bays 2 at a time with a delay between each, this doesn't solve the
inrush current problem of switching on a rack full or servers together (where
managed PDUs come in)

------
api
Too big to fail over?

~~~
rawrmaan
Underrated comment

------
0xfeeddeadbeef
Looks like Mr. Robot is at it again.

------
nodesocket
I thought at one time Stripe used Wells Fargo as their bank. Any reported
problems from Stripe?

~~~
driverdan
I signed up for Stripe in 2011 or 2012. About six months after I signed up an
account rep from Wells Fargo called me asking about my merchant account.
Stripe never made it quite clear what was going on but apparently in the early
days they opened accounts at Wells Fargo on your behalf without telling you.

~~~
patio11
I also signed up for Stripe in 2011. The Terms of Service mentioned Wells,
beginning in the first paragraph. You might want to look in your files and/or
the Wayback Machine.
[https://web.archive.org/web/20111017081436/https://stripe.co...](https://web.archive.org/web/20111017081436/https://stripe.com/tos)

~~~
peter_d_sherman
Speaking as a Law afficionado, that is one _great_ Terms of Service document!
Kudos to whichever legal team wrote it! I'm not saying that it completely
anticipates all potential customer-company interactions that could occur (no
legal document could), but it seems to go very far, and seems to show a good
understanding of the many potential issues that can arise. My future company
might use this document (and others) for inspiration on how to craft its own
Terms of Service documents. Thanks for the link!

------
rixrax
And I’m not surprised they don’t have a services status page either that many
modern companies do offer[0][1][2].

[0] [https://api.twitterstat.us](https://api.twitterstat.us) [1]
[https://help.netflix.com/en/is-netflix-down](https://help.netflix.com/en/is-
netflix-down) [2]
[https://developers.facebook.com/status/dashboard/](https://developers.facebook.com/status/dashboard/)

------
fidla
4chan's pol forum is all over it. Apparently it's a "happening" worthy of
hundreds of comments

~~~
krapp
Really? /pol/?

 _this_ is what the merry band of racist edgelords have fixated on?

Seems like it would be more /g/'s thing.

~~~
lenticular
Anything that they can spin into some insane fascist conspiracy theory.

~~~
imsodrunklol
Normal Thursday for them.

------
byron_fast
I work at a place that just had a near 24 hour outage due to a frozen water
pipe. Water!

~~~
ams6110
Cooling water? Essential in a data center.

------
samstave
This will be an interesting root cause if it manages to get leaked....

------
bigbassroller
I got an email today from WF titled “New year. New look. Continued
commitment.” Maybe they are flipping the switch on their “New year. New
look...”; which sounds like a new redesign and they are having downtime while
switching over.

------
godelmachine
Well, for a second I thought they have closed business operations completely,
keeping in mind how embroiled they were in graft cases in 2008 recession.

Pardon my immense ignorance!

------
sugarygrind
#bitcoin -- just had to say this.

------
sys_64738
Lots of new Wells Fargo job vacancies will be published tomorrow I fear.

~~~
mikestew
Yeah, but it won't be to fill the roles vacated by those that failed to fund
data center redundancy despite repeated requests and proposals.

------
meesterdude
they have a history of outages - though not as long.

~~~
MuffinFlavored
What is more likely going down in the middle of the day? Their mainframe/COBOL
stack? Their Java stack?

~~~
berbec
Their entire data center, it seems

------
eweise
Try Varo

------
chad_strategic
Wells Fargo is nothing more than a federally regulated crime organization,
supported by the federal reserve and the American tax payer, but I digress.

This thread has primarily focused on redundancy and software architecture.
That could be the case, but there is no better way to fight proxy war via
hacking of banks. It’s a domino effect... pull your money out of the bank, the
bank calls loans, then there is credit crunch, investors lose confidence in
the stock, value lost etc... the enemy has secretly inflicted civilian
problems across the economy. Distrupting the banks and the flow of money can
lead to a revolution... or take your mind off of America’s enemy or diverting
energy else where.

------
deevolution
Bullish on bitcoin b/c of its highly resilient characteristics. Might not
process as many transactions per second and it might pollute more than the
average bank, but sure as hell dont need to worry about a natural disaster or
freak incident single handedly taking out the entire network.

~~~
jameskilton
Wells Fargo customers must be terrified that their money is going to be gone
when the service finally comes back. This downtime is obviously to prevent a
bank rush while the leadership leaves the country with millions in customer
funds.

Oh wait...

~~~
shawnz
Very funny, but you're describing a failure mode of bitcoin exchanges rather
than the bitcoin network, which is not what the parent was talking about.

~~~
spookthesunset
> but you're describing a failure mode of bitcoin exchanges rather than the
> bitcoin network

Of course! With bitcoin, nothing is the fault of bitcoin and always something
else's fault. Bitcoin can never fail. It can only be failed.

~~~
shawnz
No. It just affords you the possibility of choosing a different set of risks,
that's all. Please don't change my words to fit your idea of a cryptocurrency
shill.

