Hacker News new | past | comments | ask | show | jobs | submit login
DBS Bank outage well into its second day with no meaningful update from DBD (nasdaq.com)
83 points by audiometry 12 days ago | hide | past | favorite | 60 comments





Context: DBS https://www.dbs.com.sg/ is the largest bank in Southeast Asia and very, very proud of being "The World's Best Digital Bank":

https://knowledge.insead.edu/blog/insead-blog/how-dbs-became...

Note their own "GANDALF" acronym, composed of Google, Amazon, Netflix, Apple, LinkedIn, Facebook and, you guessed it, DBS.

Most companies in Singapore also pay out wages at the end of the month, so it's an extra bad time to be down.

On the other hand, their slogan is "Live More, Bank Less", so at least they're living up to the second half.


If they put themselves in the same league as LinkedIn, that explains a lot.

Shittiest website I ever need to use from a technical point of view, never mind “content” and dodgy hard-sell techniques.


They had to get the acronym to work, and presumably "LinkedIn's-owner" sounded a bit forced.

It might sound repetitive that recently only I wrote a similar comment. Sometimes it makes me wonder why are people in FB or LinkedIn put on a pedestal when the output (we can see) is bad quality websites and apps. Maybe I’m trivialising the problem considering the scale involved but still makes one wonder.

Purely from a technology point of view, I find Facebook and Twitter far superior. They "just work" (not that I'm a heavy user, and certainly not endorsing the business they serve).

LI is slow, buggy, crashy, search works poorly etc. The only thing they have going for them is the size of the network.


Twitter often fails to load on mobile browsers, with some error that sounds like "rate limiting" or similar (I forget the exact message).

I've heard of other people having the same problem, so it's not just me either.


Twitter videos for e.g. is pretty dumb. It tries to do adaptive bitrate for a 10s video. By or before the time the video quality improves, the video has already ended.

Also before the recent PWA app, their mobile site used to very bad. PWA sure also crashes sometimes.

Facebook newsfeed itself is so buggy to me but maybe it’s because of all the trackers are blocked. Maybe it works well if they are not blocked.


DBS is a bank that has spent a lot of money on marketing that "Digital Bank" image, but is really old fashioned in all the wrong places.

Not a fan of their video-teller booths/machine? I was impressed and I'd be surprised if, even today, many retail banks have something similar. Obviously good cost-saving for them, but seems like a reasonable solution to banking with a new (younger) demographics.

The idea of needing to go somewhere to talk to a teller seems fairly old fashioned, whether or not that's done via a video stream. My bank doesn't even have physical buildings - you just do everything through the app.

> The idea of needing to go somewhere to talk to a teller seems fairly old fashioned, whether or not that's done via a video stream.

I'm guessing that it's probably the best compromise they could think of to authenticate a person's identity, mainly for services that need to be performed instantly like debit card replacements.


Singapore has a great GovTech system that provides identity verification in a really nice way.

GovTech's SingPass has horrendous UX too: they refuse to allow the toggling of password visibility — which is pretty important when designing for accessibility, because the elderly sometimes have trouble entering their password, because it'll fail some security audit.

I think I’ve gone to a bank teller exactly once in the last decade, and it was for something that, for risk reasons, they required a paper form submitted on person for (large transfer for house purchase), so I’m not sure I see the point. What do people go to tellers for these days? Most smaller branches here don’t even have them anymore.

I guess we're limiting our discussion to digitization (so we can ignore any accessible issues with digital-only services).

The obvious thing that comes to mind is getting issues resolved immediately, e.g. debit card replacement (which is why I used their video teller machine). I've also gone to the bank for cashier's order (not sure why these can't be ordered online, but again, this could be time sensitive). And I go there occasionally to get large stacks of $2 for tips. I also do currency exchange in person (though not from a bank, bad rates, but I think it generally answers the "why do we need physical locations")


> I've also gone to the bank for cashier's order (not sure why these can't be ordered online, but again, this could be time sensitive).

It might not be what you're looking for, but cheques can be issued online using DBS internet banking [1].

[1] https://www.dbs.com.sg/personal/ibanking/faq/online-cheque.p...


I've tried using it once to IIRC, replace a debit card in the wee hours of night, but some functionality seemed to have been disabled after midnight.

On some level, you gotta give 'em credit for trying.

A couple of years ago, they tried to recruit me into a platforms/SRE role for their consumer banking services. There seemed to be a pretty clear desire to Modernize All The Things! to do things the way Google/Facebook/etc. do it.


I think the first part is true as well, I mean, a lot of people are probably feeling quite alive these past two days.

I like your sarcastic tone a lot, lol.

Adding my $0.02.

I've had a non-trivial # of bank accounts in numerous countries, and generally speaking, DBS is good. I think people constantly complaining about DBS have no clue how awful retail banking usually is (especially internet-banking), and that most definitely includes other Singapore banks. Apps are generally modern, usable and responsive, and I find their fees and transparency reasonable. But what sets the bank apart is their customer service, the people on the other side of the phone (or counter, those few times) are incredibly friendly and helpful (again, compared to, say HSBC).

(I didn't need to interact with any other bank during COVID, so I can't compare, but I was particularly pleased with how well DBS let me do even complex banking remotely, and I hope they adopt it as a permanent option).

Things do seem to be getting _slightly_ worse though. One issue I have with the bank is that they use push notifications to advertise products and also use push notifications for security purposes. It makes me wish apple/google would require push notifications to be categorized. Also, over the last year, push notification spam seems to be linked directly to my purchases, e.g. "Got a new gadget, consider DBS payment plans..." shortly after buying new phones. It's like they've confused digitization with social network.


> But what sets the bank apart is their customer service, the people on the other side of the phone (or counter, those few times) are incredibly friendly and helpful (again, compared to, say HSBC).

They've started to follow a Singapore trend of limiting non-fraud phone customer support to between 8am and 12 midnight [1], which IMO is ridiculous and counts strongly against their "customer service".

I also thought that their counter staff tend to be overwhelmed and overworked, because DBS, as they bought over Singapore's post office bank [2], also acquired a lot of (now) elderly customers — I believe still a magnitude larger than other banks.

[1] https://www.dbs.com.sg/personal/bank-with-ease/contact-us

[2] https://en.wikipedia.org/wiki/POSB_Bank#Acquisition_by_DBS_B...


Yup. I remember trying to open an account and getting the “we’ll respond in 7 days”, then 5 days later getting “missing document, please provide and we’ll respond in 7 days”.

Going to a branch in person took an hour of waiting. Once I got in front of someone it was pretty efficient but hold crap that was way more painful than US banking which isn’t exactly known for its customer service.


When is this ? Are you a resident? Opening an Bank account in Singapore is just one or two click for resident. Just need to click the opening account then grant access to applicant information in Singapore's government Apps. And it will be approved in 5 minutes.

w00t!!!??? 7 days to open an account? It takes 5 minutes for most banks in Europe, fully online, low or no fees.

As an addendum to my previous reply on the DBS's abysmal UX, I will echo the point made here that once you get to talk to someone, it's generally fine. I've had some great service when I could get through to a human, but they're making it increasingly difficult to do so as they're going all digital. Many branches have closed, and even the higher-end Treasures locations are now down to just two I think (a major mistake IMO especially as you target the wealthier). Navigating their phone support. takes. a. lot. of. patience.

I also agree that other banks can be a lot worse, and HSBC in particularly has been abysmal.

(edited for grammar)


> I also agree that other banks can be a lot worse, and HSBC in particularly has been abysmal.

At least HSBC got something right with their app UX: they give me an option to log in using my PIN instead of automatically trying Face ID multiple times, in this era when masks are still mandatory.


I share the same general experience with DBS.

> One issue I have with the bank is that they use push notifications to advertise products and also use push notifications for security purposes.

This was my main issue with DBS as well until about a month ago. I then contacted their customer service and complained that I was receiving notifications/emails, despite having opted out from marketing material in my DBS internet banking profile. I didn't expect they would stop, but I figured if enough people complain maybe someone higher up will review this one day.

Within an hour I got a response that they will "opt you out from receiving non-marketing/service messages from all channels".


Push notifications are categorized on Android. At least on 11. If the app also offers multiple categories, you can disable them selectively. A few apps use ‘General’ for everything, but not seeing this often.

For context: DBS is Singapore's biggest bank, and this outage affects most (all?) electronic banking services made through the app or website, including the popular PayNow transfers (usually small transactions made by scanning a QR code, or sending it to the recipient by phone number).

I am also not really surprised as my experience of DBS IT as a customer is mediocre at best. Trying to apply for a credit card online was a comedy of poor UX. Errors preventing the submission, but not marked, files to upload could have a max of 20 characters in the file name, trying to re-upload failed until you deleted the failed previous upload (not explained), and best of all, their "We value your feedback" webform does not accept "special characters" such as apostrophe. "Your website doesn't work" cannot be submitted because of "doesn't." Really. This was a few months ago.

Oh, did I mention the DBS app takes forever to load? There's other stuff I am trying to forget.

As they're the biggest bank here they can offer some attractive packages, which is why I have put up with it...


Their customer service UX is also hilariously bad. You can send a message to the bank from the app, which still makes sense, but to read the response, you:

- need to wait 3-5 business days

- eventually get an email notifying you that there is a response and directing you to the chat to read it (!?)

- to access chat, you need to complete a separate authentication process that requires SMS (can't use the digital token used everywhere else)

- and then you finally get a chat bubble containing the precious response.

- The kicker? If you want to respond, you need to do all of this again and get routed to another random agent with no access to your previous messages, since there is no concept of threading at all!


I had to do that to unsubscribe from their marketing email lists. They initially asked me to provide my bank account number to do it, too.

Eventually a person intervened and took me off their mailing list manually.

IDK why Singapore didnt make having an unsubscribe link mandatory yet.


It does: http://www.imda.gov.sg/for-community/Infocomm-regulation-and...

Unfortunately the Act is aimed at out and out spam, not marketing from companies where you actually are a customer.


Ought to have applied to me then since I was an ex-customer.

Sounds like most customer interactions in Singapore that don't perfectly fit into the standard workflow to me.

The DBS UX is absolute garbage. Their android app is extremely slow, and is full of (probably?) iOS patterns. It's almost like they decided to implement their own cross-platform framework, but poorly. (I apologize in advance if they are in fact using someone else's cross-platform framework, but poorly.)

Oh, and their web internet banking doesn't allow pasting passwords, because that's how we do security in 2021.

I've written software for them in the past. I... wouldn't say they have high standards. At least when I was working on systems like this (many years ago), a worrying amount of code was written for them by overworked contractors like myself.


This sounds very similar to the TSB meltdown playbook.

1) Have upper level management who fundamentally do not understand technology.

2) Outsource systems development to cheapest possible bodyshop and give them strict deadlines.

3) Explosion of technical debt that is entirely invisible to middle management and above.

4) IT meltdown as a result. CEO (hopefully) resigns in disgrace.


No discussion related to internet banking is complete without someone mentioning Commonwealth Bank here in Australia.

Commonwealth Bank CIO says any code older than three months is like “food in the fridge”, past its ‘best before’ date.

https://www.itnews.com.au/news/cba-to-treat-its-software-lik...

Which explains why their online banking services have become very brittle the past couple years.


I don't know. As a Singapore resident with way too many bank accounts, it seems like the most unstable experiences for me tend to be the banks that try to reinvent features, like the aforementioned DBS.

I'd speculate that shipping new features, not stability, is somewhat linked to career progression.

It could also be because most Singapore banking apps end up using the same mobile security framework to pass their audits [1] — except that same security framework prefers security over user experience.

[1] https://en.wikipedia.org/wiki/V-Key


The interview is a bit ambiguous, but to me it looks like he's referring to prod binaries over 3 months old being stale, which is considerably more reasonable.

At a high level, CBA’s technology strategy is “quite simple to think about but quite difficult to execute”

That’s the problem right there, it sounds like it is too simplistic to support execution.


Three months?! I'm guessing he's never written any.

I’d hazard a guess you’re probably right.

Still worrying that someone who’s prepared to spout that opinion made it that far up the organisation.


I see many people dissing it and it might be valid for their location.

But DBS is in India as well. Compared to local banks, among which most are shitty in a way that it feels they present “remaining shitty” as a non-negotiable goal to their tech team, and some foreign banks, some of those are shitty as well, DBS India’s app and Internet banking is really polished and functional. They bought a local bank recently so they have got a major physical presence push.

I used to have bad experiences with their iOS app but since last 7-8 months it’s been really been good.

I had complained around a year ago. 7-8 months back, an individual reached out on email thanking me for my feedback and saying they’ve done a major overhaul and they’d be glad to hear my feedback on it. This shit doesn’t happen with banks here. Be it private or govt banks, you’re lucky if you’re not treated like shit.

They don’t spam (you’d not at all understand what it actually means if you don’t live in India!!!). One short email to my RM and even those rare marketing emails dropped to zero (one of them I had actually benefited from; another banking rarity here).

PS: In last ~decade and a half I’ve tried too many banks and I keep ~10 bank accounts active at a time (due to insurance from Govt limit when banks fail; I know, I know)


Glad to hear your positive experiences. Could I share your feedback with a few colleagues in DBS here?

As you say, it's about context. Here we keep pushing the message that Singapore is an ideal place to do business, that the regulations are business-friendly, and the financial systems are efficient, well run and reliable, and this is largely true. A failure of this nature and size may be a blip in the Japanese or European markets, but it's also about perception. That takes a hit. It will affect the investment appetite, and our economy depends a lot on international business and money moving through our financial institutions. That's the first impact.

More practically, and looking locally, about three-quarters of our GDP is in the services sector [1]. Pretty much everyone is banked, and payments are increasingly cashless. A lot of these payments (including salaries, payment for services like air-con repairs, etc.) go through our local equivalent of your UPI called PayNow, and DBS' PayNow systems were failing. I remember carrying cash in my wallet just a handful of times over the past couple of years, and I pay for pretty much everything electronically. So... these systems which failed are far more central to a well functioning economy here than in some other countries. That's the second impact.

MAS (our regulator) is rightly concerned about the long-term implications. You can be sure they will have a thorough root-cause analysis. Some heads may roll. DBS will definitely be slapped with a multi-million dollar fine and be told to present a comprehensive plan to ensure this doesn't happen again. MAS will appoint people to oversee & manage how this is done, and knowing MAS they may start other audits to identify potential problems in other systems. They'll definitely be watching DBS closely, and they will probably look at other banks too.

References:

[1] https://www.singstat.gov.sg/modules/infographics/economy


Small update from their country head:

https://www.youtube.com/watch?v=EuXtrtH0uKY

1 - problem identified with their access control services (or did he say servers?). (doesn't sound good)

2 - working with their 3rd party engineering providers (how much is outsourced?)


Story time! Years ago, I was a contractor writing (a small piece of) software for DBS. It was probably a couple of hundred lines of Java, and I probably delivered it by emailing a zip file to someone. I don't recall their code review standards being particularly ... existent.

Mine was a relatively small project with them.

There were other teams at my firm working on larger projects at DBS. Another story! A new employee joined my firm, and - on his first day - was sent to join one of these teams onsite at DBS. His day ended at 9pm /on his first day of work/. He gave notice on day two.


My experience as an outsource partner with them 9 years ago. DBS only employed Project Manager and outsourced all their technical stuff. Yes the working hours is very long.

It's well worth your time to have a second bank account with a small overdraft, just in case this sort of thing happens. It also helps in case something happens to the bank card you're given.

Even on an institutional level, I've connected my hedge funds to secondary prime brokers just because there might be some outage, either IT or business level. Sometimes there's a PB team at some bank that's keen to get business this way.


The DBS - worlds best digital bank - is an absolute joke. Their local app is terrible in Singapore and you can't even download transaction history more than 6 months!

Original headline says, "Singapore bank DBS says services disrupted for second day", so I'm not sure where the "with no meaningful update from DBD" in the HN headline came from, nor am I sure of the meaning of DBD.

More details, but not much more news yet:

https://www.channelnewsasia.com/singapore/dbs-posb-down-not-...

From that article: >DBS Singapore country head Shee Tse Koon said the bank identified a problem with its access control servers on Tuesday and has been working "round the clock" with third-party engineering providers to fix the issue.


A little more detail from the bank's own web site. Apparently they got the access control servers up again, but then they went down a few hours later, at least partially. Which, after 24 hours of downtime, probably means that most of the people who got the systems back up were trying to get some sleep.

[1] https://www.dbs.com/newsroom/DBS_Singapore_Country_head_prov...


I’m not super familiar with the full content of their regulations but in my experience working with banks that maintain a presence there, Singapore control requirements tend to be fairly specific, opinionated and far reaching. For example, in developing OS security standards, I found myself having to actually incorporate requirements specifically for the Monetary Authority of Singapore that bled into operations in North America.

These requirements extend into resiliency as well. I’m curious how well DBS has been fairing in audits.


MAS requirements are incredibly specific. However, I wonder if that opinionatedness could sometimes make things worse. We had to do a lot of work to convince MAS that AWS was safe enough for our tiny operation out there, for example. Whereas, if we'd gone for a local colo, it would have been much easier - despite having worse resilience and likely worse security.

In my case it 100% made things worse because there is was no room for contextual interpretation of the rules. The MAS requirements went too far in one direction and prevented us from developing more granular and observable control strategies. We essentially wound up stuck with security paradigms from the 90's. I was able to compartmentalize most of it to in-country workloads but some of it ultimately leaked back into our global builds because it was going to be too costly to maintain a total snowflake.

A few months from now the headline will be that MAS (Monetary Authority of Singapore) slaps DBS with a hefty fine.


The regulator MAS is already starting to make a noise about "appropriate actions" [1] once the dust has settled.

It's also a matter of national pride, DBS is very iconic and there's been a lot of publicity about the bank as a regional success story, specially as they were publicising their "successful" transition to become a tech company going head to head with global giants, while also offering banking services [2].

I hope they don't forget their roots and end up treating banking as an afterthought and failures of this magnitude as par for the course. A sizeable portion of the population still goes into the bank to transact physically.

There's the historical angle and another point of national pride [3]. Since the post office savings bank (now part of DBS) was established by the British in 1877 it catered to low income groups, and provided a very high quality of service while keeping operating costs low. They were also early regional adopters of technology, starting to computerise their accounts in 1972. For a long time POSB was run as a statutory board with directors that were directly appointed by a government ministry.

All this is to say that people have a slightly different relationship with POSB/DBS compared to other banks. There's a lot of trust that's been carefully cultivated over decades. People believe and trust in this bank, they trust that the bank has their best interests at heart, and this has been generally true. Failures of this magnitude will start to break this trust. That in itself is not a big concern, there are other banks and we can jump ship. There's a social angle with POSB/DBS, specially with the older generations. When POSB was handed to DBS [4] on a silver platter in 1998 a lot of people voiced concerns that the bank would become yet another money-grubbing profit-seeking megacorp, and about how this would affect their earlier objective of serving the weaker strata of society. It's hard to predict what may happen if this trust starts to break.

References:

[1] https://www.channelnewsasia.com/singapore/dbs-bank-service-d...

[2] https://innovationinbanking.efma.com/content/articles/transf...

[3] https://eresources.nlb.gov.sg/infopedia/articles/SIP_2014-01...

[4] https://www.dbs.com/about-us/who-we-are/our-heritage


Ransomware?



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: