From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO
---
ABC: Can you tell us why the system's gone down?
Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.
ABC: Do they know what happened though? Who's there the team that you've got working on it?
Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.
---
Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )
I worked for a large Telco (not Optus) and was constantly shocked by how little the management knew about anything.
They hired a new Director of "Core Network", and in her first all-hands meeting she loudly asked what IPV6 was. (serious, she had never heard of it).
The Director of IT had me setup her home wifi because she had no idea what DHCP was.
The VP of "IT/Internet" literally had other people make her Powerpoints because she couldn't. She had never done anything more on a computer than send emails.
It was shocking, but also really eye-opening to realize how important business acumen and playing the political game were.
Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know. Especially in these larger types of companies.
> they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know.
Well, technical competence and expertise does not necessarily translate to managerial competence and expertise. It's two distinct skill sets. Getting machines to work is very different than getting people to work.
> Well, technical competence and expertise does not necessarily translate to managerial competence and expertise.
This can't be overstated. The best engineer could be the worst manager.
That said, managers should have at least some level of technical knowledge on what they're managing. I'm a security engineer that once worked under a CISO who had very little technical knowledge and it showed. Having to explain things to him multiple times got frustrating. We're talking on the level of "Hey, our nmap scans are showing we have port 443 open on our web app. We should have all ports closed." and then trying to explain why that would make our web app completely inaccessible.
> Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know. Especially in these larger types of companies.
100% spot on, I couldn't agree more. That was the biggest thing I learned during my time there. It was painful at times, but a very important lesson to learn.
> Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain.
Being a technical person myself, and working in a small company most of my life I was blissfully unaware and then shocked. I guess it's true that closeted existence did leave me blind. But I don't think that is a consequence of being technical or an engineer - it was just because the smaller companies I preferred to work for would go broke if they made dumb decisions or failed to learn. The shock came from the small company I worked for being taken over by a huge, word wide company. And then watching that huge company roll over all their internet links to Optus.
All technical people get to know what it's like to deal with the big two telco's in Australia. They all eventually figure out the smaller ones provide much better service and a sharper price, and then avoid the big ones like the plague. So we knew what would happen when we went to Optus, and it panned out pretty much as expected. That's perhaps an exaggeration, as them leaking most of their customer credentials onto the internet wasn't "expected", but then level of engineering incompetence that lead to the leak wasn't a surprise either.
What was a shock was seeing first hand how Optus won the business. All the rules that engineers live by, like basing decisions on empirical data, "trust but verify" and so on weren't part of it. Instead it was smooth talking salesman, mandatorily weekly meetings with implementation teams who contribution seemed to be to presenting spreadsheets proclaiming how much progress that had made, paid trips to holiday destinations for upper managers and cases of wine being sent to lower managers. Later came the lies, and when we called out the lies and dug up older versions of spreadsheets to prove things weren't going a planned white anting. The white anting worked - we were threatened with dismissal.
Those tactics only work for so long of course, and eventually the exec's that threatened us with dismissal ordered us to get rid of Optus, at any cost. Fawning turned into literal hatred. They could away such displays of corporate disloyalty only because by that time all the managers who championed the charge to Optus had gone.
But despite that experience, they moved to Telstra. Telstra is the other large telco in Australia, and is just as bad. I had left by then, but I assume the decision was made by a new cohort of managers who will also be gone in a few years.
At this point I'm no longer blind, just flummoxed.
Not a surprise the CEO does not understand why their network is down. Many CEOs of companies of that size think business is moving money around, doing acquisitions etc. They have no clue how their production works or where it could be vulnerable. The information they get has been filtered through several levels of management, the cynic would say through several levels of of increasing incompetence in production matters.
Optus was down in most of Sydney for over 10 hours. Some country/outlying areas were back online a little before then.
In response to hours of silence from Optus executives (in that they weren't able to string two words together about the technical issues or the source of the fault) one caller to ABC Local Radio in Sydney asked whether the fault was in Australia or in Singapore which was an excellent question as Optus is owned by Singtel (Singapore Telecommunications Limited).
The logic behind the call was to ask about how much of the Optus infrastructure actually exists in Singapore. If, say, much of the technical control is done from there then it may have been hard to quickly diagnose the problem with the system infrastructure split across two countries.
This is important because when Optus was sold to Singtel there was much concern about the security of its telecommunications if another country had access to user data, etc. At the time we were told the Australian Government was happy with the security arrangements that were put in place but many were not completely satisfied with the assurance.
Given the huge data breach that Optus experienced some months ago and that that matter has not been resolved by any stretch there will have to be a fully-fledged inquiry into the failure and hopefully it will get to the bottom of how Singtel actually runs Optus.
That said, some background is needed for overseas readers.
Australia has a long, long history of screwing up telecommunications, radio and TV broadcasting not to mention spectrum management. Too involed to describe here in detail but briefly the stuff-ups and political shenanigans would fill volumes and they go right back to the Wireless Telegraphy Act of 1905 and the Postmaster General's Act. The Govt. PMG's Dept. had exclusive monopoly over the phone network for nearly a century (like Ma Bell but govt. owned). It later became Telecom Australia but still govt. owned. Still later the then Government sold it off and it's now a public corp under the name Telstra.
To maximize income from the Telstra sell-off the Government sold it off in two stages, and to make the sale attractive to investors it sold the telco complete—lock, stock and barrel without due consideration of the public interest. That is, instead of just selling off the exchanges and phone network infrastructure the Government also sold off all cableways and rights-of-way across the nation as a bundled deal. Also, it never put sufficiently tight regulations or conditions in place to protect customers and they've since been screwed rotten by all telcos—horrible deals, little or no service—for instance, witness today's Optus outage!
The sale of the cableways came back to bite the Government later when it needed to establish the NBN—National Broadband Network—as it had to buy back access to Telstra cableways which cost many billions—it's a huge scandal the Government somehow swept under the carpet. Also, the competition couldn't use the Telstra cableways, Optus had to completely install its new network onto power poles.
This first-class fuckup has now been 'amortized' into a surcharge that everyone across Australia has to pay for internet and phone. The extra billions have had to come from somewhere and it's the poor hapless consumer that has been paying the bill.
(In the past I've suggested some enterprising person do a PhD on the subject and in the process determine the extra per capita costs that every citizen is now paying over and above what he/she ought to have been paying had there been prudent management and less destructive politics.)
Incidentally, the NBN itself was further fucked up by politics, half-measures and technical compromises, it's story in itself. Just this week there's been an announcement that NBN network prices are to be increased.
Into this almighty shemozzle comes Telstra's competition, Vodafone and Optus. These telcos have also benefited from slack regulations, very poor consumer law and seemingly nonexistent technical oversight by Government.
Right, when it comes to telecommutations, Australia is the laughing stock of the world and its citizens have been taken for a very expensive ride.
Addendum. Well, as predicted, the Government announced it's holding an inquiry into Optus. That's less than 24 hours after the event and has to be a speed record for Govt. Such speed gives a good indication of how big this kerfuffle was and how bad the public's perception of Optus and Govt. is for allowing the failure to happen. The questions everyone is asking is why wasn't there a backup plan and why did the emergency phone numbers also fail as they are supposed to be protected (isolated) when such outages happen.
Many are saying Govt. is largely at fault and asleep at the wheel for not keeping stricter tabs on these telcos.
Given there are three major networks in Australia and a few minor ones and that only Optus went down the level of disruption was alarming. Essentially, much of the country was in turmoil for better part of a day.
If a comparatively small outage can cause such disruption to a country then just imagine the chaos that'll follow when the next Carrington Event hits us.
“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.”
“Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.”
I haven't been able to find information on what part of their overall network is impacted. Per [1] they use a mixture of leased fibre and owned fibre for connectivity between Australian cities, and per [2] (note: zoom in) many point-to-point microwave links connecting mobile phone tower sites to the fibre footprint. It's potentially as bad as Optus needing to send a technician on a flight to a remote airfield somewhere in rural Australia, then driving 5 hours to a microwave tower to fix a router within the communications hut. Multiply by thousands of routers which are no doubt very much spread out. If Optus are lucky, it's just impacting routers for part of their overall network. For example, only impacting routers within data centres of the major Australian cities where you'd expect Optus to have technicians nearby who could physically access the router with 1-2 hours notice.
Sure, but how do they reach those technicians? They can't call them.... Hopefully they provisioned sat phone backups for enough of them! Or they're HAM radio enthusiasts.
Actually a lot of newer network equipment ships with the USB to Serial adapter built in, so they have a USB Micro or USB-C connector on the front you can use.
"One Optus customer, Annie, told ABC Radio she found out about the issues through her cat, who is fed through an automatic Wi-Fi feeder, and missed being fed due to the outage."
Internet of Shit strikes again. It’s just insane to me that anyone makes an embedded system that isn’t capable of tracking time and it’s own schedule.
I’m sure some useless agile product owner scrum type person came up with the brilliant idea of needing an internet connection just to turn the motor on and off.
Because having any competing company which fails sometimes it's still better than having just Telstra with free reign. Monopolies are not fun and will lose your data too.
Apparently about 10 million customers. or roughly 40% of population. More concerningly, the cause has not been identified yet, after 7 hours. Was identified as down at 3am? ( Maybe some maintenance / upgrade gone wrong? My company used to do such critical upgrades / changes at 2am on Tuesdays ... )
just went to the store to buy a Vodafone prepaid sim card. There were about 60 people in line.
Point of sales are out in a lot of coffee shops. Utter chaos!
If you work in an emergency room or similar this could be a major nightmare.
If you are a software developer it's best to take a day off or read a book (good for those who have bought a physical one). No reason to stress, isn't your project delayed by more than a day anyway?
Many phones support a second line with an SIM/eSIM setup. As most people still use a SIM in Australia, I wonder if having a second line with an eSIM on a very basic pre-paid plan would work for those people.
In theory, you could have line 1 forward to line 2 when 'unreached'. Of course you'd want the second line to be different network.
That's true, even Telstra was slow getting to it (was only available for prepaid services for a long while). I think Aussie telcos need a bit of kick to really go in on eSIM, such as Apple releasing eSIM only iPhone's in the USA. You can't even get a add-on SIM for an Apple/Android Smartwatch from anyone except the big three here still.
It would be good to have regulation surrounding eSIM, because telcos will always find a way to make it awful. E.g. able to do 24x7, max 1hr processing time, no cost to user, able to do via self-service options.
> All of Melbourne's train services stopped temporarily due to a communications outage, and Metro Trains said it was unable to rely on its back-up system, which used the Optus mobile network.
Their primary and their backup were both Optus. Not much of a backup.
I wonder what they ended up doing to resolve it. There was a disruption but it was for less than an hour. Someone wifi hotspot the train network from their phone or something.
This person is being sarcastic - they have Israel on the mind. This is coming from a Jew who does not support Israel by the way (before anyone thinks I'm on their side).
ABC: Can you tell us why the system's gone down?
Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.
ABC: Do they know what happened though? Who's there the team that you've got working on it?
Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.
---
Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )