I've done work at CityLink, and their IT shop is just totally clueless. This does not surprise me one bit.
This is the same company that was doing "mirroring" with Veritas Volume Manager across a metro fibre link, rather than using their TrueCopy licenses. For no reason. Two fibre cuts rendered their entire production database (tolling) environment inoperable and nearly irrecoverable, on two separate occasions. I worked about 36 hours straight getting everything back online and was given a pair of movie tickets as a thank-you gift.
I've worked with State Transit in NSW (buses & ferries) and they were awful. The most embarrassing part was the UNIX team and Windows team refusing to talk to each other even though they were only one cubicle partition away from each other so any communication had to go up through two levels of management and back down.
The scariest part was every single database of user information was transformed from another database of user information; there was no source of truth, just a great big never ending circle of import & convert that had reached equilibrium and was now stable, provided nothing changed.
This is unfortunately normal in most public and corporate bodies. They are afraid that if they talk then information will filter across and they'll lose their remit or hold over the solution at hand.
You got movie tickets! A long time ago (13 years) the production database was irrecoverable! I got backups working just a week before tolling finally started. Failure of that system would ultimately result in the tollway becoming a freeway (allowing for buffering of data at the gantries, etc. etc.)
Although I had some responsibility for the tunnel safety systems at the time, we never touched them knowing that any problems would result in the road being closed.
From what I remember, the gantries were (at the time, a few years ago) able to store about 2 weeks' worth of traffic before tolls would effectively become free.
"... growing whisper within the security industry in Melbourne that there weak security might have been breached from China. ..." ~ http://news.ycombinator.com/item?id=4605335
In IT security here in Melbourne, first I've heard of it. Also why would network security specialists need to fly over 30 hours from Europe? We also have some pretty capable teams on the ground here already.
Given the time frame of when it happened it's more likely a change that has been borked in some way with difficulties in rolling back.
No trouble and agreed - rumours flying around like that don't help the problem. If there was that level of security breech, outsiders would probably do more than lock everybody out anyway.
Finding further information about it: looks like a core switch died with the backup not kicking in. 12 hours to work around is a but much though.
I'm having a really hard time seeing how this is any different than a traffic light being out. If "me-go-you-go" fails, why don't they just have a traffic officer out there directing traffic manually?
CityLink spokeswoman Kate Murphy says the tunnels had to close as a precaution.
"It's not just overhead signs [that are not working], it's also our safety systems
as well," she said.
"For example, in the tunnels the deluge systems or the smoke extraction systems
which we may use in an emergency, we can't actually operate those as well,
which is why we've had to take the dramatic measure of actually closing the
tunnels."
I had to hunt the article to find that, since the first impression is just speed limit signs/traffic lights being out which is not major. (and plenty of posts here suggest other people thought the same thing)
The variable speed signs always have extra signs along the lines of "(80) if variable limit sign is disabled" as a fallback.
Not being able to handle leaks, fume buildup, fires and other emergency conditions is a completely different story.
The fact that their safety systems (Which anyone would expect to be separate from the traffic control systems) are also affected makes me wonder about the "computer glitch" angle.
Like 99% of news reported computere glitch's it is some human mess up that caused it, either by design or error or a homer moment were somebody pressed the wrong button. Could even be somebody overloading the electrical phases in the datacentre and tripped the UPS into action for it to barth at the imbalance (seen that one).
This is why they say computer glitch and count on the majority of non technical people to not question it.
> Like 99% of news reported computere glitch's it is some human mess up that caused it
If the system allowed the user to do something that impacted safetey in such a way, then the computer glitch is that the computer should stop that sort of thing happening. Human error happens a lot and you have to account for it in your designs.
Of course there are times when you have to trust the user because it is not practical to do those tests (in which case the human should have procedure to follow that mitigates the risk), and instances where the user wil override the checks (when the machines say "I really don't think you should do X because Y, click 'I understand the consequences of my propsosed action' to proceeed") - then it is genuinly human error is something goes wrong.
As much as I don't want to sound like a conspiracy person, that level of connection between those components is a bit too much to believe. Especially in Australia where they have insanely high standards for construction/electrical engineering.
It's more like 20km of traffic lights being out. These roads are our core east-west arterial roads and the landscape dictates that there aren't many other roads for the traffic to travel on. I can't say why there aren't traffic police out there, but these are once-public roads that have been privatised. Also, these roads are highways, not traffic intersections.
Point of trivia: before the current toll road company was given its mandate, our main east-west bridge (the Westgate) held the distinction of being one of the very few toll roads in the world that stopped being tolled when planned, when the cost was paid off. A few years of free travel, then this company was given the rights to the road just beyond the bridge...
It is the only uninterrupted motorway from the east to west sides of Melbourne, but not the only core arterial. There are other roads you can use to get from one side of town to the other. A lot of traffic using the tunnels is just commuter traffic from the east anyway.
The tunnels never were public roads, they didn't exist. You can still drive on pretty much the entire length of the West Gate Freeway, right up to the tunnel portals without paying tolls. There are some short sections of motorway though that were previously un-tolled but are now tolled, though there has been some pretty extensive upgrades.
That said I'm no way in favour of City Link, but the real core 'arterial' for urban commuting is our public transport system. :-)
"... I'm having a really hard time seeing how this is any different than a traffic light being out. ..."
There are reports in the local press of these kinds of problems: "CityLink says it closed the tunnels because the IT problem affected incident detection and safety systems that are activated in the case of an accident. The network problem also affected the back-up system." ~ http://www.heraldsun.com.au/news/breaking-news/melbourne-tun...
See, this is why we need to hurry up and deploy networked computerized automobiles. We won't need stoplights anymore, so this kind of stuff won't happen!
We will still need safety systems in tunnels though, and lack of those is why they shut the tunnels down. Ignore the misleading quotes about the variable limit signs not working, the real issue is the tunnel is not safe if there is a problem like a fire of fume buildup.
I think he was being sarcastic, hinting that network computerized auto-mobile's having a similar glitch would be truly devastating.
If mankind can't get a tunnels systems to work all the time, then what about our plans to introduce computer controlled cars? A glitch could wipe out millions of lives before being patched, and even then if they get patched that also introduces huge areas of vulnerability.
I'm sure a stealth program that loads onto a car and waits till a date to kill the occupants will be used by someone or a nation-state in the future.
Apparently the reason the tunnels were closed was that Transurban could not operate the emergency sprinkler system and smoke extraction fans.
Surely it would have been cheaper for Transurban to pay a few MFB fire trucks to be stationed outside the tunnels and close a lane to guarantee them access in case of an emergency. That way traffic could flow, tolls could be charged, and nobody would be as upset. I am sure Transurban make more than the fire brigade would charge them per hour.
Bring every police officer on-duty, making a fair percentage of them as a manual fallback to traffic direction. Have the officers play the part of the traffic lights.
This is the same company that was doing "mirroring" with Veritas Volume Manager across a metro fibre link, rather than using their TrueCopy licenses. For no reason. Two fibre cuts rendered their entire production database (tolling) environment inoperable and nearly irrecoverable, on two separate occasions. I worked about 36 hours straight getting everything back online and was given a pair of movie tickets as a thank-you gift.