Hacker News new | past | comments | ask | show | jobs | submit login
Tell HN: Canada Posts systems to be down for 6 hours of “unplanned maintenance” (canadapost.ca)
87 points by neom on Dec 29, 2021 | hide | past | favorite | 75 comments



I just got off the phone with Canada Post because my package had not shown up on the Canada Post tracking system when the package was handed off from USPS to them yesterday. Guy on the phone said they've been having "issues" and he can't do any service tickets or status lookups because their internal systems are all offline for the next 6 hours due to "unplanned maintenance"


6 hours?

They know 80's mainframe could do updates without any downtime right? Did someone mess up the Kubernetes config and shut it all down at the same time?

The Canadian government is an example of how not to do software. Their Phoenix pay system is frequently used as a cautionnary tale for bad design and implementation [0].

[0] https://en.wikipedia.org/wiki/Phoenix_pay_system


> Instead of saving $70 million a year as planned, the report said that the cost to taxpayers to fix Phoenix's problems could reach a total of $2.2 billion by 2023.

Ouch.

And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death:

> Pascale Boulay, a Quebec coroner, determined that the 2017 death by suicide of a 52-year old woman from Val-des-Monts, Quebec, was preventable. The coroner assigned blame on the "flawed Phoenix pay system" that had "led her to emotional and financial ruin."


Phoenix was rotten from concept to post roll-out fuckery. It was broken before a single developer touched a line a code.

Red flags independent of software development finesse or skill:

- Centralized payroll operations to effectively the middle of no where (effectively ensuring that no existing payroll specialists would staff up the new center). Locate operations in the middle of no where to compensate for another government program and center being shut-down.

- Sole source contract. To IBM no less. To customize and configure PeopleSoft. There's literally nothing that sends good vibes in that setup.

- The very concept of running a big bang switch over of like 75% of the federal government to a new payroll system.

- Launched without end to end, full system test

- Launched KNOWING that critical functionality (like retroactive pay) was not implemented, and that critical requirements were failing tests.

- Deciding to roll out despite their new centralized payroll center was already less efficient at handling payroll on legacy systems and that their central assumptions were already being challenged

- Deciding to roll out despite receiving reports from client departments that the initial data transfer into the new system was already significantly incorrect

- Quote: "In our opinion, they had received more than enough information and warning that Phoenix was not ready to be implemented, and therefore, they should not have proceeded as planned. Phoenix executives prioritized meeting schedule and cost over other critical elements, such as functionality and security, resulting in an incomprehensible failure of project management and oversight"

You can read the audit here: https://www.oag-bvg.gc.ca/internet/english/parl_oag_201805_0...

Software is only tangentially related in that the "magic" of software lets people believe that such projects are even possible.


And none of the responsible GoC employees (I hesitate to call them "managers") was fired as a result of their mistakes.


Makes sense. Nobody Ever Got Fired for Buying IBM™


> And for those who think software engineering other than medical devices, aviation, or nuclear power plants isn't life-or-death

I don't think a single real engineer was involved in this, sadly. Probably offshored contractors and clueless non-technical government employees.

It's ironic because Quebec punch way above it's weight for engineering.


It wasn't necessarily just IBM's fault either, it was complete incompetence by the government who continuously pushed the change through in the face of ample evidence (and IBM's own suggestion) that it should have been delayed.

Not only did they push it through despite issues even after initial limited rollout, they made a ton of terrible decisions like to do training in house instead of from IBM as cost savings, made the new Public Service Pay Centre elsewhere for political reasons with new staff and they laid off most of the people that worked on the old system and decommissioned it so there was no going back.


> It wasn't necessarily just IBM's fault either

My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense.

That's why they couldn't and didn't sue, and if you read about it in the media, there's no direct blame on IBM (a lot of it is implied; being to direct could be construed as defamation).


> My understanding is that it wasn't IBM's fault at all. They delivered exactly what was aggreed upon. What the government asked made no sense.

If what the government asked made no sense, why did IBM agree to do it? Rhetorical question. Money talks.


> If what the government asked made no sense, why did IBM agree to do it?

My understanding is that the government gave IBM a complete spec of what they believed were their pay grades. But that wasn't what was agreed upon with the employees and unions (how do you calculate overtime on a coast guard boat?). So it all made sense to IBM, they shipped it and fufilled every requirements. They had no way of knowing these weren't the "correct" pay rules.

It's the magic of government contracting: you write the spec and select the lowest bidder.

Forget a 0 to the total weight the bridge must carry? You'll get very cheap proposals. Ohh and there's typically rules against a bidder contacting the government to get more information; that could be considered as interfering with the bidding process. It's also why older USPS vehicules had poor snow performance.


Yeah that was what I thought too, but I don't have a full enough understanding to definitively say if IBM had any culpability or not.


Most software people aren't "real" engineers in any capacity, especially not in Canada.


That seems oddly political for a coroner? (I'm not Canadian, this thread's the first I've heard of Phoenix Pay, I don't care.) Are suicides due to stress/depression usually so specific as to the cause that it's unavoidable? I would've assumed it'd just be 'emotional and financial ruin'.


> I would've assumed it'd just be 'emotional and financial ruin'.

In this case I think all her income was from a single source (the federal government). So it doesn't sound political to me as they were able to root cause the single source of financial ruin.

It's like naming the drunk driver who slammed into a pedestrian while being completely alone on the road. You can always say "an anonymous driver" on the report ...


I didn't know they did name the manslaughterer or even that they were drunk - that's what I meant by 'are they usually so specific that [highlighting Phoenix as the cause of the ruin] is unavoidabable'.



Canada Post is a crown corporation, they don't share employees with the government, it's entirely separate.

And yes, Phoenix was awful, but separate point.


To err requires a computer. To really foul things up requires ansible.


Thanks IBM.


What are the odds this will turn out to be due to ransomware?


That or they just realized they're using log4j


Would be a pretty fun exploit if you printed it on the address field of a letter, OCR recognizes it, cannot parse it as a postal address, logs the failure using log4j, thus giving you access to their post routing system...


Sounds like a plan... until you realize why your label printer keeps restarting.


Yes....fun...


Their cookies start with "JSESSIONID", so I'm guessing you're right.


And subsequently got a ransomware because of it.


Last year, someone signed up for PO box-equivalent with my firstnamelastname@gmail.com address. (Not the first time this guy in Montréal forgot that his email has a 0 before the @gmail.com. Or an error from the person entering it in.) I live in the US. I have lived in Canada, but over 10 years ago now.

The only two options to contact Canada Post were phone or … Twitter — which I don't have an account for — no email or web form. And the phone CSR had to get a manager who still couldn't figure out how to deal with the issue since I wasn't a customer myself and just wanted my email removed from the account.


You can try cmg@canadapost.postescanada.ca


Good to know if there's a next time.

I forgot to say that another option was to contact them by mail, so at least they're dogfooding ;-)


The domain is canadapost-postescanada.ca


It's cute in Canada, they would feel the need to go so incredibly far as having the domain bilingual, rather than just registering two separate domains and pointing them to the same address. If you actually (for God knows what reason) struggle so hard to see the similarity between French and English in this two word identifier, then surely putting them both on the same domain is only going to cause issues?


Canadians take our bilingualism seriously. To a degree that calling it "cute" would be offensive to many. It's the law that French and English need to be displayed & provided "together" in government services, I presume this is to avoid any issues there. "The Official Languages Act is a Canadian law that came into force on September 9, 1969, which gives French and English equal status in the government of Canada" (although I believe in Quebec, French is by law to be displayed before or over English)


How do you do for contracts? Is there a prevailing version or both translations are prevailing ?

Here in Hong Kong English supercedes if presents and next door in Shenzhen it's the opposite.


English deals are done in English, French deals are done in French, if a deal is done in French or English the courts are adequately able to deal with either being bilingual. There are some exceptions around jury trials, I'm not sure it's a right in a jury trial, however I doubt very much it would be an issue. Everything standard form from the government comes printed in both French and English.


It's fine when you're solely reading something. Your eyes gaze to what you understand. Just like a product manual that comes in 20 languages.

Where this approach falls flat is web domains. It requires you to actually type out both the language you know; along with a language you (likely) don't. Quick Google only 1.2% know both.

And as pointed out above, then figure out if those variations are separated by a dot, underscore or hyphen. Notwithstanding the domains get ridiculously long.

It's good we try to please everyone, but sometimes the politeness goes too far. Registering the domains in both French and English in this case should have been more than sufficient.


It's not being polite in this instance, it's satisfying a law. We are not being "polite" to the French Canadians, the French Canadians consider themselves distinct.The Quebecers would not tolerate it any other way, and we'd go back to separatist conversations in Canada. This isn't being polite, it's politics and deep rooted societal issues (that should at some point be resolved imo, also imo, it's a total utter travesty we offer everything so strictly in French and English to preserve the culture, but screw offering it in Ojibwe for the natives, boils my blood, but that would be being polite.)



This somewhat makes sense as it mentions the page content itself was in French. However then "solving" the problem by making a domain with both languages hyphenated only magnifies the issue more - to indicate an expectation you should know both languages.

Any logical personal will understand showing content in multiple languages (as text output); but when it comes to user text input (domains, web search etc), it's absurd to believe the same rules should apply for the user.


Sounds normal.


Canada suffers from losing all their competent tech workers to jobs that actually pay them market rate :^)


It's actually even worse for government (or crown corps like Canada Post), since they pay way below Canadian market rate, which is already way below U.S. salaries. The government then isn't really in competition for the best talent who want the most money.


Like most government dev work, Canada Post outsources to Accenture for their projects.


This. Exactly everywhere, not government only. Why pay market rates to actual developers when you can get bottom of the barrel scum for $2000 a day


All about that capex vs opex!


Not only that, most gov projects have to outsource (that's why we have such big consulting firms like CGI and LGS). This make project costs skyrocket. All that because they can't offer good wages (or condition - working conditions are pretty much set in stone).


And in Ontario public service, raises are capped to 1% yearly including benefits.


You mean the market rate in another market/country that they can get into. Certainly not the Canadian market rate.

That's the price of free trade and open borders. Most countries lose their best and brightest to the US and other high-pay countries, and the world is poorer for it.


Canadian market rate for software engineer is pretty high if you live in vancouver (especially american companies that operate in canada).

Its not SF levels of pay, but its definitely not peanuts.


Vancouver is nearly SF price level though. But with far worse pay.


> Most countries lose their best and brightest to the US and other high-pay countries, and the world is poorer for it

Why not simply compete?


At least in Canada it would be consider pretty uncouth for a civic worker to make what they would have to make to compete. Personally I don't agree with this, however, if my farther learned a software engineer at the post office was making $200k his head would explode, he already wants to defund the Canadian Broadcasting Corporation, also a crown corp. My understanding is that our national defence and spy agency pay quite well by Canadian standards, but that's about it for government jobs. Our crown corps also happened to be big fans of using students: https://www.lcbonext.com/


> At least in Canada it would be consider pretty uncouth for a civic worker to make what they would have to make to compete.

Yet it's a single payer healthcare system no? Meaning medical staff ultimately is on government payroll. Pretty sure they make more than $200K.

> he already wants to defund the Canadian Broadcasting Corporation, also a crown corp.

He's not wrong. Why does entertainment need public funding? Marvel and Disney made billions last year by simply making content that people want to watch. It's a hugely profitable business.

> Our crown corps also happened to be big fans of using students: https://www.lcbonext.com/

The link you shared is even more bizarre. Almost at the top of the page they brag about the companies where their former interns ended up. Most of them are American companies in SV. What does that tell us about the people working there full time? "Not good enough to come to America and innovate? Join full-time with the rest of the B-Players!"


> Why does entertainment need public funding?

You know that CBC isn't all about entertainment, but also hires a lot of journalists, right? And I don't see Disney or Marvel making content with the Canadian culture and heritage in mind.


> but also hires a lot of journalists, right?

That... weird. Having government approved journalists on government payroll sure sounds like something out of China, Russia or Cuba.

I mean let's face it, will they really critisize and investigate the hands that signs their check?

It also has the second order effect of making it harder for independant medias to compete, since they can't just get free government money.

> And I don't see Disney or Marvel making content with the Canadian culture and heritage in mind.

Is there a market for it? Right now this content is dumped by a single player that's subsidized. No reason to enter this space.


> That... weird. Having government approved journalists on government payroll sure sounds like something out of China, Russia or Cuba.

The government doesn't actively handle the hiring, CBC has journalistic independence from the government. They in fact handle all of their internal systems (email, data, phone) independently from the government.

> I mean let's face it, will they really critisize and investigate the hands that signs their check?

They do all the time in fact, and they report on themselves even if it shines a negative light.

Example: https://www.cbc.ca/news/canada/khan-cbc-canadaland-tweet-che...

> Is there a market for it? Right now this content is dumped by a single player that's subsidized. No reason to enter this space.

When the majority of the private content available is from the US, it's also hard to compete with the torrents of culture they push on us. At least CBC is actively pushing content that is culturally significant.


> The government doesn't actively handle the hiring, CBC has journalistic independence from the government. They in fact handle all of their internal systems (email, data, phone) independently from the government.

But, at the end of the year, who signs the check?

> it's also hard to compete with the torrents of culture they push on us

Not sure "pushing" is the correct term. I remember when Hulu and Netflix started to disrupt the market a lot of Canadians were trying to trick those companies into serving them, with fake addresses and VPNs.

Didn't radio stations used to play "Government mandated Canadian Content" at night because listeners were just tuning out if they did it during peak hours? [0]

[0] https://en.wikipedia.org/wiki/Canadian_content#Regulations


Gosh I've had this conversations more times than years I've lived I'd expect. I suspect I understand your position, never the less.. last I'd checked, we'd pretty fully embraced socialism for quite some time now. I'm fine with whatever the majority want if the direction was to change, however... Maxime Bernier didn't exactly do well.

My only nit pick on the CBC is that I believe the very REASON Canada has so much fantastic media is because the government funds independent artists, could someone like Matt Dusk have had a career in any other country? He certainly doesn't say so. Leonard Cohen, Drake, Shania Twain, all funded by the government, because of the Canadian Content mandate, the mandate dutifully fulfilled by our national broadcaster. Additionally, I very much enjoy shows like DNTO, Quirks & Quarks and especially: not having them chocked full of advertising.


I'm fine with a federally funded broadcaster. Media in Canada (news and entertainment) has really nose-dived since private industry all amalgamated in the 2000s. It's not perfect, but it's not as bad as it could be. (that's kind of the Great Canadian Motto)

Most newspapers are all copy/paste nationwide now, and the majority of TV production is handled by the people who own the medium into the house (cable/fibre/satellite), making "Canadian, eh!" versions of American culture and passing it off as CanCon.

The CBC are out there at least trying to keep it real. Most of them, at least.


> Leonard Cohen, Drake, Shania Twain, all funded by the government, because of the Canadian Content mandate, the mandate dutifully fulfilled by our national broadcaster

It's interesting to consider Justin Bieber's houses right here in California were purchased in part with Canadian government subsidies [0].

[0] https://www.velvetropes.com/backstage/justin-bieber-house


Doctors make more in the States. The CBC is not an entertainment organization and some things are worthwhile but cannot compete in the market.


> Meaning medical staff ultimately is on government payroll.

They aren't government employees though. They all bill the same insurance, but they still have to compete on service, efficiency, availability, and friendliness. So inevitably some healthcare providers will do better than others.

> Why does entertainment need public funding?

Because it's more than just "entertainment".

> Marvel and Disney made billions last year by simply making content that people want to watch

And now you have powerful forces who want to perma-extend copyright terms and lock up popular culture behind paywalls until everyone who cares about it is dead. And even longer than that.

> What does that tell us about the people working there full time?

In professional sports, it's common for some teams to advertise themselves as feeder teams for the big names. The pitch is "Join us and see where you could end up one day."


> They aren't government employees though. They all bill the same insurance

That sounds like using contractors to skirt employment laws for public servants.

> Because it's more than just "entertainment".

sitcoms that end up on netflix?

> In professional sports, it's common for some teams to advertise themselves as feeder teams for the big names. The pitch is "Join us and see where you could end up one day."

Again, what does that tells us about those who join those teams but can't transfer out? They weren't good enough for the real teams.

Could you picture Google bragging about their interns going to Apple or Tesla?


> That sounds like using contractors to skirt employment laws for public servants.

Not sure which employment laws are being skirted here but whatevs.

Defence companies typically also have the same buyers as healthcare providers - governments. Unlike healthcare providers, they're even more limited in who they can sell to. So whatever you're saying applies to them too.

> sitcoms that end up on netflix?

First off, cultural exports are a good thing to have. But also, the CBC does other valuable work. Other people in this thread have also explained this to you but it seems like your mind is made up, so I won't bother.

> They weren't good enough for the real teams.

Yeah maybe, so what? No shame in that; everyone's gotta work somewhere. You're also ruling out people who are there for personal reasons, or because they genuinely like the place, or they don't have any "higher" aims.

> Could you picture Google bragging about their interns going to Apple or Tesla?

Last time I checked, Apple or Tesla (or Google) weren't planning to sell liquor in Ontario. They're not competitors.


> Yeah maybe, so what? No shame in that; everyone's gotta work somewhere. You're also ruling out people who are there for personal reasons, or because they genuinely like the place, or they don't have any "higher" aims.

No shame indeed. It's just weird that they brag about their best interns leaving for better companies. Usually, the point of having internships is to attract and retain talent.

> Last time I checked, Apple or Tesla (or Google) weren't planning to sell liquor in Ontario. They're not competitors.

They are when it comes to tech hiring. By their own admission, they hire from the same pool.


CSIS and CSE pay like garbage. I could have helped my country but instead I opted to write enterprise software for 5 times the money


CSIS still seems low by most standards: https://www.canada.ca/en/security-intelligence-service/corpo...

You can get "level 6" salaries as a new grad at a canadian bank.


Yeah that's terrible, even for canadian software engineering standards.


My sister works at CSEC and the pay isn't that great. The working conditions are pretty good from what I can tell.


If you'd watched the CBC recently you'd want to defund it as well. Tons of blatant propaganda.


Why don't you just compete with Usain Bolt?

To suggest that Samoa, Bangladesh, Peru, or 150 other countries should "simply compete" shows a complete lack of knowledge about the state of the world, about economics, and about human behaviour.


> Why don't you just compete with Usain Bolt?

You've never been around athletes haven't you? There are a lot of very talented people who are working very hard to shatter his records!

> To suggest that Samoa, Bangladesh, Peru, or 150 other countries should "simply compete" shows a complete lack of knowledge about the state of the world, about economics, and about human behaviour.

Sounds like they should understand why contributors are leaving and take measures to simply make them stay. Every country will have different challenge, but it's not hopeless. Keep in mind Japan and South Korea were extremely poor a few decades ago; look at them now!

At the end of the day, I'm on the other side of the transaction (getting an influx of foreing educated professionals willing to come here and innovate) so it doesn't matter too much to me really.


> There are a lot of very talented people who are working very hard to shatter his records!

Yeah, and a handful can compete with him. 99.99999% of humanity wouldn't stand a chance.

> Sounds like they should understand why contributors are leaving and take measures to simply make them stay. Every country will have different challenge, but it's not hopeless. Keep in mind Japan and South Korea were extremely poor a few decades ago; look at them now!

You named two countries. There are 200 others.


Should be 'Tell HN', not 'Show HN'.


Unless it was posted by the person who took the systems down. /s




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: