
The NHS's 1.2M employees are trapped in a 'reply-all' email hell - agjmills
http://uk.businessinsider.com/reply-all-email-chain-1-2-million-nhs-employees-2016-11?r=US&IR=T
======
smcl
The advice at the end is pretty important - if you want the thread to die,
just ignore it. All told people mostly followed this, and if only 120 replies
were sent in an organisation as huge as the NHS then that's pretty incredible.
In our company's "social" list the silly reply-all chains have a very
predictable pattern, coming in a series of waves:

1\. original sensible email like "hey guys, I have some
honey/walnuts/fruit/spirits from my hometown if anyone wants to try/buy"

2\. sensible replies start rolling in, but which were accidentally sent to all

3\. some jokers/trolls send memes in response (also reply all)

4\. the whining begins, people reply-all asking everyone to stop spamming them

5\. the memers reply (sometimes with more memes) that it is the optional
"social" list that's for this sort of nonserious/fun stuff.

6\. some helpful problem-solvers weigh in, sending reply-all instructions
(often including MS Paint'd diagrams) on how to unsubscribe from the list or
apply an outlook filter

7\. finally the "can everyone just stop replying to this though?" emails
start, also reply-all (and apparently oblivious to the contradiction/irony of
their own reply-all) and everyone participating starts to realise it should be
ignored...

Once you recognise the pattern it becomes pretty enjoyable identifying which
stage of this month's "Emailgate" you're currently at - the whole thing can
around to 45-60 mins to play out.

~~~
branchless
One of my favourite days at work in a large bank was when someone sent a
boring email to the wrong list. It was a big list. A few people replied all
saying "can you remove me from this list please". Then more people replied
saying "can you please stop replying all on this thread". Then people said
"can you remove me from this list please" again replying all.

The mail servers slowed to a crawl. We had thousands of emails. It began at
9am GMT. Eventually after about 3 or so hours everything calmed down and we
could send mail again, albeit with a large delay.

Around 12:30 the US workers began coming in, openend outlook and saw all the
emails. And instead of scrolling down they sent an email, reply all "could you
remove me from this list". Again people replied "stop replying all" on reply
all. We were up and running again! Mail was down for the rest of the day.

The next day some people who were off the day before came in and there was
again a brief storm, but sadly this one only lasted around 30 minutes.

One of the biggest banks. Lasted a whole day. We christened it "moron storm".

~~~
brazzledazzle
Since these are so trivial to halt and blow away everything in the queue from
the thread I can only imagine the sysadmins in charge of these mail systems
are letting it continue for comedic effect. That or they have to jump through
weeks of change control hoops so letting it play out is faster.

~~~
pja
Banks have legal constraints on doing that (the need to record all employee
communications) which might have made life difficult in this particular case.
I agree in general that the mail sysadmins ideally ought to be able to nip
this kind of thing in the bud.

~~~
brazzledazzle
Ah, yes–it's a bank. That's a great point. Not sure what I was thinking. I
imagine you could set up a separate transport with special rules that dumps
them somewhere outside of the normal mail infrastructure for that message
thread but it wouldn't be worth it when the problem happens infrequently and
just adding sender restrictions to the list is quick and effective.

------
DonHopkins
Back in the days of ARPANET mailing lists, there used to be an "educational"
mailing list called "please-remove-me", that was for people who asked an
entire mailing list to remove them, instead of removing themselves, or sending
email to the administrative "-request" address.

So when somebody asked an entire mailing list to remove them, somebody else
would add them to the "please-remove-me" mailing list, and they would start
getting hundreds of "please remove me" requests from other people, so they
could discuss the topic of being removed from mailing lists with people with
similar interests, without bothering people on mailing lists whose topics
weren't about being removed from mailing lists.

It worked so well that it was a victim of its own success: Eventually the
"please-remove-me" mailing list was so popular that it got too big and had to
be shut down...

...Then there was Jordan Hubbard's infamous "rwall incident" in 1987:

[http://everything2.com/title/Jordan+K.+Hubbard](http://everything2.com/title/Jordan+K.+Hubbard)

------
jameshart
For context, the NHS is one of the largest employers in the _world_. The
list[1] currently goes:

\- US Department of Defense

\- The Chinese Army

\- Walmart

\- McDonalds

\- The NHS

And The NHS's staff are all far more likely to actually have to use a computer
and email in their day-to-day work than most of those Walmart and McDonalds
employees.

[1]
[https://en.wikipedia.org/wiki/List_of_largest_employers](https://en.wikipedia.org/wiki/List_of_largest_employers)

~~~
bradleyjg
McDonalds doesn't really belong on the list as they include employees of
franchises. Or if does than there are some other organizations that belong on
the list as well -- such as the Roman Catholic Church.

~~~
digler999
why does the accounting designation of a "franchise" imply the employee
shouldn't be counted ? They wear a uniform that says McDonalds, they show up
40+ hrs/week at a building that has "McDonalds" written on it. The product
they sell has McDonalds written everywhere on it. Advertisements say "go to
McDonalds", not "Go to your nearest McDonalds Franchise". What makes them not
an employee of McDonalds ?

~~~
Someone
If the franchise owner doesn't pay its employees, they cannot go to McDonalds
headquarters to get their money.

If somebody sues a McDonalds shop and gets awarded a large sum of money, they
cannot go and get it from McDonalds headquarters.

If McDonalds were to go bankrupt, emplyees would still have a job (possibly
not for very long, as your employer may decide to close shop, but he might
also consider continuing under his own name or to start working for another
franchise organization)

(Having said that, McDonalds exerts quite some control over what their
franchisers can and cannot do. It would not surprise me if judges, in
some/many, would rule McDonalds to be responsible for what happens at its
franchises, for example in cases involving hygiene)

~~~
digler999
so basically its done to gut workers' rights. The person behind the counter
isn't an "employee" but a "third-party contractor". How clever. But yet if you
protest "inequality" a whole segment of the population will write you off as
being a leftist hippie as if nothing's wrong.

If there is ever a violent revolution in our lifetime, I think future
generations will read about this stuff in history books as foretelling omens.
And I'm talking about the bigger picture here, not just MCD:

* You're not an employee, you're a _contractor_.

* You have "rights", just not the same rights as the _shareholders_ you work for.

* Don't hate _us_ , you were harmed by a _third party_ , take it up with them.

* Vote for tweedle-dee next time instead of tweedle-dum, tweedle-dee is "for the people" (meanwhile corporation paid equal amounts to both candidates).

I see the bigger picture as eroding the personhood and agency of the poor,
through the most elaborate and creative accounting and legislation imaginable.

~~~
cstejerean
No, it'a not done to gut workers rights. The workers are still employees, of
the franchise owner. It is done because expanding through franchises is a lot
quicker and less risky than expanding company owners stores, because someone
else puts up the capital and assumes the risk of failure.

~~~
bigiain
There's also the point of view that the franchisees are in the burger flipping
business (and employ burger flippers), while Mc Donalds the parent corp are
actually in the real estate business. (It's not _entirely_ true, but it's a
quite plausible explanation of how their business works...)

------
nneonneo
From the BBCs coverage
([http://www.bbc.co.uk/news/amp/37979456](http://www.bbc.co.uk/news/amp/37979456)),
it seems like the problem was caused by a contractor setting up a full-org
distribution list by mistake. This DL would have appeared as a single
(unfamiliar) email address to most recipients, making it less obvious that
"reply-all" would actually send a message to a million receipients.

The NHS resolved the issue by killing the offending DL reasonably quickly,
which would stop further replies from being generated. This is also the reason
why only about a hundred folks (out of a million) replied-all to the message.

~~~
nthcolumn
Not the contractors fault at all! People make mistakes. If your design depends
on user vigilance then its YOUR fault. People make mistakes! A single 'rogue'
should not be able to take down the entire show just by sending an email. The
problem was exacerbated mainly the software being used made by Microsoft - a
fairly ubiquitous but non-standard, proprietary system. It has no place in a
public entity like the NHS. Sack the highly paid idiots who procured it at
great expense over cheaper and higher quality alternatives instead.

------
russnewcomer
This reminds me of the story Larry Osterman told about Bedlam DL3 [0] at
Microsoft. I feel like this is a problem with few good, problem-free technical
solutions.

[0]
[https://blogs.technet.microsoft.com/exchange/2004/04/08/me-t...](https://blogs.technet.microsoft.com/exchange/2004/04/08/me-
too/)

~~~
johansch
"Do you really want to send this e-mail to 1,000,000 people?

[Yes, really] [No]"

With the latter option being focused by default and emphasized?

Yes, doing that is tricky deployment-wise with open protocols, but isn't the
whole point of solutions like Exchange to enable system-wide shortcuts like
that?

With an open solution the next best option is probably to have the mail server
send a confirmation request email back to the user with a link to click if
they really wanted to send it, instructions on how to reply with a
confirmation etc.

~~~
ChuckMcM
At Google they had modified gmail at one point to take an email that was going
to go to a lot of people and send it to your manager for approval. That worked
pretty well. I believe the data indicated that sending it to _anyone_ else for
them to sign off on it helped. You could override a disapproval and send it
anyway but that essentially created a willful act that later could be used as
part of the reasoning behind your dismissal :-)

~~~
wutbrodo
Do you remember circa when this was? I was there for a few years and I
remember a couple of amusing incidents of people sending "I'll be late today"
to thousands of 10k+ people. Presumably that wasn't manager-approved.

~~~
ChuckMcM
this was late 2000's (2008? 2009?) and you could circumvent it by typing "my
manager has approved this email" or something.

~~~
wutbrodo
Ah okay, that was a year or two before my time

------
hitgeek
I was caught in an email storm like this one time. Someone at a large
insurance company accidentally sent an email containing a few patient
names/dobs to a group-email mailing list of at least 1000+ outside emails.

It took about 2 months before the storm finally ended.

It became comical how many responders to the email chastised the original
sender for "HIPAA violations" using Reply-All thereby inadvertently committing
the same infraction themselves because the original email was included at the
bottom of the reply.

Also, several times, after 2-3 days of quiet, someone would come back from
vacation and reply-all with "I think you sent this to the wrong person",
thereby starting the storm all over again.

The company of the original sender seemed powerless to stop it, but I always
wondered if they could have just disabled/deleted that group-email mailing
list.

~~~
SOLAR_FIELDS
Easy to do if there are no permissions or anything else tied to that AD group
- but that is often not the case in these kinds of situations.

~~~
brazzledazzle
Easy enough to setup an authorized senders list. Really any group with a
significant number of recipients should have that or failing that some way for
a user to remove themselves.

------
dingaling
120 replies ( as of lunchtime ) out of 1.2 million recipients is a remarkably
low ratio for a non-technology-focused organisation.

Looks like there are lot of sensible folk in the NHS, or else they haven't yet
all had a chance to check their e-mails between shifts...

~~~
pavel_lishin
I wonder how many emails per day the system usually handles; I would say I
probably receive on the order of 100 emails a day (although most of those are
automated messages) and probably send half a dozen, and I'm definitely on the
low end.

------
mkmk
Reminds me of this very funny story shared on Metafilter...

(if the comment permalink doesn't work, please search for "Reply-All can get
even more fun when")

[http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-
FROM-T...](http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-THIS-
LIST#2408665)

------
whack
It's amusing how many people continue to suffer from a solved problem. Any
email system that resembles Gmail in its most basic form, won't have any such
problems. 100 people replying-all to the same email? Oh hey, they are all
lumped together into a single thread which you can easily ignore. Don't want
to see any more emails from that thread? Oh hey, you can just hit the mute
button on that thread, and you will never see any more emails there.

It boggles my mind how many people and organizations continue to use outdated
crappy systems when vastly superior alternatives have been publicly available
for over a decade.

~~~
lmm
We're talking about a European organization that handles people's most
sensitive personal data - shipping that to the US to be scanned by Google and
the NSA is not an option.

~~~
eru
OP didn't talk about using gmail. Only about copying gmail's features.

------
yagyu
I just moved to the outlook webmail in a large-ish organization. It surprised
me that Reply All is the default reply-button, and a regular reply requires
expanding a drop down! The ui really encourages the mistake.

~~~
JoshTriplett
For mails _not_ sent to millions of people, Reply All is the correct default;
anyone receiving the original mail should normally see the reply, and
narrowing the communication to a subset of people should require thought (and
often an explanation).

To help avoid this situation, when sending emails to a large number of people
(or a list that goes to a large number of people), if you _know_ that
responses should not go to that same set of people, use BCC.

~~~
Too
There is no correct default, show both buttons next to each other with the
same size! Now you have to open a tiny drop down to get to normal reply, it's
very easy to forget that there is another option there just replying to all
without thinking.

------
cs02rm0
My wife works for the NHS so I had a quick look at her inbox (as it were).

Thoroughly disappointing. Normally when I see this happen it's at relatively
IT-literate organisations. The NHS emails all seem to be single replies to the
original email asking to be removed / saying it seems to be sent in error.

I say disappointing because I didn't see any threads spiralling off into back-
and-forth between people asking others to stop replying or anyone trolling
others.

Such a waste.

------
jplahn
Ah. This reminds me of a great story in Amazon of a conference room booking
email gone wrong. The email was inadvertently sent to the entire company, so
that employees in Japan, China, Scotland, France, etc. all knew of this fabled
booking. At its peak over 280k emails were being delivered throughout the
company.

Also good to remember: replying UNSUBSCRIBE to an email list is typically not
how you go about it.

~~~
wutbrodo
I remember this happening at Google, but I'm not sure if it was the entire
company or just something like eng-announce. I myself once had the wrong CL #
on my clipboard and attached a random CL of mine to a bug that Jeff Dean had
opened about a development tool used by every engineer (with very active
discussion across the engineer on the bug). I don't remember these incidents
being treated with anything but mild amusement, certainly not "email hell".
This probably has something to do with the fact that GMail makes it pretty
easy to block an email thread.

------
todd8
I once went to a crowed midnight movie showing of the 1950's "It Came From
Outerspace". The crowd seemed quite intoxicated and so I missed the first 40
minutes of the movie's dialog because every few seconds someone would stand up
and yell SHUT UP and then someone else would yell back from the other side of
the theater NO YOU SHUT UP.

It was terrible and amusing at the same time.

------
stymaar
Pro tip to escape this kind of situation : reply to the last email but spoof a
fake email address as sender. Now, for everybody who's gonna use the last
email to reply (your email then) your email address doesn't appear anywhere.

You might need to repeat this process a few times since people don't always
reply to your email, but it works pretty well.

~~~
bcraven
>You might need to repeat this process a few times

You're suggesting adding millions of extra emails to the system here. I think
the best thing to do is wait, or create a rule based on the subject line.

~~~
stymaar
In this very specific context, your are absolutely right since the main
problem is not the little annoyance experimented by the email receivers, but
the global overloading of the mailing infrastructure.

But this is a really exceptionnel situation, while being trapped in a reply-
to-all loop is not that uncommon.

------
econnors
I think the iOS "unsubscribe" banner could be adding to the chaos. Not sure
about this exact situation, but I've been caught in mailing lists where people
instinctually hit the unsubscribe prompt which just sends an email with text
saying "unsubscribe", therefor blasting everyone in the thread and not
actually unsubscribing them.

~~~
JoshTriplett
That should only happen for lists that have a List-Unsubscribe header. If the
iOS mail client does that for arbitrary mails, that seems horribly broken.

------
iamatworknow
When I was in college e-mail was everything. Facebook was only just getting
off the ground, so when an event was happening or a club was recruiting the
best way to get the word out was to send an e-mail to the listserv, which then
sent it to specific class years or to the whole campus.

Problem was this listserv wasn't moderated at all. Any club president or
anyone else who wanted access could get it, and the messages weren't screened
for content before being distributed to every e-mail address on the list.

One such club president wrote an e-mail to the ex of his current girl that
began, "Me and Jessica fucked like rabbits..." and he accidentally included
the listserv address for the entire campus on that e-mail. The guy was
generally well known as an active student with great grades who didn't get in
trouble, so when a catty lovers quarrel was made public to thousands of
students and faculty by his own hand it was kinda entertaining.

------
elcct
I was caught in 'reply-all' email hell, but not at such scale. It was with
about 5000 users. It was hilarious actually, but I can imagine a lot of people
will find this annoying.

------
mfoy_
Something like this seems to happen, and make the news, every couple years...

Is it not possible to configure their email server to disallow emails with
over X recipients from sending?

~~~
gregmac
It's probably a list, so the actual email has only one recipient. Something
essentially expands this list to deliver a separate message to each individual
recipient.

~~~
pavel_lishin
I'm surprised that lists of certain sizes aren't locked down.

I would expect the server to refuse to send to "all@nhs.gov.uk" unless the
sender is on some sort of pre-approved list.

~~~
mobiuscog
One would _really_ hope that such an address cannot be externally addressed ;)

~~~
louthy
The domain is nhs.uk ;)

~~~
desus
The domain is .nhs.net ;)

Some individual trusts have @trust.nhs.uk addresses, most - probably two
thirds use the .nhs.net system.

------
sjcsjc
[https://en.m.wikipedia.org/wiki/Email_storm](https://en.m.wikipedia.org/wiki/Email_storm)

------
nkrisc
I was once part of a ~20k person reply-all chain. After a while I sent an
email to each person who sent yet another reply-all asking to be removed an
individual email explaining how dumb they were, in the nicest, most
professionally appropriate way possible. None of them replied to me.

------
jessewmc
Tangent: I found 1.4M to 1.7M employees for NHS on google. 64.1M people in the
UK, 31.42M employed.

1.4M / 31.42M = 4.5% of the entire UK workforce works for NHS? Is this a
normal distribution for a developed country?

~~~
sockmeistr
Approximately 9% of US workers are in healthcare.
([http://kff.org/other/state-indicator/health-care-
employment-...](http://kff.org/other/state-indicator/health-care-employment-
as-total/))

The UK does have a private healthcare sector, although smaller than the NHS.

------
sandworm101
>> 120 replies so far — meaning that more than 140 million needless emails
have been sent across the NHS's network today. As a result, they said, its
email systems are running "very slow today."

Is this 1994? That isn't my definition of email hell. 100 useless emails in my
inbox is a bad day, but it certainly isn't a hellish one. And 140 million
emails shouldn't slow anything down. That;s a drop in the spam bucket for any
organization the size of the NHS.

------
eggie5
This happened at Qualcomm once. I think it was an IT email too. It went out to
most of the corp and what really started the issue was people's vacation/out-
of-office auto-reply-all messages -- those really started magnifying problem.
Then people started making it even worse by reply-all sending "remove me from
this list!" etc. The most funny messages were the reply-all that said "Stop
sending reply all to the whole list!" It lasted most of the day...

~~~
corecoder
By the way, does anybody know of a valid reason why vacation/out-of-office
messages shouldn't be forbidden forever? Has any organization ever had any
positive outcome whatsoever from their members having those autoreplies?

------
teddyh
_Ornk!_

[http://www.threepanelsoul.com/comic/on-sea-
lions](http://www.threepanelsoul.com/comic/on-sea-lions)

------
blauditore
I received such a mail from Microsoft a few years back. It was something
related to WP/W8 development with thousands or tens of thousands (IIRC) of
email addresses in the "to" field, so probably all registered app devs. I
tried to send a witty response, but my mail server (ironically on outlook.com)
refused to do it.

~~~
jessaustin
Publicly revealing all those addresses in the "To:" could be considered really
abusive. That's a screw-up above and beyond the spamming.

------
namesbc
How about responding to the list with everyone on BCC stating "You have been
unsubscribed."

------
darkr
This is why you should access controls on configure large mail groups. I
thought this was pretty standard at large organisations. A select few people
can actually send to a given group, but anyone else will just receive a
bounceback.

------
j45
It's a little odd that this has received media attention.

What I'd do: independently of the email, ask for a procedure to be sent out to
create an email filter/rule. Wouldn't take more than 1-2 minutes to complete
for each end user.

------
enraged_camel
Email clients need a way to remove the "reply-all" button. That way, users are
forced to manually enter the emails of people other than the sender, which
would make them think about who actually needs to be copied.

------
Spooky23
It's amazing to me that Microsoft never fixed this issue with Exchange.

There are a few edge cases like this where users with no ill intent can do
lots of damage. Strong procedural controls are really the only defense.

------
giarc
I work for a large organization and this happens once or twice a year. I find
that when speaking with people, no one knows what BCC is for. Many of these
would be avoided if people used BCC properly.

------
rickette
I assume all mails run over NHS controlled mailservers. Some administrator
could just step up and kill/block the email thread, right? Shouldn't be that
hard of a problem...

~~~
desus
That's exactly what they did. "The distribution list has been removed and
associated emails are being traced and cleared." See
[http://support.nhs.net/servicestatus](http://support.nhs.net/servicestatus)

------
xyzzy4
This used to happen all the time when I worked at Cisco... and yes most emails
were of the variety "please remove me from this list".

------
cmgreen
I'm surprised they don't have a cap on number of recipients. That is often a
problem orgs had to deal with around holiday time.

------
golemotron
This might be the most clever Denial of Service Attack ever devised.

------
jlebrech
surely the exchange server can block reply alls that have too many recipients,
or maybe mute people who've already chipped in for 24hrs.

------
tintor
... or (if using gmail) just mute the conversation.

------
Thriptic
gg no re?

------
arethuza
You can tell they aren't very close to it a they refer to "the" NHS - there
are actually four NHSs for different parts of the UK:

[https://en.wikipedia.org/wiki/National_Health_Service](https://en.wikipedia.org/wiki/National_Health_Service)

~~~
bcraven
From your own link:

The terms "National Health Service" or "NHS" are also used to refer to the
four systems collectively.

~~~
arethuza
What I meant was which of the four NHSs (which have a lot of different
infrastructure) does this apply to? Or to all four - which in itself would be
interesting.

~~~
desas
NHS England and Scotland. NHS Wales use something else I think, I'd guess NHS
NI do too.

Not all of NHS England use .nhs.net, it's up to each trust if they want to
keep running their own email or migrate to .nhs.net which is centrally
managed.

