
Slack Security Incident - malgorithms
https://keybase.io/blog/slack-incident
======
the_duke
Let's ignore the rather awkward self promotion, and the fact that 2FA would
have prevented this specific incident.

This is the important part, which everyone should think about:

> What would have been way worse — immeasurably worse — is if our team had
> used Slack for anything other than what we did use it for, which was
> discussing outages of our own product. Had my cofounder and I discussed our
> company's cap table, or business partnerships, or compensation agreements,
> or ongoing legal matters over Slack; or had our team traded API keys, or
> security-sensitive matters; or had we controlled mission-critical
> infrastructure via Slack-powered "bots"; we'd be sweating bullets to this
> day that our important company secrets were out in the open, about to
> resurface at the worst possible time.

I see Slack being used for everything at a lot of companies.

There are a lot of interesting things in the chat history everywhere: SSH/API
keys, logins and passwords, sensitive internal documents, chatops bots that
would let one take control of the infrastructure, and worst of all - juicy
office affair gossip.

Combine this with often lax user management (forgetting to disable old
accounts, inviting people to channels they shouldn't be in, ...).

Most companies overlook this and don't even have a policy for what's OK to
post on Slack, and what isn't. Not even to dream of any kind of enforcement.

I'ts a big security problem, even without Slack getting compromised, and
should be on the radar for CTOs.

~~~
harryh
Slack has some options to delete all messages over N days old. Unless there is
a really good reason not to, turning on this feature generally sounds like a
good idea. At least you can drastically limit the length of the archive
available to any attacker.

~~~
Ntrails
That's a _terrible_ feature in an instant messenger client for work use? I
cannot recall the precise details of my conversations with smart people who
know things I do not. I can recall that I had the conversation and remind
myself what I learned before.

~~~
manigandham
Depends entirely on the conversation obviously, but perhaps important details
should eventually be coalesced and moved to a more permanent document or
project management system before the deletion deadline. It's a good discipline
to have.

------
numair
The author would have done well by refraining from using this as an
opportunity to make a sales pitch for their startup, as it detracts from an
otherwise important message.

Let me see if I have this right:

Slack had a major security breach in 2015. Apparently someone installed
malicious code that could even read password inputs in plaintext. They waited
4 years, after growing large and going public, to inform affected users. And
in the interim they blamed their users for any related security problems.

Do I have this correct? If so, how is anyone going to defend this situation?
And how can anyone put any sensitive data on Slack, or tell their company to
do so, and feel good about it now?

I expected some stupid apology note from the CEO on their website if this
turns into a bigger issue, which is sort of an anti-pattern at this point...

~~~
save_ferris
Would shareholders have a case to make for fraud here? Slack clearly didn't
want this information getting out pre-IPO, as a security disclosure in this
case would certainly impact public confidence in the company.

~~~
jdavis703
Every tech IPO filing has a generic statement saying something like "our
software may contain bugs, including bugs that we cannot fix blah blah blah."
So unless Slack has made fraudulent statements about this specific breach, I
doubt they've done anything illegal WRT securities fraud.

~~~
gmmeyer
I'm sure shareholders will sue, because they try to turn everything into a
violation of securities law. But securities law shouldn't be the only way to
regulate companies imo

------
cj
What makes this even more sad is the extreme difficulty you'll have if you
attempt to remove your company data off of the Slack platform. (Disclaimer: I
stopped using Slack 2 years ago)

Our company used Slack extensively for multiple years. A couple years ago, we
decided to stop using Slack for official company communication. After
switching to alternative communication tools, we tried to delete the data in
our Slack account and found it to be nearly impossible.

I recall using 3rd party python scripts (opensource on Github) that took hours
to run - the script used an API key to fetch and delete messages individually.

We also tried using Slack's Admin panel to delete messages. At the time, I
believe it required clicking a checkbox next to every chat message we wanted
to delete. Clearly not a realistic way to scrub an account.

The sad reality is that with many services like Slack, once a provider has
your data, there's often no easy way to remove it. IMO, this is a major
downfall of our reliance on modern SaaS services (companies have no incentive
to prioritize features for deleting account data - the only users who would
find those features useful are already churned customers).

~~~
gtirloni
I don't understand this. They say that if you delete your workspace, all
messages are gone.

[https://get.slack.help/hc/en-
us/articles/204067366-Delete-a-...](https://get.slack.help/hc/en-
us/articles/204067366-Delete-a-workspace)

~~~
throwanem
It says they're "irretrievable", which isn't necessarily the same thing. That
_you_ can't get that data back doesn't mean it's no longer stored by Slack in
a way that a sufficiently severe compromise of their infrastructure might
reveal.

~~~
gtirloni
I doubt they keep customer data after you're not a customer anymore. Sure,
they could be lying but we don't have information pointing to that, do we?

They are subject (or have subjected themselves) to various security standards:
[https://slack.com/intl/en-ie/security](https://slack.com/intl/en-ie/security)

And their security whitepaper [0] mentions:

 _Customer data is removed immediately upon deletion by the end user or upon
expiration of message retention as configured by the customer administrator_

It would be a stretch to say a message is deleted permanently but a
_workspace_ is kept forever, for unknown reasons. Occam's razor and stuff.

0 - [https://a.slack-
edge.com/78b2/marketing/downloads/security/S...](https://a.slack-
edge.com/78b2/marketing/downloads/security/Security_White_Paper_2019.pdf)

------
privateSFacct
Wow - for a sales pitch fantastic. Many of these security issues leave you
little to actually do. This write up provides an alternative.

What’s super bad here is slack misleading about the cause wasting all the
users time.

Quick question, anyone use key base - can u give a quick review? Team
currently use slack

~~~
Alex3917
I use Keybase. I like it, the thing to keep in mind though is 99.9% of the
time the biggest threat vector to your company is going to be preventing your
employees from sexually harassing each other, not preventing the Russians or
whoever from reading your internal messages.

~~~
braythwayt
_99.9% of the time the biggest threat vector to your company is going to be
preventing your employees from sexually harassing each other, not preventing
the Russians or whoever from reading your internal messages._

Yes, but:

1\. We always have to combine the likelihood and incidence rate of threats
with the magnitude of damage. I do not personally subscribe to this thinking,
but some bloodless financial types will suggest that you can compensate
harassment victims, but a major security incident could cost you the entire
company.

2\. Insert animated gif of a little girl asking, "Why not both?" Companies can
and _must_ address threats like sexual harassment AND exfiltration of
sensitive data.

------
bluk
The post says "If the attackers inject server code, 2FA or U2F or any Web-
based security practice does little." Understandably if the attacker was
actively scraping passwords with 2FAs and was using the credentials
immediately, that would be an issue.

However, for the issue mentioned in the post, wouldn't 2FA have saved him?
Supposedly, the compromised credential is from 2015.

~~~
tialaramex
U2F saves you, most of the other options potentially make it worse.

If the bad guys are dumb and just were there to passively steal passwords then
almost anything works. But if the bad guys are paying attention and stealing
authentication secrets then it breaks down like this:

Passwords: Stolen if used while bad guys could watch

TOTP/MOTP/ similar software one time codes: Seed gets stolen by bad guys, they
can generate the same codes as you now.

SecurID/ similar key fob type one time codes: Seed gets stolen by bad guys,
they can generate the same codes as you now.

SMS: Now bad guys know your phone number. In a targeted attack they will
"jack" the number and cause even more havoc, but you might be safe against
script kiddies

U2F (or WebAuthn): The server doesn't know any secrets. It knows a cookie
(meaningless gibberish except to one FIDO device) and a public key (public).
Stealing these is futile and cannot be used to impersonate the user.

TOTP and SecurID surprise people because they forget that although the
credential transmitted is just a one-time code, the _server_ (which bad guys
broke into here) needs to know the same seed as the user does to get the same
codes.

------
Alex3917
Not only does Keybase not automatically update its client, there is no way to
even figure out if your client is out of date and in need of security updates.
Even if you look up the exact version of your installed client, which you can
find, there is nothing on the website that says what the most recent version
is. The only way to even get a hint is to look on GitHub, and even that isn't
accurate; version 4.2.1 is the latest release, but when I download the mac app
it's still version 4.2.0.

For whatever problems Slack has, at least I know if there is a new version
that I need to install.

~~~
ashelmire
You know what's better than installing slack? Not installing it and using it
in the browser.

If there's a website option for any tool, I recommend using that over native.
It's usually more performant and is less of a security risk. And it's always
up to date.

~~~
FabHK
> It's usually more performant and is less of a security risk.

Source, particularly for the "less of a security risk"? (It might well be now,
I'm not an expert, but a few years ago I'd have thought "no way").

~~~
ashelmire
Which do you think is more potentially damaging?

1) Using a website which has had its server code compromised (slack).

2) Installing and using an application which has had its code compromised
(maybe also slack).

The installed application is going to have more access and potential to damage
your system and to compromise your data. There's not really anything more to
it. One's in a browser sandbox and limited by browser capability, the other
can do literally anything it wants.

~~~
FabHK
Makes sense, thanks. I think my mental threat model was different:

you're talking about dodgy (potentially compromised) software: better run in
the browser sandbox (albeit imperfect) than natively.

I was thinking of sensitive software (eg secret chats) I want to protect from
attack: better run natively than in the (imperfect) browser sandbox.

In the context of this discussion (Slack), your threat model probably makes
more sense.

------
valar_m
>By contrast, Keybase currently runs all of its mission critical chat
applications over Keybase itself.

Reminds me of the time Amazon learned that using S3 to host status indicators
for S3 is a bad idea if S3 goes down.

------
zuck9
> Were our Dutch friends sifting through our messages for four years before
> Slack notified us of a suspicious login?

The author might know this already but the hackers aren't Dutch. They simply
bought a cheap NL server from LeaseWeb to mask their real IP.

~~~
nacs
> bought a cheap NL server

.. or exploited/hacked a random cheap server to use to proxy their requests
and have access to high speed 100Mbit/1Gbit downloads/uploads of Slack data.

------
lurker458
slack in 2015: "We have no indication that the hackers were able to decrypt
stored passwords, as Slack uses a one-way encryption technique called
hashing."

slack in 2019: "In 2015, (...) The attackers also inserted code that allowed
them to capture plaintext passwords as they were entered by users at the time.
"

This reflects poorly on them. Unless they discovered only now that the 2015
hack included capturing plaintext passwords.

------
dlgeek
Scary, and certainly doesn't reflect well on Slack. But, do keep in mind that
the author runs a company that does compete with Slack in some ways.

~~~
ekc
I don't think that's relevant.

Poor security practices are poor security practices despite conflicts of
interests, and Slack's are certainly _extremely_ poor.

~~~
ejcx
From the blog posts slack has released we know nothing about their security
practices.

They have a lot of high quality security features and you can see they
actually work because they alerted Max that his account was compromised.

Saying their security practices are extremely poor based on an incident they
had in 2015 when their company was 1/20th the size it is today is ridiculous

------
emdowling
Encryption for a business chat app limits the potential users rather
significantly, as I understand it. A number of sectors (like banking) have
strict rules which require keeping a record of company communications. How
does Keybase deal with this, or do they choose not to play in that market?

~~~
icebraining
Well, the bank could still record from their end of the conversation,
capturing the chat logs from the employee's terminal and shipping them
somewhere.

------
lallysingh
That's a nice sales pitch! A little on the nose, but I think that's ok.

I think one issue keybase still had is it's minimal web presence that sounds
to focus too little on what keybase can do for regular users. People need more
explaining of the day to day benefits.

------
speeder
I wonder if this affected companies like IBM... I know for a fact IBM uses
slack to talk internally about client billing, meaning if IBM was compromised
the attacker knows what most of IBM clients bought from IBM.

Same applies to a bunch of other companies...

------
paulcole
> Unlike Slack, it is free. And without ads.

How does Keybase make money?

~~~
anchpop
It's supported by the foundation that maintains Stellar, a popular
cryptocurrency

~~~
paulcole
Thank goodness. I was worried it was something unsustainable.

------
est31
> Keybase messages are end-to-end encrypted, and only our users control their
> decryption keys.

End-to-end encryption isn't just good for the users because the service can't
access the messages and sell them, it's also good for the users as it provides
good protection if the service gets hacked: message content can't leak unless
the attackers can change client code.

------
asdkhadsj
As people are discussing Keybase for teams and whatnot - could anyone comment
on Keybase for individuals, families, etc?

My family are debating moving to Matrix (and away from iMessage). I had
briefly debates Keybase due to some interesting features. Anyone have
experience with Keybase for families and individuals?

~~~
madspindel
Why move away from iMessage? It's end-to-end encrypted.

~~~
lloeki
Either one of:

\- iOS only

or

\- an extremely-crypto/privacy-aware family who is concerned that although
end-to-end encrypted, there is no out-of-band user-validated key exchange (e.g
through a digit code or QR-Code) or prior validation (e.g per-key TOFU) of the
other party's identity or update to their keys. Which means that theoretically
Apple could insert a new key in the user's set (there is a key per user's
device for perfect backward/forward secrecy†) anytime to MITM/intercept
messages and you would not be able to monitor it††.

† This is obviously defeated by Messages in iCloud (which you can disable) but
the iMessage protocol has this built in.

†† Of course if there was some feature like that, Apple could also make such a
key hidden in some way because they also control the Messages app code.

------
rvz
The second I read this, I immediately signed up to Keybase today and deleted
my Slack account and switched to the Keybase chat system instead which I am
setting up right now.

I am confident to say that this incident was the final nail in the coffin for
me to abandon Slack. Choosing Keybase was a no brainer.

------
Finster
From the slack post:

> In other words, if you’re one of the approximately 99% who joined Slack
> after March 2015 or changed your password since then, this announcement does
> not apply to you.

Hackers compromising plaintext password would seem to apply to everyone using
Slack, whether their account was compromised or not??

------
lorenzsell
I’m a little confused here. Isn’t Keybase a slack replacement product? Why is
the CEO of Keybase using Slack for any company communication instead of his
own service? Am I missing something obvious here?

~~~
oconnor663
It's described in the article:

> what we did use it for, which was discussing outages of our own product

You can't use Keybase to communicate about why Keybase is down :)

------
mnutt
Slack gates a lot of its exfiltration security features behind enterprise
licensing. Expect to double your spend if you want to try to prevent sensitive
data floating around in slack.

------
Boulth
Is there any indication why did this post drop from the front page?

------
workerthread
Does Keybase support chat message search yet? Otherwise claiming to support
all important features of Slack is stretching the truth. I use search
extensively at work.

~~~
nemoniac
Yes, it does

------
gpjanik
Yeah, this is basically marketing material.

------
Nelkins
Does Keybase have any kind of revenue stream? I didn't see any mention of
pricing on their website.

------
tshanmu
this needs to go up on HN!

------
RyanShook
So why was the Keybase CEO using Slack in the first place?

~~~
ithkuil
> We basically just use a #breaking channel in there in case we have Keybase
> downtime.

The problem wasn't they were using Foo and Foo was compromised.

The problem was that he couldn't be 100% sure whether Foo was compromised or
not (when asked, they claimed they weren't).

Now, if Foo is a "random-cheezburger-meme-generator" startup you just try out
with a throw-away password, then you can probably assume that it's likely they
have been hacked and they are lying to you.

That leaves one alternative scenario: _you_ have been hacked.

You'd expect that Slack (in 2019), while it's still possible they lie to you,
has decent security practices and thus makes it less likely to be hacked and
thus increasing the odds that you might indeed be hacked.

In a way, it acted as a honeypot. But one that you don't control so you can't
be sure whether it caught a hacker or whether the bees are drunk.

