
An update on our security incident - pseudolus
https://blog.twitter.com/en_us/topics/company/2020/an-update-on-our-security-incident.html
======
ben509
> We have zero tolerance for misuse of credentials or tools, actively monitor
> for misuse, regularly audit permissions, and take immediate action if anyone
> accesses account information without a valid business reason.

Okay, so who has been fired?

That's what "zero tolerance" means: no excuses, not even "someone tricked me."
And no punishment but the maximum.

Anything less would involve some degree of tolerance, and when you say "zero"
that means no tolerance whatsoever.

It's obviously stupid to manage any organization that way, of course. It's a
fatuous, dishonest phrase.

So stop talking about "zero tolerance" since all it means is "we make
hyperbolic claims that we have no intention of living up to."

~~~
ytdytvhxgydvhh
It’s like when motorcyclists say “safety first” about wearing a helmet and
other protective gear. If they really put safety first, they’d choose a safer
form of transportation. They mean “given that I’m going to engage in this
risky activity, I’m going to try to make this activity as safe as possible”.

In this case “zero tolerance” is short for something like, “except for
understandable slip-ups that aren’t fully your fault, we’re not going to
tolerate any slip-ups”.

~~~
NotSammyHagar
I think you can honestly say you follow safety first in terms of what is
available to safely ride your bike.

Just like when I used to rock climb, I felt like we were basically following
safe practices - but there was no one to adjudicate them, probably far less
testing of various practices with stats than bikes. Also, where we climbed
there wasn't expected rockfall. I had barely heard of that being an issue, and
we never wore helmets. Later on I realized that was something I might have
missed out on. And then the next step was "what other safety practices was I
unaware of" ;-)

And of course I get that rock climbing is much more dangerous than hiking.

------
toyg
_> the attackers targeted 130 Twitter accounts, ultimately Tweeting from 45,
accessing the DM inbox of 36, and downloading the Twitter Data of 7_

So much effort for so little gain... With proper preparation (i.e. a simple
app ready to download everything from an account), they could have made out
with the whole data of 130 accounts, silently, _before_ tweeting the hopeless
scam message. Instead, this seems like a mostly-manual effort, done in haste.

Just dumb thieves pulling off the scam of their life, or cover for a targeted
attack of one or two of those 7 they actually siphoned properly? I hope US
authorities will figure stuff out beyond the usual "it was a
Chinese/Russian/Eastern European gang", which is just code for "fuck knows".

~~~
vsareto
I wish I knew why security does this character assassination routine with low-
skill hacks. They clearly got in, bumbled attempt or not. Is it somehow
helping everyone to know they fucked up? Whose expectations are we changing
here?

~~~
spenczar5
Who is “security” here? In my experience, if you ask most working security
professionals, they would agree that social engineering is by far their
largest vulnerability. All the cryptography stuff is fun and cool but really
the work is all about preventing phishing, so I don’t know any working
security engineers who would call this a low-skill hack.

------
badrabbit
As someone who works to stop these, the most frustrating part is how even
infosec people thik enough training or $vendor's email security solution will
stop this. It's like boy scouts that think they will stop navy seals. There is
too much focus on entry point of an attack,especially by news media.

~~~
thaumasiotes
Speaking from experience at a major infosec company, the impression I got
internally was "we offer phishing tests, but we don't recommend them, because
phishing succeeds 100% of the time".

So I'm confused by the idea "even infosec people think training will stop
this".

~~~
TheAdamAndChe
I disagree. There are services that regularly send fake phishing emails on a
regular basis. If they click a link or fail to flag enough emails, their boss
gets notified that more training is necessary.

At the bank that I see this used at, the employees are far less trusting of
emails and such.

Training works if it's done right.

~~~
meowface
Yes, I've been involved with such a program before, and it definitely helps a
lot. Phishing email click rates go way down.

This is carefully planned phone-based spear phishing, though, and that's a lot
tougher to protect against. It can be easy for a skilled con artist to gain
someone's confidence over the phone, no matter how much you warn about vishing
(voice phishing). I'm sure training can still help there, but attackers just
keep trying again and again until they find someone it works on.

~~~
zentiggr
Same principle can apply though. If email phishing can be simulated and used
as training, voice can be added to that training drill.

Any successful attack vector can be turned into a training scenario and
repeated until better responses are trained into the target group.

Military casualty drills are very effective at instilling near instinctive
responses... same principle applies.

~~~
meowface
Absolutely. It's just that highly motivated, targeted, and sophisticated
social engineering is really tough to totally prevent. It just takes one
person to fall for it, and the attackers can keep cycling through people
(quickly, to get ahead of company-wide warnings about the social engineering
attempts) until they succeed.

------
heinrichhartman
What I find most problematic about the attac is the incident response by
Twitter.

As people pointed out here, hijacking Twitter accounts can lead to big stock
market crashes, mass panics ("bomb found at XXX") and maybe even military
escalations.

Under this circumstances, leaving a platform with an unknown number of
compromised accounts online, seems irresponsible to me. In such a case you
must stop the bleeding ASAP, either by locking up "important" accounts (what
they eventually did, after a few hours!) or taking the site offline.

Next time this happens, we might not be so lucky.

~~~
bzb3
No. We need more things like these to happen so people lose trust on
everything they read on the stupid internet.

This was good. I hope it keeps happening.

~~~
bananaface
If they'd taken the site offline then the event probably would have generated
more publicity.

------
ridaj
Freaking Twitter needs a serious auth infra upgrade. Unless phishers hijacked
employee devices, they accessed the tools remotely, meaning there's no form of
client authentication?? Something like U2F which by now is pretty old seems
like it would prevent this kind of attack

~~~
zobzu
what if.. they used the authentication?

But yes u2f/webauthn would probably prevent this.

That said keep in mind they also do PR/damage control so we only know what
they tell us. For all we know maybe they have u2f and an employee still did
bad stuff while a phone was somehow involved. Or whatever else.

~~~
reportgunner
Yes they gained working credentials from an internal slack channel by reading
them in plaintext.

[https://www.dailymail.co.uk/news/article-8536603/British-
tee...](https://www.dailymail.co.uk/news/article-8536603/British-teenager-
denies-Twitter-hack-admits-bought-stolen-account-Bitcoin.html)

------
gregors
Social engineering will never be stopped, people want to be helpful. And
generally speaking the cost for stopping it at non-secure businesses is going
to be too high until a security incident happens.

Phishing email attacks? Why do employees have business emails at all?

Phishing phone attacks? Why would employees have phones with external access?

Front of the house (dealing with users) should probably be disconnected from
back the of the house (admin access).

Before you know it you're in DoD or Bank territory. No Wifi allowed etc,
where's your badge buddy?!

Things get complex quickly with security.

~~~
tialaramex
> Social engineering will never be stopped, people want to be helpful.

They really do. And so you should design security systems with the
_assumption_ that your employees will actively undermine security "to be
helpful" to adversaries.

> And generally speaking the cost for stopping it at non-secure businesses is
> going to be too high until a security incident happens.

The cost for Yubico's "Security Key" is $20 and there is a volume discount.
You should buy each employee a key, and if there's no secure means by which
they can be re-authorised when they inevitably lose it, a second one to keep
safely for that case.

The attackers correctly anticipated that while "Can you get me Jenny in user
assistance's phone number?" is just being helpful, "Can you disable Elon
Musk's 2FA and give me control over his account?" is a bit... obvious. So they
got themselves credentials to do that stuff. But there is no need for Twitter
employees to be able to give away those credentials.

------
iamleppert
They should require hardware security devices (dongles). Really Twitter should
be ashamed of their poor internal security.

~~~
sushshshsh
Dongles are rare here in the US. But I know that bloomberg uses them. I was
shocked when I learned that retail banks in Singapore give everyone dongles to
log in. In the US that's tyranny Lol

~~~
Osiris
I work for a crypto currency company and it was the first time in my career
that I was issued a YubiKey (I once had an RSA 2fa token for vpn access). It
took some getting used to but now I just keep in on my keychain and I always
have it with me. I need it for SSO, git, VPN, and basically all internal
services.

They aren't sufficient by themselves however, they don't protect from is
malicious internal employees.

~~~
MAGZine
Preparing for malicious internal employees seems to me like preparing for "the
big one," in the northwest.

Do a cursory amount of preparation. Outside of basic measures, you're probably
doing more harm to the business than good. The likelihood of internal
malicious attackers is very low in the grand scheme of things, and the attack
surface is huge.

Most companies are going to be compromised by outside attackers—its there that
you should focus your energy. If internal attackers are your biggest threat,
you've done a fantastic job.

~~~
Thorrez
Well Twitter did indeed have malicious internal employees.

[https://www.washingtonpost.com/national-security/former-
twit...](https://www.washingtonpost.com/national-security/former-twitter-
employees-charged-with-spying-for-saudi-arabia-by-digging-into-the-accounts-
of-kingdom-critics/2019/11/06/2e9593da-00a0-11ea-8bab-0fc209e065a8_story.html)

If you're hit by a paywall:

[https://web.archive.org/web/20200717083254/https://www.washi...](https://web.archive.org/web/20200717083254/https://www.washingtonpost.com/national-
security/former-twitter-employees-charged-with-spying-for-saudi-arabia-by-
digging-into-the-accounts-of-kingdom-
critics/2019/11/06/2e9593da-00a0-11ea-8bab-0fc209e065a8_story.html)

------
dboreham
No employee should have the power to subvert 130 high value accounts in a
short time period.

~~~
djsumdog
Why do we even have 'high value' accounts on a centralized platform?

Why isn't there a whitehouse.gov ActivityPub instance that no single admin can
censor or subvert?

~~~
m463
We will do that now. We will start the competitive bidding process, and we
expect the RFP paperwork to be returned by October, 2021. After that, if there
are no injunctions filed because of the bidding process, preliminary design
documents will start being created. Preliminary design review will occur
August 2022. ...

~~~
rudolph9
I can’t tell if this is trolling or a serious comment of how this will roll
out?

~~~
Shared404
It's a comment on government inefficiency.

~~~
m463
Seriously I worked on projects subject to government scrutiny and it was ok,
but you had to be the right kind of person.

Account for your time in 6-minute increments. Milestones I recall off the top
of my head were preliminary design, detailed design, 3-5% of your time coding,
software integration, hardware software integration, acceptance.

It was stable, predictable, and (to me) very soul-crushing.

~~~
lovecg
None of that sounds “ok”...

Technical question for you, how does this time tracking work in practice? Do
you pause every 6 minutes and note what you’re doing? Or just roughly remember
at the end of the hour/day?

~~~
detaro
That's what time tracking software is made for. No need to pause, "just"
remember to switch the software if you change tasks/projects.

~~~
tialaramex
It becomes a reflex. Commercial lawyers all do it, tracking time this way is a
fact of life, not least because if you end up in court arguing about costs the
judge is going to throw out hand-wavy "I spent about a week on this" claims
from professional lawyers who should know better.

------
iKevinShah
Sorry if I am taking this on a tanget, but in one of the HN threads regarding
this exact security incident, it was recommended that this is why something
called as "Blast Radius" needs to be implemented.

Anyone here with any literature / sessions one could go through for a good
gist of things with respect to Blast Radius?

~~~
ak217
"Blast radius" is a general term for the worst case impact of a specific type
of breach of a given system.

The recommendation you read was probably about limiting the blast radius. It's
a general security best practice, and you implement it through techniques like
federating (compartmentalizing) services away from each other, limited
lifetime credentials, attribution, SSO for single point of control for
invalidation of credentials, principle of least access (PoLA), privilege
separation with role-based access control (RBAC), session logging/audit
logging, etc. Most importantly the underlying system needs to have a well-
defined and pentested authentication/authorization architecture. The hallmark
of systems that limit the blast radius is that they have well-defined limits
on how much they trust each other.

OWASP ([https://owasp.org/](https://owasp.org/)) is a great starting point for
reading about this stuff.

------
praveen9920
> We will be slower to respond to account support needs

So, they limited the account support access to smaller trusted team. This I
can understand but it might also cause the delay in identifying such attack,
if it happens today.

------
thereyougo
> We’re acutely aware of our responsibilities to the people who use our
> service and to society more generally. We’re embarrassed, we’re
> disappointed, and more than anything, we’re sorry. We know that we must work
> to regain your trust, and we will support all efforts to bring the
> perpetrators to justice. We hope that our openness and transparency
> throughout this process, and the steps and work we will take to safeguard
> against other attacks in the future, will be the start of making this right.

It's not easy to say those words about your own company.

I really believe them.

~~~
bananaface
I think it's a mistake to personify organisations like this.

~~~
baby-yoda
Agree. That reads like pure PR-corporatespeak to me - oops sorry, wont happen
again, business as usual.

“They” only “care” to the extent it materially affects the business.

------
vinniejames
Whats the final determination around the rumors employees were payed for
access to their creds?

------
gcpwnd
Slightly OT: I use private tabs and write my banks URL by hand and log in via
2FA. Are there cases or have there been attempts to poison the users address
bar history with malicious URLs for fishing?

------
exabrial
This is one of the better rcas, but I'd wish they'd punish the details of the
fishing attacks so the rest of us could learn.

------
bananaface
The spirit of Mitnick never dies.

~~~
jacquesm
Why would it have died? He's very much alive.

~~~
bananaface
He's not hacking.

------
patagurbon
Are account support tools available off premises? I know nothing about
security for big companies like Twitter but it seems like tools that enable
you to post from any verified user (outside of Trump, someone here once
mentioned he had additional account controls) should only be accessible from
secure offices regardless of individual credentials.

~~~
bawolff
That would probably not work well during the pandemic...

~~~
TwoBit
It could with an appropriately secured private VPN.

~~~
bawolff
Original poster said on premise. An appropriately secured vpn may or may not
help security, but it is still "virtual" and does not meet the definition of
on premise.

------
IncRnd
Account access is one thing, yes, and a hardware 2FA key can help with that.

But - what is the reason to allow support personnel to pose as specific users
and send tweets from their accounts? There is more than a security issue here.
There is a complete security breakdown.

~~~
Thorrez
I don't believe there was a tool allowing support personnel to pose as users.
I believe the tool allows support personnel to reset emails on accounts. Then
the attackers used did password resets on the accounts then logged into the
accounts and tweeted.

~~~
IncRnd
I didn't see that in the page for this article. But, that's a good point.

The specific text is, "Using the credentials of employees with access to these
tools, the attackers targeted 130 Twitter accounts, ultimately Tweeting from
45, accessing the DM inbox of 36, and downloading the Twitter Data of 7."

So, this doesn't say what actually happened. If it was employees posing as
users in order to post, that is an permission which should not be granted. If
it was as you suggest, a password reset, then there is a separate issue with
2fa that would be expected on these accounts.

Either way, there are serious security issue. This is similar to Oracle
calling itself "Unbreakable" and then getting broken. If Twitter cannot
safeguard against so many accounts getting injected with tweets, then
something is broken with Twitter's security model.

~~~
Thorrez
>separate issue with 2fa that would be expected on these accounts.

I believe the tool also allowed deleting 2FA from the account.

> But because the attackers were able to change the email address tied to the
> @6 account and disable multi-factor authentication,

[https://krebsonsecurity.com/2020/07/whos-behind-
wednesdays-e...](https://krebsonsecurity.com/2020/07/whos-behind-wednesdays-
epic-twitter-hack/)

~~~
IncRnd
In other words, there was a tool that allowed posing as users.

------
info781
You should need a PIV card to log into in the admin tools.

------
dirtylowprofile
They were hacked by kids on Discord.

------
0xUser
Source (with more details):
[https://blog.twitter.com/en_us/topics/company/2020/an-
update...](https://blog.twitter.com/en_us/topics/company/2020/an-update-on-
our-security-incident.html)

> The social engineering that occurred on July 15, 2020, targeted a small
> number of employees through a phone spear phishing attack. A successful
> attack required the attackers to obtain access to both our internal network
> as well as specific employee credentials that granted them access to our
> internal support tools. Not all of the employees that were initially
> targeted had permissions to use account management tools, but the attackers
> used their credentials to access our internal systems and gain information
> about our processes. This knowledge then enabled them to target additional
> employees who did have access to our account support tools. Using the
> credentials of employees with access to these tools, the attackers targeted
> 130 Twitter accounts, ultimately Tweeting from 45, accessing the DM inbox of
> 36, and downloading the Twitter Data of 7.

~~~
dpweb
No disrespect to those challenged with protecting such a huge target, but why
do admin tools even have these capabilities? I could see needing to disable a
user account or change some attributes, but why would an admin ever need to
tweet from it? There shouldn't be tools with God privileges even for admins.
Not surprising human error was involved in a breach this huge. So, how many
people had access to this tool? Is there a killswitch for the tool itself
available to very few, really very few, persons? edit: I dont know if the tool
can tweet but surprised 2FA can be stripped without a human being confirming
(ie.. the acct owner's social media person), especially for famous people.

~~~
extrapickles
The admin tool was used to change the email on an account, then the attacker
reset the password and got full access to the account. Apparently having 2FA
enabled did not stop this attack (admin tool probably had the power to strip
2FA from accounts).

So while the tool did not directly have the ability to tweet, it effectively
did.

~~~
csunbird
I feel like the power to reset emails and remove 2fa should be only held by a
very small subset of customer support, with proper training.

~~~
chedabob
That seems like it was the case, but the attackers got access to lower
privileged accounts and used them to find who had that access so they could
target them.

~~~
toyg
The key being "proper training". Those few god-level admins should be drilled
enough to defeat a phone-phishing campaign. In fact, they should probably have
custom procedures to look after their own credentials.

------
ngcc_hk
What they need for you is some Network ai (dark something) to see change in
user behaviour eg access accounts and honey pot accounts that any internal
staff would raise alert immediately

------
snoshy
Frankly, I love Twitter. They've always seemed like the "scrappy" big player
in the tech space that's played just the right side of the norm in many cases.
An appropriately filtered experience on Twitter as a user is actually quite
fun.

That said, much of this is atrocious news. For all of their engineering
prowess, seeing poor opsec failures combined with the lack of basic security
principles like "containment of blast radius" and "fast response to critical
failures" is not something you can easily forgive at this scale.

But who am I kidding... it would be especially rich if some of these takeovers
were enabled by simjacking-like attacks. Not that long ago, the only two-
factor auth mechanism that worked for me was SMS.

------
MoZeu
It is inexcusable that Twitter is employing people who are susceptible to
social engineering attacks like this. This is simple training and seriousness.

~~~
bawolff
Training that is notorious for being ineffective in practise and usually more
about box ticking.

Assuming that none of your employees fall for phising, much less targeted
phising, is woefully unrealistic. Especially at twitter's scale.

Assuming humans won't do stupid things 100% of the time is never an effective
security control.

~~~
idoh
Where I work there is training software that is somewhat effective at
preventing phising - it actually sends out phising emails itself. Then
employees who fall for it are given extra training (in a no fault sort of
way).

~~~
bawolff
Perhaps, but im also wary of these types of things, because i worry that
people will feel embarassed at being tricked, and will (maybe subconciously)
see the internal security team as the enemy, which is also a bad outcome.

I also worry that the emails might not represent real attack emails, and we
end up training users to identify the test emails but not real attack emails.

(Not that i got any better solution)

~~~
hutzlibu
Nothing is 100% secure. Having users fail to spot a pishing mail, is a very
good training on general awareness, but no guarantee, that they will not make
misstakes under pressure.

