
Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK - LinuxBender
https://www.theregister.com/2020/07/21/twilio_sdk_code_injection/
======
kirillzubovsky
I love AWS as any other developer, but this is their fault.

They've invested all their resources into building out the infrastructure, and
have spent very little if any time on building the UX around it. You have to
be fairly technical, never mind painstakingly detailed-oriented, in order to
manage their services, a big chunk of which is hidden behind black screens and
various control nobs.

Amazon's logic is probably to provide the piping, and leave the rest of the
end user, but at this point it's not enough. In my opinion, they need to
heavily invest into the customer-facing layer of their applications that shows
all the status, gaps and holes, or these things keep happening.

Then again, they can continue as is, and the incidents will keep as is, and as
long as nothing implodes and no big customer leaves, financially it makes
sense to do nothing. Just thinking out loud.

~~~
ericbarrett
I really have to disagree here, at least with your conclusions re complexity.
No doubt AWS S3 configuration is very technical and has a lot of docs and
corner cases. However, if you are a company like Twilio, and S3 is front-and-
center the source of your business (and if it's serving your SDK, it probably
is), your S3 bucket permissions are clearly worthy of periodic review. And you
should have this done by somebody who knows it well, or who learns it well in
the process.

I worked for a company where our CTO identified the same thing (unauthorized
modification of JS SDK served via S3) as a potential vulnerability. It took us
half a day to get the bucket locked down to where only the root AWS IAM
user—secured by physical 2fac—was permitted to change the SDK, and then only
after "unlocking" the configuration changes. Since the SDK was not updated
frequently, having a little manual ceremony was fine. If it had been a high-
touch path we would have had to do a little more engineering to keep it safe,
but not that much more.

That said, I don't blame Twilio for not catching this; shit happens at a
company with years of legacy built up. I do blame anybody who says "S3 is
hard" and throws up their hands. You're a professional! Read the docs, they're
dry but quite extensive! Play around with it in a test bucket! It's no harder
than learning any other moderately complicated professional system. If you can
learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn
S3 ACLs and the IAM authz model. It may not be as interesting as the topics
you want to read about, but it will probably be even more useful.

~~~
bjclark
The flaw in your logic is that "S3 is front-and-center the source of your
business" is never true with AWS.

To do anything in AWS there's at least 3 or 4 vaguely connected services,
including IAM and RAM as a completely separate UIs. Usually with S3 you also
have Cloudfront or some other CDN. You probably also have Cloudwatch logs and
CLoudtrail event tracking. You might have VPCs involved. Some or all of these
things, in an org like Twillio, could have entirely different teams managing
them.

The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of
actually doing anything non-trivial across half a dozen services is somewhere
between abysmal and war-crime.

~~~
orf
Each of these services are highly different concerns and don’t belong
together, and are honestly not _that_ hard to orchestrate together with the
liberal application of Terraform.

If you’ve got one team managing your logging who don’t talk to whatever team
who manages your s3/cloudfront setup, then that’s your problem.

~~~
morvita
> and are honestly not _that_ hard to orchestrate together with the liberal
> application of Terraform.

I love Terraform, but the fact that you require an entirely different
company's orchestration tools to make AWS "not that hard to orchestrate" is
evidence of how poor the UX of AWS is.

~~~
aldarisbm
But terraform is using the aws sdk behind the covers.

Most of the stuff available with terraform is available with CloudFormation
minus a thing or two. (takes them a minute to catch-up to other teams new
services/features)

------
whoisjuan
Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I
know many people open buckets to the world at the read-level for whatever
reason. I do that myself for static websites. But also writing permissions?

This requires a Bucket Policy that is written explicitly to allow this. I
don't want to make assumptions because of course we don't know all the
information and I personally know very little about infosec, but that sounds
very strange.

Why would you do that? I assume the SDK gets uploaded from a secure
environment that has the credentials to upload objects, so there's no
legitimate reason to have writing permissions open to the world.

~~~
kirillzubovsky
Potentially because it was spun 10 years ago while permissions were
basic/different/even harder to track, and has since fallen off the wagon?

It seems this incident has reminded Twilio to audit all their access right,
which is a net positive.

Actually, imagine you were an employee who wanted to get this done, but
couldn't get management approval? This is one way to fix things ... :)

~~~
fimbulvetr
For such an important bucket, they should have

a. adopted the resource into a Cloudformation stack & a1. Enabled drift
detection.

b. Use an AWS config rule to monitor (appropriate) s3 buckets for any public
access.

~~~
stevehawk
1\. Drift detection is a relatively new feature. 2\. CloudFormation was so
basic for so long that most AWS professionals don't use it. They use Terraform
or a similar, non AWS product.

~~~
fimbulvetr
You're absolutely right, and I think most of us have a hard time a. learning
about all of the new features and b. justifying going back and fixing up stuff
to adopt the new features, but IMO for such a critical piece of infrastructure
it should have been done and the engineers/architects should have been adding
that sort of stuff to the backlog or requesting training or asking for
resources, etc. For me, that sort of stuff is the only way I can get the green
light to go back and refactor.

------
tyingq
Doesn't sound so non-malicious to me:

 _" Specifically, the modification added code to the end of the TaskRouter.js
v1.20 SDK that made an HTTP GET request to
hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned
in the HTML by that request."_

~~~
oefrha
> followed the URL returned in the HTML by that request

Translated: we can't possibly know it's non-malicious.

TFA even has this to say:

> And judging from the URL involved, it appears to be an attempt to install a
> payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets
> targeted by miscreants.

Details in the linked blog post[0].

So, very much malicious.

Why the hell they included "non-malicious" in the title (granted, in quotes),
I don't know. Readers could have easily dismissed this as actually non-
malicious.

[0]
[https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/](https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/)

~~~
FDSGSG
FWIW, that's a redirector and not a skimmer.

~~~
dsl
Magecart is a well known framework in the underground for stealing credit card
numbers from websites (if they can get the JavaScript running on a checkout
page).

If you hit it directly they will redirect to a random "Spin the wheel win a
prize" type site in an attempt to hide the malicious nature of the payload.

~~~
ryanlol
This is not magecart.

~~~
heavenlyblue
So they got really lucky it wasn’t magecart - does this change anything?

~~~
ryanlol
Of course it does.

~~~
heavenlyblue
If someone drove drunk and drove into a pedestrian who survives after a month
in ER does this diminish the severity of drinking or not?

~~~
ryanlol
An incident where the pedestrian isn’t killed is almost certainly less severe
than one where the pedestrian is killed.

~~~
heavenlyblue
I just don’t see how that diminishes the action of crime though.

A failed drug dealer (who got caught), by your definition, should get less
time because they never managed to sell the drugs they were transporting over
the border and got caught :)

~~~
ryanlol
But why are we talking about the justice system here?

The impact of these actions matters far more to the victims than whatever
consequence the perpetrator may face.

If your website was affected by this hack, surely you’d care about how your
customers were impacted.

Lots of people here keep repeating the claim that this was magecart, that’s
just not true.

------
rafaelturk
S3 security is a pain. S3 options are tricky to master, docs are beyond
confusing.

A common S3 use case is: Use a Bucket for Read-only static content, (js, html,
imgs). From a dev perspective ideally it will work like a protected folder +
Web Server, whereas the webserver will have read-only privileges, but reality
is far more complicated.

I can understand why was easy for Twillio to have this infosec issue, since
we've done it ourselves many times when using S3.

~~~
ed25519FUUU
Is it really a pain though? Maybe because I’ve been working with it since the
beginning. Bucket ACL with NO permissions, then manage all permissions on an
IAM role on the account.

If it’s cross account then allow assuming to other accounts, but no reason to
bother with the bucket ACL.

Leave the bucket policy blank and you’ll never have to worry about an open
bucket. Better yet make a deny rule to everything but a single role, and you
don’t have to worry about rogue roles exposing the bucket.

Perhaps some of the issue is that permission can be managed on _both_ IAM
_and_ a bucket policy?

~~~
somehnguy
It is absolutely a pain. I'm using S3 for the first time on a project right
now and literally every time I have to do anything in AWS I end up confused
and scared that I'm leaving a wide open security vulnerability. The
documentation is complete insanity to anyone just trying to accomplish what
should be an extremely common use case in a reasonable amount of time.

Before using AWS I was in the 'what morons!' camp whenever some company got
breached via leaving things wide open on AWS. Now I'm in the 'ok I understand
why its so common' camp.

~~~
txcwpalpha
The problem with S3 is the exact same problem with everything else in AWS. In
their quest to be compatible and attractive to every single possible niche use
case of the world's largest companies, they completely forgot the lone
developer who just wants to upload a couple files.

You'll even see this in their sales pitches. AWS will spend so much time
talking about the most advanced use cases and how they are possible, but if
you ask them a simple question about how to run a single bucket that _doesn
't_ also involve using all of their ML tools and global accelerator and
cloudfront, they'll be blindsided. It's almost as if they consider the common
use cases to be "too simple/obvious" and therefor they never bothered to
create any documentation or put any thought into it.

~~~
pantulis
Your point is good, but one could suppose Twilio is not a "lone developer".

~~~
txcwpalpha
Certainly not, and in their layers of defense against this stuff, they should
have had processes to train people, audit bucket policies, conduct penetration
tests etc, and there's no excuse for them not having caught it before.

But with that said, one of those layers of defense is also using tools that
are easy to keep secure so that the vulnerability doesn't appear in the first
place. While Twilio isn't "a lone developer", I wouldn't be surprised if the
person who did create that bucket on behalf of Twilio _was_ just a lone
developer fumbling around in the console. And again, that's no excuse, but it
still is an area for improvement.

~~~
pantulis
I kind of miss the times when working with the cloud was easier or more
straightforward.

Now, as has been stated before, reading any cloud vendor documentation is like
reading the Oracle manuals of years ago, not a happy experience.

Granted, these clouds now do things much more powerful and target more
difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic
Beanstalk for that, but EB gets much more difficult fast.

------
carterklein13
One of the most shocking things I realized when I graduated college and
started working as a software engineer is how insecure, or really just
improperly-configured, large enterprise systems are.

This is anecdotal, of course, but it seems to me that any company that didn't
start cloud-native is going to have flaws like this, and it's often not at the
fault of developers but at the fault of top-down initiatives to "get to the
cloud" ASAP instead of correctly.

------
nwsm
Wow. They were serving one of their SDKs out of this unsecured bucket and
someone was able to upload a new version and it was distributed to customers.

------
kolencherry
Hey folks, we've published a post with more details on the incident here:
[https://www.twilio.com/blog/incident-report-taskrouter-js-
sd...](https://www.twilio.com/blog/incident-report-taskrouter-js-sdk-
july-2020)

(I work for Twilio)

~~~
aka1234
Thanks for posting this. I'm really impressed with the transparency Twilio
showed in actually admitting to having such a silly, silly bucket policy. Not
impressed that it was there in the first place; but that should go without
saying.

This incident report should really put to bed all of the "It's AWS's fault for
making things so complex" complaints. (To be clear, it won't... but it
should.)

Even a cursory look at that bucket policy should tell you something named
"Allow Public Read" should NOT be associated with anything named 'Put'. This
takes 0 AWS knowledge to figure out.

~~~
oefrha
Really not impressed with the obligatory "really impressed with transparency"
pat-on-the-back under every incident report for a big corp screw-up that
provides any details at all.

And stating to the press the clearly malicious payload is "non-malicious"
(assuming TFA didn't lie about Twilio's statement)? That's ridiculous.

~~~
ficklepickle
Even if the payload was not malicious when they looked, it could change at any
time. I don't see how that can be confidently labeled non-malicious.

------
natpalmer1776
I was curious about the URL that they indicated was non-malicious and so
looked it up on google. I found this any.run report:

Report:
[https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf9440...](https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf944069b625577889b83e425d54f887a/a5f1e345-5559-4955-960d-752bf877e028)

Original URL: h t t p s : / / gold.platinumus.top/track/awswrite?q=dmn

~~~
jlgaddis
Nice find. That certainly indicates the exact opposite of "non-malicious".

------
_alex_
If it’s world-writable, then is it really a break-in?

~~~
TheSpiciestDev
"In local news, stranger off the street walks through residence's open door
and moves around furniture."

~~~
ljoshua
Well, even in such a world, that would still be considered trespassing.

~~~
jfoutz
I am not a lawyer, I'm definitely not your lawyer. Trespassing and possibly
theft, I think. Theft includes moving stuff without the owners permission. so
if I ,say, have a tow truck and move a car across the street without the
owners consent, I'm a thief.

I only know this because a friend talked about a case where they rotated a car
in-place, and there was a question about is that theft? its center of mass is
still where they left it, so it wound up not being theft. Moving furniture in
a house is a weird one. Probably leans on ill intent, moving a chair to block
a door seems like it's pointed to giving the owner a hard time. moving a chair
to perform cpr has much purer motives.

So anyway, there's my ill-informed view.

~~~
benmmurphy
I thought in common law countries theft required you to have the intent to
permanently deprive the victim of the property. this is why there are separate
statutes to cover stealing cars because people just claimed they were taking
it temporarily.

------
cj
Quick shout out to GuardScript which exists to help catch malicious 3rd party
(or 1st party) javascript changes:
[https://www.guardscript.com/](https://www.guardscript.com/)

It's a really simple service that fetches a list of script URLs that you
provide, and notifies you with the before/after diff whenever the file
changes.

I found GuardScript through a Show HN a while ago
([https://news.ycombinator.com/item?id=20265141](https://news.ycombinator.com/item?id=20265141))
and have been very pleased with it.

~~~
jszymborski
Isn't this kinda solved by Subresource Integrity (SRI)?

[https://developer.mozilla.org/en-
US/docs/Web/Security/Subres...](https://developer.mozilla.org/en-
US/docs/Web/Security/Subresource_Integrity)

~~~
lozenge
Some sites use ad networks that prevent them from setting good SRI policies.

~~~
jaywalk
So you don't set the SRI on those scripts. Nothing about the Twilio script
referenced by this article would prevent SRI from being applied.

------
Someone1234
Twilio's examples[0] for taskrouter.min.js, even now, don't include
Subresource Integrity hashes. Subresource Integrity would have stopped the
modified script from being loaded, and removed a single point of failure.

Subresource Integrity is supported by every evergreen browser and 80%+ of
market share (no IE11/pre-Chromium Edge).

[0] [https://www.twilio.com/docs/taskrouter/js-
sdk/workspace/work...](https://www.twilio.com/docs/taskrouter/js-
sdk/workspace/worker)

------
orenfromberg
would using the `integrity` attribute in the `<script>` tag mitigate this
attack?

~~~
pquerna
Yes! Subresource Integrity is the exact counter-measure to this kind of
attack:

[https://developer.mozilla.org/en-
US/docs/Web/Security/Subres...](https://developer.mozilla.org/en-
US/docs/Web/Security/Subresource_Integrity)

However, on Twilio's documentation site, they do not include the integrity
attribute in their examples: [https://www.twilio.com/docs/taskrouter/js-
sdk/workspace/task...](https://www.twilio.com/docs/taskrouter/js-
sdk/workspace/taskqueue)

~~~
untog
It's difficult. I like the idea of subresource integrity a lot, but a counter
example: Twilio discovers a bug in their JS SDK. Without subresource integrity
they can push up a fix and propagate it to all clients immediately. With
subresource integrity they would need every client to update their web site to
change the integrity attribute. If it was a serious bug then that could be a
huge issue.

Even putting the security arguments aside you can bet a lot of customers would
be really irritated having to constantly update this stuff on their site. If a
competitor didn't have that restriction it would become a selling point. IMO
the ideal is that Twilio provides both options: a "sdk-latest.js" that's a
moving target, as well as "sdk-v1.3.js" that is frozen and can be locked with
subresource integrity.

~~~
erichurkman
What about an alternate for these 'platform script includes' like Twilio and
Stripe:

    
    
        <script src="..." report-integrity="https://cdn.stripe.com/report">
    

in which clients compute the integrity and asynchronously ping a lightweight
payload of the integrity to the third party? You would then seed your report
service with a list of known hashes and set up alerting for when new hashes
were observed.

~~~
TheSpiciestDev
Script tags (added programmatically) should support such errors via "onerror"
functions. Though, I don't know if integrity errors could truly be determined.
If they can then you can do whatever you want with such an error.

------
donor20
One thing I wish AWS would do.

Allow admins to "down feature" the account. Ie, simple mode.

Basically, allow admins to toggle OFF S3 ACL's and even S3 Bucket Policies -
zero those out. Then let users use IAM policies for access which may be a more
familiar / more used mental model.

~~~
codeduck
A better approach is simply not to grant normal users modify access to IAM,
VPC, or anything else potentially destructive, and force them to make all
changes via a ci/cd pipeline with ops/sre oversight and review.

------
ebg13
> _someone was able to get into Twilio 's Amazon Web Services S3 bucket, which
> was left unprotected and world-writable_

"Someone" was able? If the bucket was unprotected and world-writable, then
_everyone_ was able.

> _' non-malicious'_

> _And judging from the URL involved, it appears to be an attempt to install a
> payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets
> targeted by miscreants._

In what world is that non-malicious? Non-immediately-damaging _maybe_, but it
seems clear that there was a malicious chain of reasoning involved. Twilio got
super lucky if this is all that happened.

------
_josh_
My immediate thought was about their Authy Desktop app - looks and feels like
a web app in a container. If someone gets access to that then boom, all your
2FA tokens are stolen.

Time to find an alternative.

------
dbielik
Doesn't look isolated to Twilio - seems to be on 85 websites via other
affected 3rd party scripts: [https://www.nerdydata.com/reports/gold-
platinumus-top-track/...](https://www.nerdydata.com/reports/gold-platinumus-
top-track/c5bb39e3-baa4-40ec-9dcb-f53876a59471)

------
paulie_a
Non malicious? Does the person. That wrote that bit of bullshit even believe
it.

They might as well have said "we are not smart enough to figure out the nature
of this code. In unrelated news we are hiring senior JavaScript engineers,
experience in opsec is a bonus"

------
zelly
This wouldn't have happened if they just used an old school nginx file server.
It has sane defaults like never letting random people edit your files through
an API. Oh and it also would have been cheaper than S3. Just sayin.

------
madmulita
English is not my main language, but "broke into" doesn't implicate "breaking"
something to get "into" something?

~~~
fokinsean
Yeah, in reality its like complaining someone "broke" into your care when you
left all of your doors open.

~~~
JadeNB
> Yeah, in reality its like complaining someone "broke" into your care when
> you left all of your doors open.

And you wouldn't get much sympathy, but this is a legally viable complaint.
When my car was stolen, the thief was charged for theft, but his girlfriend
who was joyriding with him was charged for criminal trespass. Just being in
the car when you don't have permission is illegal.

------
Schnitz
Is there a Twilio page that's confirming this?

------
bzb3
How did they track if the modification worked? By calling home? Because then
they leaked PII

~~~
aeyes
This is the Javascript SDK so checking any website which uses it would be
enough to see if the modification is present.

~~~
bzb3
Twilio says the only modification done to the script was to check if the
modification could be done. If the hackers modified the script so it would
call their own servers to check for that, they now have a list of IP addresses
and timestamps.

------
fmakunbound
Don't they sign their code so that package managers can verify it's really
from Twilio?

------
coding123
Aws should start taking blame for this shit, then maybe they would start
auditing their customers setups...

~~~
aka1234
AWS's shared responsibility model is clear. AWS is responsible for security
_of_ the cloud. The customer is responsible for the security _in_ the cloud
(i.e. the customer's resources). By the way, enterprise support customers do
get access to well-architected reviews by AWS.

If you ask for help from AWS, AWS will provide it. It may not be free, but
it's available.

Even if AWS were to start proactively auditing customer setups, how in the
world is AWS supposed to know what a customer's usecase is? Nevermind the fact
it's a breach of the customer's privacy to just go rooting around in the
customer's account without permission.

But let's assume AWS is going to take responsibility for customers'
configuration decisions and violate customers' privacy by proactively auditing
their accounts. Would AWS auditing Twilio's configuration here work?

The default is for S3 buckets to be private. The customer has to take
specific, affirmative steps to give s3 buckets public access. You really have
to jump through hoops to make a bucket public accessible.

Since the Twilio _chose_ to make their bucket public, AWS auditing Twilio's
setup wouldn't be helpful. AWS would just assume the Twilio knows what they're
doing. How is AWS supposed to know there is a misconfiguration? Because Twilio
clearly decided to make their S3 bucket publicly available.

~~~
donor20
Not only do you have to jump through hoops to make it public, if you are
worried can't you use Amazon Macie and have a separate org level view that
alerts you to any public buckets?

[https://aws.amazon.com/macie](https://aws.amazon.com/macie)

I don't bother because it seems pretty clear what buckets are public.

That said, quick tip to make your life easier.

DON'T use S3 ACL's DON'T use S3 policies.

If I was AWS and didn't have so many customers I'd probably just create one
mental model (IAM policies probably) as the place to manage things and block
the rest.

