
Engineer admits he wiped 456 Cisco WebEx VMs from AWS after leaving - swatkat
https://www.theregister.com/2020/08/26/former_cisco_engineer_aws_webex_teams/
======
kstrauser
I left one job to switch to another. A year later, my new job started using a
cloud-based task management app. When I went to sign in, my 1Password auto-
filled the credentials I'd used for the same app at my previous job, and there
I was looking at all of my old employer's current projects and other
confidential info. I called my old boss (who I got along with just fine), told
him what happened, and asked him to please cut off my access immediately.

When you leave a job, it's in your own best interest to make sure that all of
your access is removed. It's a lot harder for them to blame unexpected
happenings on you if you can't even log into the thing. (Not that this
happened here. I just wanted to point out a gotcha you might not have thought
about.)

If you find out that they missed something, report it to them immediately _and
keep that paper trail_ demonstrating your good intentions toward them. Then
hound them about it until they get around to fixing the situation. And for the
love of God, _don 't ever, EVER_ log in "just to look around". Absolutely no
good can come of that.

~~~
andi999
How would you know that they didnt disable you accounts without trying to log
in.

~~~
steve_adams_86
I think they meant ‘don’t log in, discover you have access, then browse and
take no further action’. You can try to log in, you just need to inform
someone immediately if you have access when you shouldn’t. That’s my take
anyway.

~~~
andi999
Successfully trying to log in can already be a crime depending on your
jurisdiction in my opinion.

~~~
steve_adams_86
You're right - I guess in that case it would be wise to ask your previous
employer to provide proof that you're no longer able to access anything, and
no longer liable. Many wouldn't offer that, but I'm not sure how else you
could navigate that.

------
saidajigumi
This article leaves more questions than it answers. Room-elephant number one:
access being available after an employee has left is bad. That access
remaining _five months later_ is beyond the pale, unless the real story is
that the employee created a backdoor. Barring a backdoor, there are further
serious questions about the employee retaining this access, presumably without
any employer-provided and controlled hardware (e.g. laptop, yubikey, or what-
have-you).

Room-elephant number two: motive. The reported facts naively summarize as
"oops, ex-employee blew up some stuff in prod, caused problems". <meme>But
whyyyyy??</meme> There's no indication of specifics, and seeming denials of
some obvious guesses: attempts at hacking (e.g. data exfiltration for profit,
which are denied), ransomware, revenge, or anything else that would explain
this behavior.

Further confounding everything is the bit where the new employer's response to
these revelations is apparently "shrug".

~~~
tootie
I worked in consulting up until Covid. When I got laid off, my employer locked
me out of every corporate system within 15 minutes. But every client who gave
me VPN, AWS or other credentials didn't get notified.

~~~
sangnoir
Interesting angle. I wonder if the perp was employed by Cisco directly or was
a contractor and Cisco wasn't informed when he changed employers.

------
nixgeek
Biggest question for me would be why the employee still had access so long
after terminating employment with Cisco.

A common piece of auditor evidence across many compliance frameworks is
whether employees have access proportionate to their role (which is naturally
highly subjective), but also proving that access is revoked when employees
leave the company. This seems like an outright failure on Cisco’s part.

Hopefully they’ve learned from this and put effort into enhancing their
identity governance situation.

~~~
chromedev
I knew someone that worked at WebEx in their FedRAMP environment performing
vulnerability remediation, and most of the work was outsourced to China
because the founders are from China. They posted screenshots of a conversation
with their teammate saying that because of the convoluted process to install
packages, they had concerns that they were installing some sort of backdoor.
They reported this and had investigators constantly checking out their
LinkedIn profile for months.

------
viraptor
I can't find any place explicitly saying this was done maliciously. A theory:
this could also be a really bad accident where (for example) he works without
a company provided computer and didn't clean out his old AWS profiles from
previous company - ended up deleting resources from the wrong account.

~~~
TheCondor
That seems like a lot of resources to delete... It’s “possible” but I’m not
sure they I find it plausible. Is there some magic terraform or cloud
formation that just destroys everything? And a Cisco engineer with devops like
experience wouldn’t at least sniff around before running it? Doesn’t seem
realistic.

Cisco should wear this too though, this is shockingly negligent. The only
reason I can think of suggests a lot more problems and likely noncompliance
with regulations and standards I’m sure they claim to comply with.

~~~
danielheath
I was at a large company some time ago that had a script to delete any
instances not tagged with a cost centre (from the testing account - they had
planned to eventually run it in production but that was still a ways off).

One day an AWS API had an outage causing it to return an empty array for the
tags list; the script deleted over 500 in-use instances.

~~~
RandoHolmes
I built an automated VPS system that would do things like create new VPS's,
install various pieces of software, and delete test VPS's, etc.

I kept asking the sys admins to create a limited access account for my testing
so that it flat couldn't delete existing customers VM's. I would walk into the
office of the lead for that team once every few months and make the request
again.

Until 1 day a bug in the VPS automation accidentally deleted a customer VPS
thinking it was a failed deploy. They finally got around to giving me a
limited account for dev/testing work.

It's scary how often this stuff falls through the cracks, even when employees
__KNOW __it can happen.

------
gruez
This paragraph is baffling:

>According to a court document, Ramesh is in the US on an H-1B visa and has a
green card application pending. "Although he and his employer recognize that
his guilty plea in this case may have immigration consequences, up to and
including deportation, his employer … is willing to work with him regarding
the possibility of his remaining in the country and continuing to work for the
company," the document [PDF] says.

Why would you re-hire someone who quit and wiped your servers?

~~~
cstejerean
It doesn’t sound like this is what happened though.

> During his unauthorized access, Ramesh admitted that he deployed a code from
> his Google Cloud Project account that resulted in the deletion of 456
> virtual machines for Cisco’s WebEx Teams application

It sounds like this may have been more accidental than malicious.

~~~
bpodgursky
Pure speculation, but I wonder if he had gcp service account credentials
sitting around his laptop which applied terraform to the wrong project.
terraform apply -auto-approve can wipe out a lot of infrastructure in a few
seconds.

~~~
gruez
Most crimes require mens rea for conviction. That is, the prosecution has to
prove beyond a reasonable doubt that you intended to do it. If it's really the
case that he accidentally ran terraform with the wrong project/credentials, I
doubt he'd accept the plea agreement.

~~~
nitrogen
HN has previously discussed the egregious power imbalance at play in plea
agreements.

------
blinkingled
This feels like may be he at some point discovered that his AWS access to
Cisco account was intact and got curious about it and maybe even did some
harmless things. Then while playing around with GCP he managed to run
something that deletes stuff (Terraform maybe) but the credentials used were
that of Cisco AWS account which wasn't what he intended - clearly just
deleting stuff is not the smart thing to do when it will be recorded against
his AWS credentials.

I think he is pleading guilty to unauthorized access which was intentional -
but not to the deletion which was unintended.

------
TwoBit
With that kind of sloppy security in place, Cisco is going to be easily
ransomwared.

~~~
chromedev
They already are. Based on someone I know, they were warned by their coworkers
at WebEx that much of the work was outsourced to China and they had concerns
that they were installing backdoors.

------
eithed
Phhhh, 6 months... I'd still have access to my first company's Google local
directory console almost 14 years after I've left it (this is despite numerous
messages on my part to remove my access) if Google were not to remove the
listing altogether

------
gregoriol
This is a really scary situation for an employee: maybe this case was a
mistake or maybe it was bad intentions, but imagine you are leaving a company
and your access should be closed, but is actually not, then your account still
active without your knowledge might get hacked... and you could be
responsible?

------
turowicz
It also means there was no orchestration for the VMs. System should have
recreated them 1 by 1 on a health check triggered restore action.

