
Citibank IT guy deliberately wiped routers, shut down 90% of firm’s US networks - jackgavigan
http://www.tripwire.com/state-of-security/featured/citibank-it-guy-deliberately-wiped-routers-shut-down-90-of-firms-networks-across-america/
======
daenney
What I find curious in this article is that they gloss over that this is also
a people management problem. Sure you can put restrictive policies in place
but apparently things went down at Citibank that caused so much ire for the
person in question to take action like that.

I like to think that even though people see sysadmins are grumpy rage-machines
in a basement most are super-decent people caring deeply for the systems under
their control. Perhaps I've got a too rosy view of my fellow engineers.

I'd be very curious to hear what went on there on the people side of things.
What led to this person feeling so angry that they would take an action like
this. I'm also curious how it could even get so far. I have the benefit of
being able to regularly talk to my manager so even if we did performance
reviews in that way they wouldn't come as a surprise.

~~~
alt2319
> I'd be very curious to hear what went on there on the people side of things.

It's important to study that in these cases. I was convicted in a similar case
and am actually a chapter in a CERT book, but they never reached out to me for
my input, so they're completely oblivious to my motives and missing key facts
about the case. I'd write more about it or speak at conferences but I've been
able to bury that past and move on. Maybe some day.

I think he had a similar case to mine where he felt he was stuck in a
situation with his manager(s) and felt like he had no recourse. HR should be
involved in employee reviews and should provide a way for the employee to give
feedback on his own review.

Another aspect of it is education about the law. I see case after case where
the defendant had no idea he could face federal charges at all, much less one
that can result in such stiff penalties. (The penalties he was facing were
much, much worse if he had been tried and convicted rather than taking this
plea deal.) That could easily be part of any degree program and/or employee
orientation.

~~~
pc86
> _I think he had a similar case to mine where he felt he was stuck in a
> situation with his manager(s) and felt like he had no recourse._

Still no excuse to take down 90% of a company's connectivity.

~~~
tracker1
If a company is bringing in child labor in dangerous factories, and employing
prostitutes to gratify the upper management, while you're being whipped daily,
would that, maybe be an excuse?

I know it's a really out there example, but there are times where outright
rebellion, even destructively, is morally acceptable. I don't know the
specifics of this example, however.

~~~
runamok
That's a pretty big strawman. A better recourse would be to whistle blow in
that scenario. Though clearly that carries a great deal of personal risk too.

Destroying a company damages all of that companies customers and employees as
well. The difference between a surgical strike of a "bad guy" and carpet
bombing a country.

~~~
tracker1
And if the government is complicit, and acting against its' own laws? That
isn't such a strawman.

Also, blowing the whistle can mean many things... in this case, it was a
temporary disruption, not much different than an organized strike would be.

~~~
johansch
Corrupt stuff in third world countries do not get entire chapters in CERT
books - that's where things like these are _expected_ to happen. Over and over
again.

~~~
tracker1
The NSA spying incident that Snowden revealed wasn't a third world country.
Then again, he didn't take down the NSA's routers or spying aparatus on the
way out either.

~~~
johansch
> The NSA spying incident that Snowden revealed wasn't a third world country.

Yeah, isn't that what I said?

------
riprowan
Used to be there was "the guy" that knew the "secret formula." Like the guy
that knew the Coke secret formula, or the guy that used to maintain Lay's
potato chip fryers. Most everyone else was just a cog in a big machine with
very little power individually.

What this guy - and Snowden - show is that today's IT professionals have
_tremendous_ power that generally they never choose to exercise.

If an IT shop in a major corporation decided to unionize and turn on its
management, they would have the company totally by the balls. So many people
in an IT shop are positioned to control so much, and management often treats
its IT staff like disposable widgets.

It would take two+ hands to count the number of major multinationals that I
could have _seriously damaged singlehandedly in a matter of hours_ if I had
ever gone rogue back in the day. Few non-IT rank-and-file employees have such
power.

Of course the fact that I never did such a thing is one of the reasons I
remain a well-paid consultant ( _trustworthiness and honesty_ is a
consultant's #1 asset), but the fact is that many corporations are quite blind
to the level of risk they undertake when they mistreat their IT staff.

~~~
Retric
The critical difference is factory workers can shut everything down by not
showing up. IT needs active sabotage which is a very different situation.

~~~
dragonwriter
> The critical difference is factory workers can shut everything down by not
> showing up. IT needs active sabotage which is a very different situation.

If you have achieved a genuinely 100% ops-free setup, sure. Otherwise, an IT
walkout will shut things down just as surely as a factory walkout.

~~~
joezydeco
Or worse, if management decides "any monkey can do this! Let's get Carl from
engineering and put him on the job."

~~~
tachion
Or worse, if management says about 'Java Build Engineer' (who's main task was
setting up extremely poorly written Jenkins jobs) 'He is a Build Engineer, I
think he can build stuff' and puts him in charge of designing and building new
infrastructure for a multi million dollar product, because you know, these
pesky, nerdy, grumpy devops guys are overpriced, I mean, look at devs, they
build their own stuff in AWS. This is madness, who needs infrastructure people
anyway?!

------
rabboRubble
I worked in Citi some years before, during, and after the 2008 crisis. Held a
two positions, IT and a business aligned roles. I left IT because the
treatment of staff was atrocious, and I could see the cost cutting hammer
dropping on IT.

In regards to the integrity of their employee evaluation process, I have an
anecdote to share. We used a 360 evaluation process. My review set was my
direct manager, plus 5 other people with whom I directly worked. That year, I
received a 5 out of 5, the highest rating possible.

During the official manager / employee review sign off meeting, my manager
noticed that my 5 was no longer a 5, but a 3. He canceled the meeting to
investigate what happened. Apparently, his manager with collusion with a
senior HR manager, went into the employee review system and manually overwrote
the ratings provided by my manager and colleagues. The reasoning provided was
that there were too many high ratings within my employee pool and I was
selected by "someone" to be downgraded.

I can see going into a Citi employee review meeting, coming out, and wanting
to nuke the entire site from orbit.

~~~
pc86
Understanding the feeling and following through with it are two very different
things.

~~~
wutangson1
underrated post.

------
sqldba
I don't want to know about what he did unless there's some information on why
he did it also. Was he pushed over the edge after years of mistreatment? Or
was he mentally unstable and it was petty revenge?

Most of us have the keys to the kingdom in one way or another and while I (and
others) could never do this it doesn't mean I don't have sympathy for someone
who does - for some justifiable reason. I can't imagine any that are
justifiable... but I can ask.

I found the other linked articles about, "The Malicious Insider", a bit fucked
in the head. Unapproved hardware! Email misuse! Unapproved workaround!
Unapproved software! Are you fucking kidding me? That's a pretty far cry from
this guy.

~~~
semi-extrinsic
From the SMS quoted in TFA, seems like he at least thought he and his
colleagues were being continually mistreated by upper management:

“They was firing me. I just beat them to it. Nothing personal, the upper
management need to see what they guys on the floor is capable of doing when
they keep getting mistreated. I took one for the team.”

“Sorry if I made my peers look bad, but sometimes it take something like what
I did to wake the upper management up.”

~~~
jameskegel
This may sound petty, but it is sincere nonetheless; I'm curious how someone
with his command of the English language rose to such a position of power to
be able to cause this much damage.

~~~
mfoy_
I had an IT internship where my manager handed me a master key to the building
and root logins for quite a few boxes. I also had the admin credentials to
every user computer in the building.

Literally the "keys to the kingdom".

Sometimes the default level of trust is just really high...

~~~
tracker1
It really depends on the company (size, number of staff) and the position
itself. I've been in similar situations... I've also worked in smaller orgs
that operated tighter operations than fortune 500 companies.

------
throwaway11teen
This happened when I was at Google, but it was an accident.

On someone's last day they ran rm -rf / to wipe their desktop machine. I can't
remember whether they typed it into the wrong terminal window or had mounted
an external filesystem somewhere deep under their home directory and forgot
about it, it was something like that. They wound up wiping some unfathomably
large amount of data (I think most of a data center) before noticing what
they'd done.

There were backups, but it was still a huge hassle.

~~~
dba7dba
Oops. Wrong terminal.

Always a fear of mine. I always try to pause for a second before I run a
command like that.

~~~
tedunangst
Easy solution: start a new terminal.

------
ryanmarsh
One of my clients, a well known fixture in American finance, has been hacked
by employees or contractors several times over the past 10 years. These cases
have been public and supposedly cost millions (it's so hard to know how
accurate the assessed damages in these cases are).

Because they now fear employees and contractors more than external threats it
takes 30 tickets to different groups to set up a server. You can't chown a
file in a directory you own without a ticket. You can use one of two old text
editors. No new IDE's or modern text editors. You can't upgrade language
runtimes without corp. approval. It's nuts. Ironically it leads to more attack
vectors and it's ground their software dev to a halt.

~~~
st3v3r
All because they decided that was easier than simply treating their employees
like human beings.

~~~
pc86
> _has been hacked by employees or contractors several times over the past 10
> years_

It's simply an overreaction to risk. Just like insisting the company doesn't
treat its employees like human beings is also an overreaction (and gross
oversimplification of what is actually a complicated topic).

~~~
st3v3r
Considering we're talking about Citi, who does have a track record of treating
their IT staff poorly, you can't call my statement an overreaction or a gross
oversimplification.

~~~
ryanmarsh
I certainly wasn't talking about Citi.

------
ChemicalWarfare
I worked for a large company that went through drastic downsizing at one
point. Some ppl who were getting let go were "well positioned" to do some
damage. The way the company handled this is there was no advance notice. Your
manager and a couple of security dudes shows up at your desk with boxes and
you're outta there in 5 min.

There were things like managers getting punched, ppl screaming etc but no
malicious hacking that I'm aware of. Somehow one dude (I'm assuming :))
managed to take a dump in the elevator on the way out :)

They had EMS parked outside too. Wasn't pretty.

~~~
seren
Sorry for the naive question, but in that kind of situation, how do you assure
the continuity of the organization ? What if the people being let got were
supporting customers, about to close a deal, and so on ? How do you make sure
people are not leaving with a critical "know-how" ?

~~~
MagnumOpus
People in the middle of a deal or with crucial know-how will be called to
serve out their notice period for handover - which if the HR is switched-on is
1-3 months. Usually some extra severance money will be offered as a sweetener
compared to running away immediately.

However, potential to do damage usually supersedes potential to benefit the
business in the notice period. A company with sane HR has preventative
policies in place:

\- Never have single-person key-man risk in the first place; always have a
backup/deputy designated well in advance because anyone can be hit by a bus.
Helps with covering holidays too.

\- For highly sensitive jobs like sysops or people dealing with customer
money, revoke access to systems first thing in the morning, then tell them,
then immediately escort them out.

\- Don't have shared or master passwords anywhere, but have user-based ACL
instead so access _can_ be revoked.

------
zeveb
> “They was firing me. I just beat them to it. Nothing personal, the upper
> management need to see what they guys on the floor is capable of doing when
> they keep getting mistreated. I took one for the team.”

> “Sorry if I made my peers look bad, but sometimes it take something like
> what I did to wake the upper management up.”

Ah yes, the old double 'they are harming me by holding me accountable' and 'I
am helping them by holding them accountable' defense.

What I'd like to know is how mindsets like this are formed in life, and how to
prevent them. One's employer doesn't owe one a job; one owes one's employer
fair labour in return for one's wages. The world does not consist of shadowy
forces plotting against one. Don't minimise one's own agency: be an active
force for good, not a passive subject of whatever happens.

~~~
oxryly1
> One's employer doesn't owe one a job

Labor laws disagree to a certain degree -- at least in that you can't fire
someone for any reason you choose (see the EEOC poster in your
breakroom/mailroom/kitchen for examples). Also employment contracts can
stipulate various things that prevent a company from firing you, particularly
in union jobs.

------
carlmcqueen
I started on a help desk, from my experience the management knew which ones
were smart enough to do real damage and triggered enough to actually do it.

If I had any advice it would simply be to avoid letting a corporate
environment allow you to forget you have real people as employees with
feelings. "It's just business" doesn't translate to everyone.

~~~
superswordfish
> triggered enough to actually do it.

What does this mean? Do you mean "provoked"?

~~~
wccrawford
I think they mean that they have a lot of (or very strong) triggers, or things
that will make them act irrationally.

~~~
basch
usually triggering leads to paralysis not retaliation, no?

~~~
daenney
A trigger is an event that precipitates other events, aka it sets things in
motion. It could be paralysis, or it could be ejecting a bullet from a barrel
or wiping a set of routers.

~~~
jameskegel
The commenter above you is trying to make you aware that "triggered" is an
internet SJW dog-whistle to attract attention to an event. His statement is
forcing you to acknowledge the definition of the word "triggered" in its most
literal sense.

~~~
ZeroGravitas
Can you explain this more? It doesn't make any sense to me.

Note, I am aware of the use of trigger as in "trigger warning" for potentially
disturbing content. Even with that context I'm lost. Did you mean "literal"
literally?

------
hardwaresofton
Yeah or don't be shitty to your employees, or don't hire managers that are so
bad at giving negative performance reviews that the employee goes out for
revenge. This is definitely a people management problem.

The only way to rid yourself of the threat posed by other humans is to run a
single-person company, and even then you might screw yourself over.

You will never be able to completely neuter a threat that does any useful
service for your company, and it's foolish to think you can. Any useful amount
of delegated power or responsibility without the ability to misuse it is very
rare, if not impossible.

~~~
ceejayoz
> Yeah or don't be shitty to your employees, or don't hire managers that are
> so bad at giving negative performance reviews that the employee goes out for
> revenge.

There are people unable to take even the nicest, most constructive of
criticisms (or even perceive non-critical statements as such). Hard to say
without knowing the people involved on a personal level.

~~~
pjmorris
Can you see that it's a people management problem to be careful about the
roles you give the kind of people you describe?

~~~
ceejayoz
Some of these people have learned to hide it pretty well.

------
agounaris
You cannot control the human factor so easily but you can definitely keep you
employees happy! I don't get why the article does not mention the quality of
working environment. IT systems can be extremely stressful if there is not
help from management, and most of us know what bad management means...

~~~
computronus
I agree. There's a lot of focus on punishment, consequences, and all the
lockdowns for frustrating insider threats, but not so much on the flip side:
reaching out to people, valuing them and their needs and opinions, working
together, defusing problems so that there are fewer threats.

There are analogies to broader threats, say, to entire countries, and these
days a lot of people are more interested in being antagonistic and hostile in
dealing with the threats than working to understand them and their grievances.
The two sides may never agree, but the effort makes all the difference.

------
123456Seven
I wonder if the fact that this was a bank had something to do with the short
sentence. Back when I was a pr activist, it was known that the level of public
anger - the public's trigger - on an issue would sway the decision of high-
profile cases - that is, the public's perception would factor into the judge's
decisionmaking. If crimes were committed inside certain "pathways", those
crimes were easier to get away with under certain constraints. This man has
sabotaged a bank using a computer and made pro-union statements. If he had
made broadly anarchist statements I wonder how this sentence would have
changed.

I argue that these dynamics are relevant from a criminal psychology and a
government-hacking standpoint. Being mostly derived from firsthand resources,
Toffler, Bernays, et. al, along with some authors whose mention is straight-up
dangerous, I am constantly seeking additional resources.

Uh. Hack the planet.

Edit: or don't. U du u.

~~~
bryanlarsen
21 months in a federal prison is a short sentence?

~~~
123456Seven
Yes. The AT&T hacker 'weev' was sentenced to 41 months prior to the vacation
of his conviction. As far as I know no systems were disrupted in his case. He
expressed vaguely anti-authoritarian leanings without appealing to a stronger
base.

This is arguably what keeps shows like Mr. Robot on the air btw. We will
tolerate hackers down certain pathways and within certain constraints. Even if
they're "insane". Leave the pathway and God help you.

Sometimes staying on the pathway is dangerous too, but my job is to motivate
for great justice.

Edit: Don't kill or torture anyone. This should be in your base operating code
anyway; if it isn't, enforce it down to pith level.

~~~
alt2319
Federal sentencing guidelines are pretty rigid and most computer fraud and
abuse cases get the lowest end of the range.

The EFF has a good breakdown of weev's sentence and the D&D-like charts and
calculations used to reach it. He got hammered for "special skill",
"sophisticated means" and "means of identification" bonuses.

[https://www.eff.org/deeplinks/2013/03/41-months-weev-
underst...](https://www.eff.org/deeplinks/2013/03/41-months-weev-
understanding-how-sentencing-guidelines-work-cfaa-cases-0)

------
mc32
This is reminiscent of the San Francisco network administrator who took it
upon himself to "protect" the city from his supervisors and would only divulge
his password to mayor Newsom.[1]

IIRC it was also precipitated by perf reviews or discipline.

Hope AWS has better systems in place to make something like this nearly
impossible to carry out.

[1][http://m.sfgate.com/bayarea/article/S-F-officials-locked-
out...](http://m.sfgate.com/bayarea/article/S-F-officials-locked-out-of-
computer-network-3205200.php)

------
893helios
This is a total nightmare scenario for a lot of organizations, and is quiet
doable for a lot of systems. Specifically on the router side (thanks Cisco) I
haven't seen a lot of controls that would stop this. Network Engineers need
full access to get things done in a timely fashion, and the limitations of
control in TACACS. Recover from insider attacks of this nature are all based
on how much to drill to recover from this style of attack.

~~~
brianwawok
Someone has to have root at some level right?

So why is the router attack any worse than a "Delete exchange server + all
backups" command, or the "use ansible to remote format entire server farm in 1
click" command?

You have to trust employees at some level. If people know that doing things
like this will equal jailtime, I would assume that would stop most people.

Now think if this employee lived in Russia and did this remote. What would the
recourse be?

~~~
zeveb
> Someone has to have root at some level right?

In Unix, yes, but there have been systems without a single ultimately-
privileged user.

One could imagine a system in which ultimate authority belongs to a 51% share
of stockholders, whose keys delegate authority to the board of directors, who
delegate authority to the CEO or CTO, who delegates authority on down the
line. Each certifying party could revoke (or allow to expire) authority prior
to firing a delegee.

> If people know that doing things like this will equal jailtime, I would
> assume that would stop most people.

What I want to know is how people grow to physical adulthood without realising
that this is _wrong_. One simply doesn't destroy others' property.

~~~
brianwawok
Okay so a deploy to prod should be able to require a stockholder voter? This
sounds like a Utopian paradise not the realities of IT in 2016.

~~~
zeveb
> Okay so a deploy to prod should be able to require a stockholder voter?

Well, every deploy to production in any company is the result of a shareholder
vote — it's just made obvious.

By using delegation and certificates, an employee who is delegated authority
to deploy to production can do so without requiring a majority of shareholders
to actually vote on that particular deploy.

That's pretty awesome.

> This sounds like a Utopian paradise not the realities of IT in 2016.

We've had the ability to do this for almost twenty years: it was made possible
by RFCs 2692 & 2693 in 1999. The necessary processing could be performed in a
split second.

~~~
zzzcpan
The system able to do that will have to be designed by someone and maintained
by someone, because it cannot just appear out of thin air, work perfectly and
never have bugs. So, ultimately there will be someones with enough power to
change the way every single thing in the system operates and therefore enough
power to destroy the system.

~~~
brianwawok
Agreed, it sounds like he shifted the blow up the world key from Engineers to
a Board president.

Which is fine, but seems like a roundabout way to do it.

------
nstj
> The truth is that the person hacking you may not be someone you’ve never
> met, wearing a hoody on the other side of the world.

Take note - just in case, don't wear a 'hoody' as people may think you are a
'hacker'.

------
GrantSolar
This is really off topic, but the inclusion of the man "icon" was bizarre.

~~~
voltagex_
It's an "administrator" apparently.
[http://www.iconarchive.com/show/windows-7-general-icons-
by-i...](http://www.iconarchive.com/show/windows-7-general-icons-by-
iconshock/administrator-icon.html)

------
atmosx
What precautions can one take against their sysadmin? I can't think of many
places that have stricter security policies than 3-letter agencies, still
Snowden got away with a USB stick (man, can't wait to see the movie - I wonder
if he really passed the USB stick out using the rubik's cube as in the movie
trailer).

The thing is that that you can't do much against someone who has root access
for a living.

There are other ways to make a point, especially when you're prepared to get
fired. Damaging everyone else is selfish and stupid.

~~~
lgieron
> The thing is that that you can't do much against someone who has root access
> for a living.

In banks, significant decisions (like personally authorising multimillion
dollar transfer) require two signatures (of two senior officers), so they
can't be abused by just one person going rogue.

It can be imagined that we apply the same approach for system administration -
"pair-programming" work only, perhaps with proximity cards which detect when
one of the admins is actually not at the desk.

~~~
superuser2
How does that _actually_ work? I'm tempted to think it's down to one person
who validates that two signatures are present.

~~~
compsciphd
I think the LISA paper does a good job describing an approach.

basically don't allow people to commit changes directly to the underlying
system, require the changes from 2 administrators to be captured and then
compare them for equivalence. (what equivalence means can be bit for bit
equality at the most basic binary level, or perhaps some parsing that can
verify equivalence at language level for configurations)

if the changes are equivalent commit them. otherwise point out the
differences.

As the paper also discusses, can be used with a single administrator to make
the administration auditable, simply capture all changes before they are
committed.

------
nickpsecurity
Sounds like a management issue more than security. However, post-Snowden, I
did some exploration of a strategy for countering sysadmin risk in
organizations whose IT activities were pretty regular.

[https://www.schneier.com/blog/archives/2013/08/nsa_increasin...](https://www.schneier.com/blog/archives/2013/08/nsa_increasing.html#c1623226)

------
rm_-rf_slash
It's a little unsettling to think about how the biggest threats to an
organization can be from within. The inside hacker doesn't even have to be
disgruntled: they could just download the wrong link or put in a flash stick
from the parking lot and BAM! More hacking in a minute than any Hollywood
effort could get you.

The worst part about it is that to be completely on the ball with internal
threats, you need to be constantly paranoid, assuming by default that any
employee with enough access is a threat, or a threat waiting to happen. Such
things do not add to team morale or cohesion, but in this case, a pound of
prevention is still better than a metric ton of cure.

~~~
mcbits
Regarding morale, there's always a "bad country" making headlines each week. A
disgruntled employee and an employee coerced by Bad Country's operatives can
do pretty much the same damage, so you might as well frame the risks and
policies in terms of protecting employees from the latter.

------
typetypetype
I would think the first and easiest thing to do to prevent situations like
this is to make it so a single person cannot run certain commands. Sort of
like the digital version of two keys on the opposite side of the room to
launch a missile.

~~~
swalsh
Honestly, it's not a bad idea even if the only reason is to just prevent
accidental execution.

In the company I work at, we have something similar for database executions.
In Dev you can run anything you want. However in prod, everyone only has read-
only access. If you wish to execute something with higher authority, there is
a web page we go to, where we can paste the code, a review board request, a
ticket number (if available) a business authorizer, and a technical authorizor
(other than yourself). It sounds like a lot, but it's usually pretty quick.
Every command is then logged (which I believe is more for SOX reasons). It's
annoying, but it's not that terrible of a system.

~~~
marcosdumay
It's always the same problem: Who will deploy such a system?

At some point, you'll get somebody with full access to your system. You can
reduce it into a very small window (at big cost), but you can't make it go
away.

~~~
PeterisP
Have a few people at a single computer perform the following ritual:

1\. deploy the system

2\. create new admin account with a long random password and/or 2FA physical
token

3\. print the password out, seal it in an envelope and put it in a safe not
accessible to any IT person; place the 2FA token in a separate safe controlled
by different person than the first safe;

4\. remove all other admin accounts - with the other present technicians
checking that no other account will have access

5\. if needed, with proper authorisation you can retrieve the password, and
implement the required changes (again, with a second person controlling what
exactly is being done).

If the systems require changes rarely, then this cost isn't that high.

------
gypsy_boots
> he has been sentenced to 21 months in a federal prison for transmitting a
> command that caused damage without authorisation to a protected computer

I'm genuinely curious, is this the actual charge? What specific law did he
break here?

~~~
maxerickson
Yes. It's from the famous CFAA.

[https://www.law.cornell.edu/uscode/text/18/1030](https://www.law.cornell.edu/uscode/text/18/1030)

It's a perfectly reasonable principle to write into law, a password or key or
whatever doesn't give an employee the right to damage the systems of their
employer.

~~~
merpnderp
What if it's and accident?

~~~
alt2319
Then there's no criminal intent (mens rea) and the law doesn't apply.

------
rbc
I'm kind of surprised that Citibank had such weak internal controls on their
network configuration. On the other hand, internal controls are not free.
Separation of duties and the two person rule are expensive in manpower and
coordination of larger teams. It would appear that whoever accepted the risk
of the network internal controls at Citibank, lost that gamble.

------
ebbv
This article is clickbait titled spam. There's a snippet of an interesting
story there but not enough for a real story. The "analysis" at the end of the
article is not really analysis at all.

There's interesting details behind this; were they really firing him? Did
Citibank make the horrible error of telling an IT worker that they are fired
but not immediately revoking their privileges? Could management have handled
his performance issues better, or were there red flags that he should have
been let go sooner?

All of that could be interesting but because this is just clickbait blog spam
none of it is in the article.

------
l3m0ndr0p
I've heard, first hand that Citi is extremely toxic. People are mistreated,
are abused (verbally) on a frequent basis. There's no accountability within
the ranks and management doesn't care and encourages it

I almost accepted an offer to work in their NY offices. When I was being
interviewed, the Manger of the group was divulging his dislike for the person
I was going to report to.

It's a sick place & the IT guy in the article sounded like he was being
abused. He shouldn't have done what he did, He should have documented the
stuff he went through and maybe get a lawyer and sue for pain and suffering.

~~~
st3v3r
One might say that these things happening are the market telling Citi to stop
being a dick. Unfortunately it seems like they're not really taking the hint.

------
lagadu
As someone who works as an identity and access governance consultant and
developer, I endorse this article.

I mean, there's very little you do about this particular scenario other than
some small mitigations (at the end of the day you'll have to trust someone
with something) but it's genuinely amazing how vulnerable the processes inside
large, multi-million companies are to any internal attacker.

~~~
atmosx
> I mean, there's very little you do about this particular scenario other than
> some small mitigations (at the end of the day you'll have to trust someone
> with something) but it's genuinely amazing how vulnerable the processes
> inside large, multi-million companies are to any internal attacker.

Maybe the scenario is too rare for companies to care, so when it happens they
just accept the cost of _once in a while_ instead of setting up additional
security policies which will turn procedures cumbersome.

~~~
coredog64
Maybe you're working at different firms than I am, but the typical reaction I
see from management is that there is no problem that can't be solved with yet
another layer of process.

------
geggam
The more automation we have the harder its going to be to figure out if the
large scale outage was intentional or not.

------
iamapatsy
Throwaway.

Obviously stupid on his part. He needed a patsy. He needed to get someone else
to do this. Some kiss-ass golden boy whose only value to the company was
unerring unthinking loyalty. I have seen these douchebags destroy so much
infrastructure, even without a malicious shadow revenge plot.

------
kusmi
A godamn hero if there ever was one.

------
Overtonwindow
That is one pissed off IT guy.

------
Theodores
There are more constructive ways to handle workplace problems when one has
root access to everything...

If people from marketing, accounts, sales or wherever else are giving you
problems then you can sometimes automate them out of existence. This is
particularly so in ecommerce where the gig is with one of those regular small
businesses that haven't quite realised that they need to be an internet
company.

What you can do is automate all the accounting, doing it right with decent
database schemas and everything the server has to offer. The accounts can then
be a live thing and not some dead Excel export from some 90's era accounts
package. Then that guy in accounts who spends 48 hours a week to get to those
reports is redundant. He does not bother you any more and the managers are
pleased they can see their charts online whenever they want without someone
having to prepare them specially.

Meanwhile in customer service, let's automate all of those procedures and put
the process in code. Put up some forms that work and make it so easy for
everyone that the customer service team no longer need to be the size that
they are.

Then there is the non technical web team, the layer of cruft that collects
with non-web companies after the 'webmaster' in the corner of the room is
deemed to need 'help'. These people are like Chinese whisperers (no offence,
just turn of phrase, sadly...) in that they liaise between those making
content 90's style in places like the graphics/print department and then
mangle those assets for the web. Again, put them out of a job, put procedures
in place so the guys in graphics are aware of this web thing and create
content first and foremost for that. Simple things like file naming procedures
can be all that is required and the guy that collects 'assets' from one part
of the company to then upload them is no longer needed, neither are those
pointless meetings that were needed.

Have a problem with the actual graphics department and their print media
tendencies? Move the requirements on to responsive SVG instead of the legacy
PSD/PDF exported to blurry jpeg they love. Automate all that processing of
'assets' with some imagemagick. Take all the design layout out of 'design' and
into UX with everything 'aria' tagged.

Then there is HR. Put in place the tools for them to manage recruitment in a
modern online way and get rid of the need to have people copying and pasting
stuff out of Word/Excel/Email. Do that right and you can erase the need for
there to be an H.R. assistant running around doing this stuff.

Distribution and warehouse stuff is also similar, why use those in-house
legacy systems when you can go XML SOAP to the distribution center? Get rid of
the heart of the existing company - the legacy IT systems - piece by piece and
the people that go with it.

If you dislike someone and you can automate them out of existence then that is
a very good incentive to get on with that aspect of the overall project.
Having done this, the company is even more reliant on IT as there is now no
Excel spreadsheet solution.

Note that the above is reverse empire building, it is doing the right thing
and making a company a web company rather than a 90's SME built on Outlook and
Excel. Also it is a different exercising of power to going for the jugular,
'rm -fr' stuff. You also don't have to strictly automate people out of
existence, you could make their job massively easier with less repetition and
difficult sums done and have it so that no managers know that a tedious hard
job is now coastable, even making it so that managers don't have the shortcuts
that the staff do.

------
juandazapata
His grammar and writing style are bad for a person with such level of access.

------
sevenless
Not totally different from, say, Edward Snowden. Down-trodden information
technology workers becoming radicalized and committing acts of sabotage, like
the factory floor workers also did a century ago.

A society where IT workers have great power over corporate fortunes but little
political representation is going to be unstable...

~~~
tonysdg
I'd personally argue that there's quite a bit of a difference. Snowden, to my
understanding, wasn't protesting a bad performance review and perceived
neglect from upper management; he was protesting what he judged to be an
illegal action on the part of his employer.

That being said - I do agree with your sentiment that the more dependent
society becomes on technology, the greater the risk posed by those who possess
knowledge of how to operate and maintain that technology. This doesn't just
apply to human beings too - electronic actors (e.g., automated systems or even
AI) may someday if not already hold just as much power.

The question in my mind then is how to "democratize" this power, so to speak.

