
When Employees Use Software That IT Hasn’t Approved - r0n0j0y
https://hbr.org/2019/06/when-employees-are-using-software-that-it-hasnt-approved
======
S_A_P
I see this a lot in consulting. When a new CIO (or CEO or other C level)
arrives, they want to make their mark with a digital transformation intiative.
This usually just means that the new C level employee is coming into a medium
to large business and would like to add a bullet point to their resume and get
that new shiny object everyone is talking about. Tableau, Salesforce, Data
lakes, blockchain, ERP, Identity Management and "cloud" projects are often the
result. It seems to also stem from the new C level employee having a close
relationship with a sales rep/partner/C level employee at the vendor. Left a
project a couple years ago that had Hadoop interfaces from every system. The
user count of all this data? exactly 0.

One somewhat disturbing trend I've seen at some of the largest corporations-
cut/outsource IT support staff to near egregiously low levels to "save money".
At the same time kick off 7-9 figure ERP/consulting projects that at best
provide fractional value to the organization.

Of course there are counterpoints to this. One of Houston's major pipeline
operators pulled off a digital transformation and actually ended up with well
designed, highly integrated and easily maintained systems. It took about 5-7
years and had a few reboots, but it eventually landed. That brings me to my
final point. These projects often have a timeline that is divorced from
reality. Whatever time frame you think a major IT project will take. Double
it. twice, then add 50% and you are close. It also seems that C level folks
are hesitant to hire boutique/small shops that have industry experience and
years of experience in favor of big consulting. Nobody every gets fired for
hiring Accenture/Deloitte/PwC. What usually happens in the non trivial niches
is that these big shops sleeve the boutiques through them to get things
done...

~~~
snowwrestler
We've got a Salesforce implementation going at the nonprofit where I work.
While there was some debate about which big CRM we'd buy, the need to
consolidate was blindingly obvious.

Why? Because our organization has been quite forward thinking about allowing
managers and executives to source the technology they think they to succeed.
As this article advocates for, IT was largely consultative rather than
dictatorial, and a lot of business units were able to pick what they wanted.

But what this has left us with is dozens of places where customer data was
being stored, some of them now past their end of life. No central visibility
into customer experience. People getting multiple copies of the same email
from different departments using different email platforms. Poor
deliverability. Subscriptions on random credit cards that suddenly turn off
because the person left and no one knows how to get into the admin account and
update the card.

We hired a boutique shop to do the Salesforce implementation; we're not scared
of doing that. Unfortunately this time it did not pay off... their performance
fell off, to the point that they couldn't even reply to emails on time. As
sometimes happens with small firms, they grew too fast and exceeded their
ability to operate. We can't wait for them to figure it out... so here we go
with a big dog firm. Let's see how that goes.

Maybe I'm lucky in who I work with, but I find the "add a bullet point to the
resume" take to be maybe a bit too cynical. Tableau, Salesforce, data lakes,
ERP, identity management, and "cloud" infrastructure each seem like useful
tools if implemented smartly. (Note that I took out blockchain...)

~~~
ownagefool
Your problem statement does make it sound like you need a CRM, but I do wonder
why is has to be a big CRM with a big consultancy, and why IT aren't
delivering it?

Who's going to run the thing afterwards? Will the bigdogs deliver something
that you can maintain, or is that generally against their own interests?

Finally, who's gonna secure all this customer data? Are they taking that on as
part of their remit? They rarely do.

~~~
DEADBEEFC0FFEE
I expect the significant discounts offered to NFPs has some bearing on this
decision.

Having been involved in IT management of NFPs, the low price is a significant
draw, and the total lack of internal skills is rarely able to counter this.

If you don't have some sort of architecture function, technical risk
management, application management, and data management these projects simply
won't deliver value.

~~~
ownagefool
Fair enough. Do they usually complete on the cheap, or do they turn the screws
after delivery?

------
maxxxxx
This is exactly extremely common. In my company there is this constant battle
about the devs having admin rights on their machines. We need admin rights to
do our job. We have had dozens of meetings explaining the situation but IT
can’t come up with a solution so the devs go around security because they have
no alternative if they want to finish their work . Same with Dropbox. They
block it but we have suppliers who use Dropbox. So the result is that people
download confidential files from Dropbox on their home computers or phones and
transfer them to their work machines.

In my view security shouldn’t be isolated at corporate headquarters but they
should be close to the end users so they see what users need to do and help
them to balance security with getting things done. They can’t just block stuff
without providing alternatives or they will either hurt the business or they
will be circumvented.

~~~
rb808
I think most of the reasons for admin rights are no longer valid. Its easy to
change user environment variables and lots of applications can be installed as
a user. Why would you need admin rights?

Dropbox/googledrive is a huge security hole that is definitely blocked at most
companies I work at.

~~~
maxxxxx
This is about Windows desktop development. I need admin rights to install sql
server, I need them to customize my machine so it’s similar to our target
environment. I need to change user permissions all the time yo see how things
behave under different conditions . There is a ton more I could walk you
through and have done multiple times. Comments like yours come repeatedly from
people who don’t know about the work we do. I have offered them to demonstrate
doing our job without admin rights but so far nobody has even tried. They just
keep sending the same email about not needing admin rights which has
repeatedly been showed to not work.

~~~
cmdkeen
Being punchy about it you've never moved on from thinking you need admin
rights on your machine. Chocolatey for Business' self-service installer, SCCM
jobs, and a variety of other tools exist to enable you to get _specific_
things that require elevation executed. If you're changing things to test
various configurations wouldn't it be handy to have those scripted, get them
peer reviewed / linted and you've got yourself the start of a process to get
that script executed on demand.

This stuff isn't that hard - but those of us doing it see the mad things that
people do when they're given blanket, even time bound, admin access. They're
the ones dealing with the support calls when then every SQL Server
installation has been done differently with no details of what specifically
was done. IaC works.

~~~
novok
Then I hit another 'no-admin' roadblock, that requires a day or weeks of
hostile IT bureaucracy and the IT department has just wasted another +$3000 of
employee time. This behavior might drive them to quit, leading to a premature
+$30k recruiting and on ramping cost to replace them.

Now iterate that over 1000s of other instances and you see the financial
reason why devs need admin.

------
brixon
Let's ignore the SaaS security issues for a second. When IT says "No" it's not
like the area asking is going to go away and not try to solve their problem.
Organizations are going to find ways to solve their issues and IT can either
help from the beginning or help clean up the mess later. I try to take the
stance of offering the right solution and a lot of the times a now solution at
the same time. There is no saying "No" in the long term, either help them now
or get stuck with the shadow solution the magic macro guy cobbled together
that became a critical business function.

~~~
x38iq84n
I have seen IT being unaware of and unwilling to meet requirements of highly
specialized technical teams, such as network engineering. You cannot have a
TELNET client because the use of TELNET is prohibited by corporate policy,
test TCP connections another way. You don't need vim when you have vi. You
can't have admin rights but we don't support drivers for RS232 dongle so nope.
Sometimes it's quite a challenge to get some work done.

~~~
user5994461
Use netcat.

Telnet is pretty hard to procure since it's not included by default since
Windows 7.

~~~
beatgammit
It ships with Linux...

------
drevil-v2
I still shudder thinking about my time working as a developer on corporate IT
locked down IBM leased laptops. Every time I did npm install I needed to
request admin access to Windows which took 2-3 hours to action by IBM team
sitting on the other side of the world in India.

One day a grey beard took pity on me and installed a Linux VM where I was
admin, copied the security certs from the Windows host and I could access all
corporate resources at my leisure. Never logged a single IT Helpdesk ticket
after that.

~~~
flurdy
Yup, back then my way around corporate locked down Windows machines was to
only get permission to install VMware Workstation or Virtualbox.

Then fullscreen Ubuntu in a VM from then on. Slower but no restrictions. Even
better when Workstation supported multiple screens.

~~~
Macha
Honestly on my work machines, though we have root, the difference in FS
performance tilts in favour of VMs/containers due to the slow endpoint
protection affecting native FS access, and for a lot of tasks we do that
outweighs any virtualisation overhead.

------
rsuelzer
Our IT security department was incentivized to deny everything from new tools
to new internal applications.

We had an outside firm making security decisions and if there were any
security issues it would end up being on them. So as long as they did not
allow us to release any products and or install any software they could not be
held responsible.

I made friends with a lower level contractor who told me off the record to use
my judgement on what to install to get the job done, because the security
department would never approve anything new unless directly instructed to by
the CEO.

Fast forward three months, there was a major security flaw on our website
(also built with outsourced labor) which allowed anyone to access private data
without a login.

A few of us had reported to the security department that the code running the
website was so poorly written that the odds of being insecure were close to
100 percent. We suggested upgrading the website and rewriting the code, and
management was on board with this but security department refused to allow us
to use any new frameworks since they were not approved. Of course in a matter
of a few months the site was hacked and millions were spent as a result.

I quit this job after we were unable to release several products after a year
even though we jumped through every hoop we needed to. That department killed
all innovation.

~~~
closetohome
I think it really depends on the company. If you're something like a
nontechnical non-profit, sure, turn that decision making over to IT. In that
case IT is performing a vital, skilled function.

But in most software shops, the workers are probably more qualified than the
IT department to be making decisions about what applications to use, and what
kind of security they need. IT is just there to make things run and fix them
when they break. They don't really need to offer guidance.

------
jmkd
Joining Google was an eye-opener for me on this. Was the first time I
encountered an IT department (TechStop) that didn't act like a police force
and instead had your back, helping you get where you needed to be. Was always
the first thing I would show guests on a tour of the campus.

~~~
repolfx
But you're probably a developer?

TechStop is/was great. But Windows users had locked down workstations where IT
whitelisted binaries. I assume the approval process sucked about as much as
normal.

Many Google employees use desktop Linux which is basically unheard of outside
the tech world. That by itself simplifies things quite a bit. Not many people
writing viruses posing as screensavers for Google's in house Linux strain.
Anyone who cracks that is probably an APT attacker and those require different
approaches.

------
protomyth
Its the user who downloaded a program they "needed" which had malware which
sent out a lot of spam email because this was a user that did announcements
which basically got an e-mail server listed on blacklists that creates these
IT policies.

You want to treat people like responsible adults, but they aren't the ones who
have to deal with the fallout. Developers know the score for the most part, so
full privileges are expected with the caveat, if it all goes bad, we are
wiping the machine[1], not doing a recovery.

IT dreads the moment we are called to account for something some user decided
they needed to do.

1) most developers understand backup tools and code control - those that
don't, well...... with great power comes great responsibility

~~~
dx87
Yep, a company I worked at hired a tech writer that downloaded some cracked
version of software that included ransomware on their first day of work
because they said they didn't want to wait for the company to get them a
legitimate copy.

~~~
toyg
Well, that used to be common practice... in the '90s.

Thank $deity for the rise of opensource.

~~~
protomyth
OpenSource still isn't a force in that particular area. Its Microsoft or Adobe
there.

~~~
toyg
Yeah, what I meant is that, these days, the culture is such that one assumes
there will be an OSS tool somewhere, before one even considers a sketchy
binary. Maybe the OSS option will be inferior, but it's almost guaranteed that
it will get some stuff done and not nuke your machine. That's a significant
improvement (of course we know that having a github repo is no guarantee and
blablabla, but it correlates well enough for most purposes).

------
tyingq
_" Soon enough the CIO sniffed out the project and called her in to a
disciplinary council."_

Somebody has apparently lost touch with who the customer for IT is.

~~~
thisisnico
To be honest, I find it odd when you treat it as if everyone else that you
work with is a customer. I don't believe in this philosophy. The business is
my customer. The business is what IT is trying to protect. If you have
individuals that are not following policies, they would be disciplined like HR
would discipline for not following policies. It's all in place to protect the
business and what's best for the business. Sure you'd like admin rights to
your own machine, that will help you individually, but will it help the
business as a whole if we get hit with cryptowall again?

~~~
scarface74
I find most “IT security policies” that hamper developers to be mostly
security theatre. No matter how many policies they put in place, since they
aren’t developers, one junior developer can write:

    
    
      var sql = “select * from Customer where firstname = ‘“ + firstname + “‘“;
    

And thwart all of your security “best practices.”

I was the lead dev at a medium size non tech company, and the hoops I had to
go through to get anything done dealing with the “security team” was
ridiculous and of course I didn’t have access to production to troubleshoot
for awhile.

I had ultimate control of all the code that did go through the process. If I
were to do something stupid or purposefully malicious, while I didn’t have
access to the environment - my code did.

As far as someone mistakingly installing a “crypto wall”, if a user can
download a program that doesn’t require admin access, that program has access
to the user’s files. The system can be restored much easier than the user’s
data.

~~~
alistairSH
_I find most “IT security policies” that hamper developers to be mostly
security theatre. No matter how many policies they put in place, since they
aren’t developers, one junior developer can write..._

IT policies at large corporations aren't implemented for developers (only).
They're implemented for everybody. For every developer, there is a
salesperson, admin, manager, or HRBP who will do things they might not fully
understand to be "bad".

I came into the industry in the late-90s and still remember the chaos that the
ILOVEYOU and Anna Kournikova style viruses caused in corporate offices. Non-
technical users didn't know that Windows hid file extensions by default. They
didn't think that opening a picture could start a shitstorm the brought the
corporate network to its knees. Fun times.

~~~
scarface74
Yes. I remember ILOVEYOU too. It also confirms my point.

\- It spread by reading the person’s contact information which doesn’t require
administrator access.

\- It also corrupted the user’s files and didn’t require administrator access
for that either.

------
alkonaut
Security training focuses way to much on email phishing and not enough on this
kind of stuff. Actually getting your work done, managing your own computer. Of
course people can't be trusted if they havem't been trained. How to handle USB
drives. What and from where you can download and run programs. What actually
IS a program and what isn't. Many of us learned this the hard way by playing
lots of cracked games in the 90s. But not everyone did that.

Try explaining to a non-technical person how how a desktop background image
isn't a program so it's basically safe to grab from anywhere, while a screen
saver is _definitely_ a program and usually unsafe to get from most places,
and a word document is _some times_ a program that might eat your computer.
Training could involve things like "which of these 5 webbpages would you
consider it safe to download and run executable from"?

Having too cumbersome rules around security just means it's ignored or
circumvented, increasing risks.

~~~
JakeTheAndroid
It's tough, I give security awareness trainings myself and I completely agree
with what you're saying. However, that's a lot of information to give to a
group of new employees that can span any department and technical
understanding.

I actually was talking today with a customer during a logical assessment about
if I talked about downloading malware in the training. I dedicate an entire
section to downloading documents, but I don't really give people the
information you're talking about. I tell them how to avoid ever having to
download anything, and if they _must_ do it, how to try and do it properly.
All of this is ended with the process on how to report incidents because
eventually something bad will happen.

As a company you kind of expect this to be solved at a number of layers.
Endpoint management should hopefully help resolve this issue. Restricting web
access where it makes sense can help. Sec Awareness Training helps keep people
aware. Etc, etc, etc. You hope your controls are what save you from incidents,
because there is no way you can effectively train your entire company on
security topics to a degree that they can make good, security conscious
decisions. That said, many of these SAT's are really just checking compliance
requirements, because thats the real need. I put my own training together
starting with what I know needs to be covered for compliance (pii handling,
passwords, acceptable use policy, common threats, security incident response
reporting, etc). Anything else that makes it in is purely because I have extra
time and I know it to be important.

~~~
alkonaut
Concrete example happened just this morning: I needed some documentation that
exists on archive.is, but has been taken down from the original site. I
navigate to the cached content on archive is, and archive.is is DNS blocked
when going through my VPN by Cisco Umbrella because apparently it's an
"anonymizer" service.

So I change my DNS settings to use an 8.8.8.8 dns first, and my company dns
second. Now I can access both archive.is and sites on the company network.
Excellent. But in doing this I circumvented all the DNS filtering, not just
for this site. The reasonable thing would have been a warning like a https-
style warning "Are you sure you want to continue to this site"? Or a way of
whitelisting, perhaps temporarily, a single address. Instead my options were
to ask an administrator or disable the whole security feature entirely. (Or
connect/disconnect the VPN temporarily every time I needed something
blacklisted, but that didn't feel like a good solution).

------
jmspring
This brings back a memory. The only time I was fired "for cause". Summer after
my freshman year at college, I was temping and got an assignment doing real
estate purchase comps with a company in the East Bay. At the time, there were
laser printers, but often printing sucked up CPU time and let's just say
multitasking was still not a widespread thing.

I found myself tired of sitting around. I found a TSR / print spooler that
would use RAM and offload the process of printing. This allowed me to keep
working. My productivity (as a temp) was higher than many others including the
person I was "reporting to" at the company.

They found the print spooler, labeled it "unapproved software", and I was
walked out the door.

The funny thing is, a friend at the time (and I didn't realize it) was higher
up in the management. He reached out to me on a multi-line BBS that was
popular in the area and offered me a full time job a few days later. I was in
school and obviously declined.

Working the rest of the summer for a Chemical Engineer in Martinez/Benecia
ended up being incredibly more interesting. So it was a net win.

------
jrjarrett
This thread hits home. I switched jobs a few years ago because the IT policies
on workstations were being ratcheted down to make my job as a developer
difficult to impossible.

Now, the company I work for, ostensibly a _software_ company, got its ISO
certifications, which meant policies and procedures that make developing hard
or impossible again.

How does a software business _successfully_ implement stringent access
controls while still allowing for efficient software development? I'd like to
see/hear what works.

~~~
CrossWired
I'm heading down this path right now. How do I obtain my certs while also
allowing enough freedom for the dev teams to operate.

We have to deal with the fallout when they screw something up, there has to be
a happy medium somewhere.

~~~
user5994461
What's the issue specifically?

Developers don't need admin rights for much of anything in this decade. No
need to bother with that.

Common software has to be made available in self-service, so developers can
install development tools like notepad++ or visual studio.

Deployment is usually the challenge because you have to store binaries
somewhere, copy it to some random servers and finally execute it, each step
causing numerous security headaches, so there has to be some approved tooling
to handle that.

~~~
stefan_
You can't get past "docker run" without admin rights. No, the challenge is
access to production data without auditing.

------
exabrial
IT is a service to the rest of the company. If you don't approach it with a
servant's heart, people will go find their own solutions without you and
you'll be part of the cleanup crew.

------
TheRealDunkirk
I just witnessed a very similar situation, on a smaller scale, but there are
many of these in my company, and they add up.

Boss: "We need access to the database of our primary application that you
wrote for us so that we can pull the data into this new tool to track
progress."

IT: "No. Not only can we not give you access to YOUR data in YOUR application
that we wrote on YOUR dime, we will not allow you to have this new application
written by someone who isn't in our group. If you wanted something like this,
why didn't you just ask us? We would have written this for you."

Boss: "We had a meeting about this over a year and a half ago, and you told me
that you didn't even have the time to discuss it further."

IT: "... Well, we're still not going to let you do this."

IT is effectively holding the rest of the company hostage, and the corporate
technical debt is becoming epic. So skunkworks solutions will continue to be
developed.

------
noonespecial
Its the same dilemma companies face with their legal team. The "safest" thing
to is nothing at all so sometimes the overabundance of caution hamstrings
business growth.

That's what the CEO is there to figure out.

------
tvanantwerp
The ideal IT team is one that proactively learns the needs of others in the
business and works with them to solve problems. It's no wonder that so many
companies end up with shadow IT when so many IT teams are just people who tell
you "no" whenever you ask for something. Doing it right is harder in the near-
term, but much easier in the long-term as you're not putting out so many fires
or going to "disciplinary council" meetings.

------
geekamongus
It sounds like the author recommends embracing the Agile philosophy of letting
your teams choose their tools, then working with IT/Sec to make sure
implementation is sound. I like that philosophy.

~~~
FooHentai
Great until you have five teams, each having chosen a different tool, and now
you're wondering why the IT support costs are out of control.

Still possible to support but requires a different model e.g. one where IT
delivers a new, unconfigured workstation to your new team member and it's up
to them to build it. If it breaks, their loss of productivity is their problem
and not ITs.

Authority for something (e.g. software selection) must go in conjunction for
responsibly for consequences arising. Those things must always move in
lockstep to avoid perverse outcomes.

~~~
closeparen
Our IT does not support development environments. Just network, backups,
printing, client certs. Dev tools support is all essentially peer to peer and
ad hoc, with escalation to the internal owner of the tool (another engineer)
sometimes possible. If you mess up in an unrecoverable way, IT will give you a
loaner to work on while they reimage your machine. It works fine.

------
thisisit
As a born and bread corporate (mostly banks) corporate IT guy, I used to frown
upon this behavior. Then I got one of my bigger career breaks because the
finance team went behind IT, bought a software and installed it in a machine
which kept under their desk. They further hired people from an IT service
company to configure the machine.

The configuration was so bad that it exposed the company's network to whole
wide world. Google contacted the company and after searching high and low IT
security managed to track the pc down and take it away. Finance team promised
to hire someone with skillset required to run the software in a closed
environment. And that's how I ended up getting my job.

~~~
lapnitnelav
Really interesting story, especially if you put it in the context of being
corporate world and a finance team, both of which aren't the type to rock the
boat.

It says a lot about the struggles of enabling change in structures with strong
silos that (what are probably among the most risk averse type of) people would
go to such lengths.

------
ConfusedDog
This is my current situation. Being a SWE, IT and security are always putting
out fires with networks or upper echelon cybersecurity violation complaints
(mostly people downloading software without authorization). They have very
little time, almost none for investigating new software, and all software must
be installed by them. End of the day, nothing gets done on our work computers.
I once waited two months for them to say no for a piece of solution we as the
team approved. It's absolutely frustrating.

------
burfog
I've been amused by VMWare being on the strictly-enforced official software
list, and the VM being considered data. Nothing in the VM counts as software!
It's not even being sneaky. Official policy is that the VM is data.

------
davvolun
> The CIO admitted that he had been approached and explained that he had
> informed the VP that IT already had a project with SAP to deliver what the
> VP needed. “Yes, but that won’t be ready for me to use for three years, and
> I need something today,” retorted the VP. The CIO was silent. Then the CEO
> asked the VP, “I’ve known you for ten years. You don’t seem like someone who
> would do something to harm the company. Why did you do this?” The VP hit
> right back: “Since I started this digital customer acquisition program,
> we’ve increased revenue $1M per month. Before we were losing revenue. If you
> want, I can shut it down right now. What do you want me to do?”

Maybe not for this particular project, but another interpretation of that is
"who cares about security if we're making money" which is a very dangerous
argument as well.

~~~
yebyen
When a person says "we need this infrastructure project" and a project is
commissioned, acknowledging the need, it is in my experience that
unfortunately that person's job function is rarely placed on hold until the
appropriate infrastructure has been made available.

"Who cares about your pie-in-the-sky infrastructure project, my boss continues
to measure our real performance with basic accounting, and is expecting to be
able to report on growth each quarter, which I can't help without tools" seems
to be a bit closer to the argument posed here, IMHO.

~~~
davvolun
Of course that's the real question with the actual story in the actual article
-- could they have implemented their security on the temporary infrastructure,
or "good-enough" security, if the CIO knew about it?

In that story, is the blame on the VP for going ahead instead of getting
dialogue started between CEO, VP and CIO? Is it on the CIO for just saying
"no" instead of recognizing the need and the value? Is it on the CEO for
failing to empower the VP and CIO to get that conversation started themselves?

And then, it's all well and good to worry about the bottom line first, until
you're sitting in Equifax's shoes right.

~~~
yebyen
I've been the one in the story who hears "no" enough times myself that I know
which side I'm naturally going to fall on. But I've never been the one that
did an Equifax, and now has to explain themselves to the board, so there's
also that.

I've seen it said well in another comment on this post, I feel like could be
said about more than a handful of orgs:

> The problem is, I write up a proposal identifying the risks associated with
> the exemption, along with minimum and recommended compensating controls.
> This then gets discussed among IT Management, where it is usually decided
> it's too much overhead, and to just deny the request or if the user can
> scream loud enough, allow it outright and get some director to sign
> something. The third oft-used response is ignore the problem and hope the
> user finds their own work around so we can get back to the 13 projects we're
> somehow expected to complete this quarter.

> ignore the problem and hope the user finds their own work around

> _ignore the problem and hope the user finds their own work around_

If this is even remotely the story of what happened, you can't really be
surprised when the user went off and did their own thing. If they came to you
with a specific priority business problem and an expectation of your support
to solve it with a sense of necessary due urgency, and your answer is returned
in the format of a 5 year plan... I don't think you can really act surprised
in fairness when they end-around you and solve the problem somehow else,
anyway.

If it means standing on a mountain of chairs for them to do so then I guess
there'd have to be shared culpability. So how do we make sure that it never
looks attractive to build that mountain of chairs?

I wish I knew more about the "digital customer acquisition program." The story
makes it sound like this "VP for a declining line of business" honestly was
not going to make it another 3.5 years without some help.

I struggle with this myself, when it seems like we could go ahead and solve a
problem for like $80/mo, but instead we're going to study the problem and
spend $20-40k out of peoples' salaries on coming up with a recommendation for
an even more expensive project that can only be justified as necessary in
order to avoid this other, cheaper tool we could have used.

There's obviously some mismatch when on one hand there's a major project with
a vendor like SAP in the picture, but on the other hand there are basic needs
that aren't being met, to the point where someone is going to set up "shadow-
IT" on a personal credit card just to keep the basic business of the company
moving in the right direction.

------
fphhotchips
How many years since _The Phoenix Project_ and this conversation has barely
moved an inch?

CIO probably wins this battle and gets the VP fired, but will be mystified
when they're reporting to the CFO or a Chief Digital Officer when it happens 3
more times by the end of the year.

~~~
finnthehuman
>How many years since The Phoenix Project and this conversation has barely
moved an inch?

People are still fighting the lessons in The Mythical Man Month; it's gonna be
a while.

------
analog31
Something that's crossed my mind is John Gall's observation that complex
systems operate in failure mode 100% of the time. I understand "failure mode"
to mean that built-in guards have been bypassed in order to enable the system
to do _anything at all_. Germane to this thread, the "guards" are IT
approvals.

I suspect that if a business is complex enough to have IT policy, that policy
is always being bypassed in some way, at any given time. Somebody is using
unofficial software, or using official software in an unofficial way.

------
eithed
To me it reads like this - VP didn't care about the consequences of utilizing
their solution and didn't care about IT; they simply wanted their stuff done,
without acknowedgling prioratization of tasks.

The proper way this could have been resolved is by VP utilizing people's
skills they've hired. Does this solution look good and will accomplish the
task that was prioritized? Excellent! Pass it to IT to evaluate. If the task
has specification - excellent, have somebody in IT look for a product that
ticks all the boxes and let's choose it together.

------
vinay_ys
Does your IT team use key loggers or other employee monitoring software at
your company? I hear some big trillion dollar companies do this. Is that true?

------
thomasjudge
I work in an IT organization & I see (in the sense of witness) both sides of
this. We are over-tasked and under-resourced and new
projects/ideas/initiatives that come in the door go into a backlog of
requests. So I see business/end users signing up on their own for SAAS
solutions to solve their problems.

~~~
argd678
Right, the CIO is also being held to a lower standard than a P&L. The CIO
could have planed out an interim solution to meet the business needs quicker.

If the tables were turned, say the CIO needed to deliver a service and didn’t
have a big enough budget, then what?

------
raxxorrax
> If you don’t think this is happening in your organization, think again

That story probably never happened anyway. But the essence of the article is
very true. I never have been in a corp where IT enforces 100% conformity
anyway (apart from medical industry).

Sure, there are actual successful attacks, but that is mostly not the fault of
unsanctioned programs.

But there are systems where people should not just start to use any system,
because information gets lost on the way. That would include CRM and ERP in my
opinion. That a company can exist without a CRM is questionable to begin with
and solutions are plentiful. If they did not have anything like that...

If the story were true, it would not be the fault of Chief Input/Output.

~~~
cannonedhamster
I've been in corporate IT where this happened. All company apps were built
internally. None were able to run on anything past Windows XP. On top of my
regular help desk, asset management, software project, and lease refresh
program I was also somehow supposed to make the software work with Windows 7
as they had let the developers go. This is the same company that refused my
sane security requirements and ignored just about everything until too late. I
hear they have since outsourced IT and networking and it's failing
dramatically, but they are saving money right?

------
sgt101
Well, that cleared that up then! Gosh I had no idea that the solution would be
so simple.

It does shock me that the people who've had their whole infrastructure
compromised and held to ransom by viruses and the people who've been held over
a barrel by suppliers or had vast amounts of money burned by being locked into
a dozen vendor contracts for the same service are so silly and hysterical
about it when the solution is as simple as "identify when you need to be best
in class and stay small everywhere else".

------
la_barba
Hehe, if you think this is nuts, come to pharma. We can't do jack shit with
our machines. If you so much as change the time on your machine, that is a
'data integrity breach', and if your actions are determined to be malicious it
can result in a firing.

~~~
GuB-42
To be honest, changing the time on a machine is a very serious concern.
Accurate timekeeping is crucial in security, that's how you connect events
together.

~~~
la_barba
Well, all the rigid policies like no dropbox or no FTP or no whatever, also
arise from serious concerns. I just wanted to point out another seemingly
innocuous one. Most of our equipment is not internet connected, and we need to
manually change the time for daylight savings or other corrections. We have a
company policy and procedure to do that periodically so that our audit trails
are accurate. Sometimes folks get busy and the shop floor guys take matters
into their own hands.

------
jccalhoun
I just wish I had a decent computer at work that didn't have 3rd party
antivirus that would just slow the software to a crawl.

------
hartator
Isn't IT something from the past? I would expect people knowing how to use a
computer and what they need to do their job.

~~~
kleborp
As a lowly help desk technician perusing this thread, you couldn't be any
further from the truth. The software devs never open tickets, everyone else
does and its for the most banal problems

~~~
rhinoceraptor
I've opened plenty of tickets as a developer, mostly the tickets are IT's
garbage software or horrible network setups preventing me from doing my work.

------
noja
There's a difference between using software that IT has approved and shipping
customer data outside of the company.

------
adwww
how about trust your staff.

~~~
justinclift
Maybe, "empower your staff as they each prove competence", instead?

------
dingo_bat
In our company people just started using free slack en mass, boycotting the
horrible IT approved Skype for business. When it was discovered that thousands
of employees were using slack, the CTO had to step in and tell IT to fuck off,
and started paying for the full version.

------
philipodonnell
> The CIO admitted that he had been approached and explained that he had
> informed the VP that IT already had a project with SAP to deliver what the
> VP needed. “Yes, but that won’t be ready for me to use for three years, and
> I need something today,” retorted the VP. The CIO was silent. Then the CEO
> asked the VP, “I’ve known you for ten years. You don’t seem like someone who
> would do something to harm the company. Why did you do this?” The VP hit
> right back: “Since I started this digital customer acquisition program,
> we’ve increased revenue $1M per month. Before we were losing revenue. If you
> want, I can shut it down right now. What do you want me to do?”

Shut it down right now and ask the VP to tender their resignation. Any company
doing a 3-year SAP implementation is a very large company. That $1M in
additional revenue pales in comparison to the risk introduced by sharing
company or personal customer data with a vendor who has not passed the
required security auditing. Data is no longer a thing to be thrown around in
search of additional revenue and "but I made money" or "I had to because IT is
slow" is not a post hoc rationalization for the behavior.

Regardless of the merits of large enterprises acting this way, this is a VP
who clearly cannot function within the enhanced risk-controlled environment of
one and should find a position with a smaller company where they have more
freedom to pursue personal initiatives at the VP-level. Those companies exist.
Go find one.

~~~
enraged_camel
>>Any company doing a 3-year SAP implementation is a very large company.

Not at all. I have a client with ~100 employees who are past year 2 of their
Salesforce implementation because the director of technology keeps changing
priorities and project requirements.

~~~
philipodonnell
That's fair, but just using the time-frames in the article, 6 months had
passed before the VP got caught and still had 3 years left on the
implementation. I think a director of technology who take 3.5+ years to do a
CRM implementation at a 100-person company... isn't doing a very good job. :-)

