
Hackers Were Inside Citrix for Five Months - feross
https://krebsonsecurity.com/2020/02/hackers-were-inside-citrix-for-five-months/
======
arminiusreturns
One thing I have learned as a sysadmin who has had the privledge to see the
inside of hundreds of companies from medium sized law firms to F500 oil
companies:

There is a lot more incompetence than you would ever want to believe, and it's
not always where you think. I've traced most of it to a failure of
connection/communication between IT departments and C-levels/boards. The
CTO/CIO and the person immediately below them (and the person immediately
below them) are the "buck stops here" people for these kinds of issues, but
often are either one of two types. 1) Too much MBA, not enough tech. 2) Too
much tech, not enough MBA.

Both tend to have pretty similar results.

~~~
BiteCode_dev
I work with a F500 oil company from time to time.

Half of the devs from there that I was in contact with were not capable of:

\- googling a solution to a problem efficiently. When they hit a wall, they
turned to me with an empty look like they were lost.

\- read an error message to troubleshoot. A stack trace is utter mystery.

\- use effectively the UI of their laptop. Some can't even Ctrl + S to save,
they look up the "save" entry in the menu.

We are talking about people writing code every day, in several programming
languages: fortran, c, c++, java, Python...

Because I'm a freelancer, I don't care. I'm paid extremely well to be very
nice to them and solve all their problems.

But I'm very glad I don't have to be held responsible for anything those
people end up putting in production. And I have no reason to believe it's
different in their security department.

However, and this is a good lesson to all of the geeks like me that think work
is about doing the right thing: the output they produce is good enough in our
society. Its cost/value hits the sweat spot. Business is not about doing
things right, it's about being profitable.

If you have one scandal a year, but it costs you less than making sure you
have a secure system, and you are not legally challenged, then you are golden.

In fact, the chances to have even one scandal are very low. Actual risks of
failure or attack are low. And consequences in case of crisis are low too.
People don't care that much about privacy, cyber-security, etc. And policy
makers won't enforce their laws anyway, at least not to any extent that will
endanger the company.

So if the software allows people to do their job IRL at a reasonable price,
under an acceptable deadline, good enough.

In fact, David Goodenough is a very funny French meme:
[https://www.youtube.com/watch?v=ho4W5LnFl6s](https://www.youtube.com/watch?v=ho4W5LnFl6s)

~~~
PopeDotNinja
I used to work with a brilliant chip designer who couldn't find the start
button on a Windows machine. We all suck at something. Personally, I think CSS
is the devil and should we should nuke it from orbit.

~~~
neltnerb
True, I have a Ph.D. and am very skilled with electronics and embedded
programming. If you handed me an iPhone I would have no idea how to read text
messages (actual situation).

Same with the people I help technically at work. They're all brilliant
scientists. They get confused by the difference between VGA, DisplayPort,
HDMI, and DVI. Or get extremely frustrated when a button on the UI moves.

I think software developers don't quite understand how big a deal it is to a
70 year old when the button to do something moves. Probably a quarter of my
day is often just figuring out how to reconfigure things to their liking or
else spend an hour retraining them because of some unnecessary UI change in
Windows 10, after which they will still forget and ask for help again.

God forbid you break apart an application into multiple programs or have
online activation or a license server. I think I hear at least a daily rant
about how you can't just _buy_ software anymore and now you can only rent it
for a bit.

We have versions of software that are 13 years old because the publisher
switched from an unlimited permanent license to a per-seat per-year license
model. Rarely worth it when the instructors get confused by new software
anyway.

~~~
PopeDotNinja
> They get confused by the difference between VGA, DisplayPort, HDMI, and DVI

Back when there were RS232 (aka 9 pin) connectors on PCs, my dad's computer
had two mail connectors, one CGA and one serial port (I think it was a serial
port, but I'm not sure, as those connectors were usually female). I took the
VGA cable and accidentally plugged it into the serial port. When I turned on
the computer, I heard the startup chirping noises, the screen was black for a
few seconds, and then white smoke started pouring out of the power supply. I
turned it off REAL fast :) Somehow the computer still worked after that.

EDIT: changed VGA to CGA

~~~
anonsivalley652
VGA is DE-15 (3 rows), DE-9 (erroneously called DB-9) is 2 rows. ;)
Interestingly, VGA only really needs 6 pins to operate: R G B VSYNC HSYNC &
GND, and monochrome only needs 4.

I don't see how that's possible without really crushing it in there. Also, CGA
& EGA were the same connector as serial (DE-9), which would've been easier to
confuse.

It could've been worse: I knew a guy in high-school who plugged a parallel
printer into a Mac classic's SCSI DB-25 (the same physical connector as a
parallel port, female on the computer; DB-25 serial is a male connector on a
PC) and baked it into "apple pie" with that "lovely" magic smoke aroma.

~~~
PopeDotNinja
Yup, you are right, it was CGA. Fixed.

------
streb-lo
> In March 2019, the Federal Bureau of Investigation (FBI) alerted Citrix they
> had reason to believe cybercriminals had gained access to the company’s
> internal network. The FBI told Citrix the hackers likely got in using a
> technique called “password spraying,” a relatively crude but remarkably
> effective attack that attempts to access a large number of employee accounts
> (usernames/email addresses) using just a handful of common passwords.

Pretty bad when the FBI has to step in and alert you that someone has brute
forced their way into your servers.

~~~
basch
Weird timing that Dec of 18 they forced a password reset to most of its
Sharefile "customers." (aka including anyone who has ever received a file from
someone through sharefile, and accidentally signed up for a service they didnt
want.)

[https://krebsonsecurity.com/2018/12/a-breach-or-just-a-
force...](https://krebsonsecurity.com/2018/12/a-breach-or-just-a-forced-
password-reset/)

“This is not in response to a breach of Citrix products or services,” wrote
spokesperson Jamie Buranich.

I want to know if they knew already in December, and if they lied to the
public and their customers. Maybe they could argue that "yes a breach
happened, but this password reset was completely unrelated" but thats a load
of livestockwash, if thats the case.

Edit: maybe i should read the article. Looks like they were in back in
October! Jamie is likely just a sacrificial lamb, who is there so they have a
head to roll, but somebody on the executive team should be in trouble for that
kind of lie, unless there were government gag orders.

>Citrix’s letter was prompted by laws in virtually all U.S. states that
require companies to notify affected consumers of any incident that
jeopardizes their personal and financial data.

Excuse my French, but thats fucking bullshit that they are just admitting to
this a year and a half later.

~~~
dublinben
As Matt Levine would say, this is probably securities fraud.

------
jens-h
On its own website, Citrix is using the real costs of a data breach as an
argument to buy its products: [https://www.citrix.com/de-de/products/citrix-
workspace/resou...](https://www.citrix.com/de-de/products/citrix-
workspace/resources/weighing-the-risks-infographic.html)

But because of the widespread use of the software, a recent Citrix
vulnerability puts 80K companies at risk: [https://www.infosecurity-
magazine.com/news/citrix-vulnerabil...](https://www.infosecurity-
magazine.com/news/citrix-vulnerability-puts-80k/)

It is always dangerous if a single company has a monopoly.

------
anonsivalley652
I worked at a big name university in the IT department for housing and dining.
Long before I got there, one of the Oracle database servers for meal-related
activities had been pwned for years because it hadn't been behind a firewall
and it had a routed public IP address. It was running Windows so it had
accumulated a number of interesting malware including obscure rootkits with no
antivirus patterns. I once booted it up off of clean media, ran some forensics
tools and found a warez dumpsite on it. This box "couldn't be down" so all the
happened to it was it was place behind a bidirectionally-restricted firewall.
It still kept limping along with funky malware because they didn't want to
spend time or money fixing it. _Sigh._ If it were my box, it would've been an
immediate disconnection, image hardening, wipe and reinstall from backups
(data-only).

I remember sending some binaries and other deets over to Mark Russinovich at
then SysInternals, who's now the CTO of Azure.

~~~
doublerabbit
Wouldn’t even do backups, you couldn’t trust them.

If a box is hacked, it’s hosed. Cast fire and rebuild start again.

------
euroclydon
How much source code was committed in that span? Time to audit it all. Plus
the binaries and other resources that are pulled off network shares at build
time. Plus the compilers...

~~~
jmiskovic
You already hinted at it, but anyone interested in computer security should
read Ken Thompson's short but majestic "Reflections on trusting trust".

------
jokoon
Well, no official government-supported agency have even stepped in to
establish a list of norms involving software security, to force large
corporations to abide by them.

While the private sector is the sole responsible for their own cyber security,
and while the NSA wants to keep the upper hand in cybersecurity by holding a
cyber-weaponry supremacy, events like these will keep happening.

Cyber chaos will continue because the NSA is obviously holding massive
advancements in cyber weapons. The day the NSA will have an adversary that can
be at least 50% as good as the NSA, you can be sure you will see cyber
security standards being passed into law.

If you think about it, having cyber supremacy is a good way to have total
power over the world. When you have all the information, you have everything
you need to do whatever you want. That sort of describes the US right now.

~~~
xp84
>list of norms involving software security

These norms are already well-known. However they are never followed 100%
because they all rest upon a sandy foundation of:

 __" Don't be clueless." __

Social engineering /phishing will always eventually work on _someone_
somewhere in a company with more than 20 employees.

Not to mention the other part, where passwords are allowed to be a vital link
in the security chain, yet software vendors like Citrix simultaneously
discourage password managers by getting in the way of using them and by
forcing perfectly good passwords to be cycled endlessly. Resulting in people
using passwords like Citrix20! (will change to Citrix21! in 90 days... iron
clad security there guys.)

------
H8crilA
Isn't this normal? I mean after you break in keeping a low profile and staying
undetected for as long as possible sounds like a no brainer to me. I wouldn't
be surprised if some APTs were inside some "worst" companies for even 2 years
at a time.

Edit: from the Wikipedia page on Advanced Persistent Threats:

> _The median "dwell-time", the time an APT attack goes undetected, differs
> widely between regions. FireEye reports the mean dwell-time for 2018 in the
> Americas is 71 days, EMEA is 177 days and APAC is 204 days.[4] This allows
> attackers a significant amount of time to go through the attack cycle,
> propagate and achieve their objective._

------
moepstar
What i find particularly embarrassing for Citrix (although only marginally
touched in the article) is the amount of time taken for them to close the hole
that was in their Netscaler/ADC components.

I mean, this is not a one-man show and an open-source project....

~~~
thedance
How do you know? There are lots of commercial products that have no staffing
at all. "1 man" might even be above the median staffing.

~~~
montalbano
Citrix had 8200 employees as of 2018 according to Wikipedia.

[https://en.wikipedia.org/wiki/Citrix_Systems](https://en.wikipedia.org/wiki/Citrix_Systems)

~~~
thedance
I meant the NetScaler project in particular.

~~~
frollo
They still have nearly 10k employees and are a big, affluent company. None of
their project should be a one-man project, but, even if it were, they should
still be able to find enough engineers to work on it in case of emergencies.

If it can be done by a small startup with a total of 3 devs in the entire
company (and I have seen it done), it can be done by a company the size of
Citrix.

~~~
thedance
I guess you'd be surprised at how easy it is for an organization to just de-
staff a launched product.

~~~
throwiay987
why are we defending gross incompetence and laid back attitudes again?

~~~
mynameisvlad
Who is defending, exactly?

thedance is pointing out that we have no idea of the inner workings of Citrix,
and how they staff their projects. It's not completely unreasonable to believe
a non-core project has minimal staffing levels.

------
noident
I was relieved to see that only internal employee information was impacted.
You don't even want to know how many banks, hospitals, and power plants rely
on Citrix Receiver for remote desktop access.

~~~
tyingq
_" I was relieved to see that only internal employee information was
impacted"_

Believing that puts a lot of credence in their analytical/forensic/security
skills. Which doesn't align well with _" inside Citrix for Five Months"_.

------
jerry1979
I have always felt very uneasy about Citrix in the workplace, especially
around PHI.

~~~
Swtrz
I know of...three major HI providers in the midwest using citrix

------
tomrod
I look forward to seeing this covered on Darknet diaries. Just found the
podcast and it rocks!

Having used Citrix this really, really doesn't surprise me.

------
amaccuish
I've always found Citrix perform way better than RDP (feels more responsive,
handles multimedia better)

I don't understand how after all these years and being originally based on the
same tech, how RDP hasn't caught up.

~~~
Spooky23
Microsoft does a mediocre job with RDP because windows pc was the cash cow.
Pretty sure that Citrix invented RDP and licensed it back.

Now they are changing, and are partnering with Citrix and VMWare in Azure.
Eventually, they’ll crush both.

~~~
LilBytes
PCoIP has been a huge shift in the landscape for remote access protocols for
this reason.

------
Stierlitz
If the hack was conducted through the use of account hijacking then why didn't
anybody at Citrix notice. Unless once the hackers got in they migrated within
the network using as yet unknown vulnerabilities. Makes one wonder of there
are backdoors in all networking equipment. So as the various state security
entities can keep an eye on us.

------
aSplash0fDerp
So instead of the old days of having a sign that says "We haven't had a
workplace accident in XX days" they need "We haven't had a security breach in
XX days".

How times change...

~~~
stef25
Not sure if there's more breaches or just more exposure because people started
caring. Reading the The Cuckoo's Egg at the moment, apparently it was common
in the 1980's for military networks to have guest/guest logins, or even ones
with no passwords at all.

------
blintz
I hope we can move to a world with ubiquitous two-factor and hardware roots of
trust (FIDO2, U2F, etc) across enterprises. That is the only way I see things
like this ending.

~~~
anonsivalley652
The core of OSes need to be treated more like read-only firmware that only
gets updated as-needed and is unable to be over-written by itself, e.g., send
a request to the BIOS to look for a valid public-key signed image file to be
applied on reboot.

Flash is so cheap, 64 GiB mirrored SSD devices should be available for
operating system images on system boards. Leave OS images as signed squashfs
files on a dumb flash FS like exFAT. Delta updates can be applied by
stripping-out entropic-metadata, patching and recompressing a previous release
to arrive at the valid signature of a complete latest release.

Mixing operating systems, configuration, programs and user data in together is
a recipe for fail.

------
PaulHoule
People still use Citrix?

~~~
westmeal
Honestly, you would be surprised how many giant companies still use Citrix.
Mostly medical. Terrifying.

~~~
dman
And financial companies

~~~
gyc
And law firms big and small.

~~~
jacquesm
And places that use RDP like setups for development because they don't want
their employees to walk out with the crown jewels on their laptops.

~~~
PaulHoule
Why not just use RDP? It is because it as hard to get an enterprise to stop
paying for a software license as it is to get one to start?

~~~
jcrawfordor
Citrix XenApp or whatever it's called this week is a lot better marketed to
enterprise and offers a lot more integrations than Windows MultiPoint/Terminal
Server. But a lot of is history, Citrix was a close partner of Microsoft and
so Citrix essentially _was_ RDP in this context for some time before Microsoft
decided to try competing on their own. Microsoft's entries have never really
caught up in terms of adoption or features - for one, Citrix supports just
about every platform there is for the receiver while Microsoft only has an
officially supported RDP client for a couple.

------
justlexi93
Networking software giant Citrix Systems says malicious hackers were inside
its networks for five months between 2018 and 2019, making off with personal
and financial data on company employees, contractors, interns, job candidates
and their dependents.

------
tinus_hn
From what I’ve heard their advice on securing servers running their software
used to be to ‘make good backups’.

I’m not surprised by an incident like this happening to cowboys.

------
s_dev
Stories like this really show Citrix and Cisco are so far behind Huawei.

If Huawei chips were similarly vulnearable the CIA and FBI would be disclosing
this as vocally as possible.

~~~
vb6sp6
The FBI or CIA would keep any vulnerabilities secret. But here is a recent
example: [https://www.businessinsider.com/us-accuses-huawei-of-
spying-...](https://www.businessinsider.com/us-accuses-huawei-of-spying-
through-law-enforcement-backdoors-2020-2)

