

Security Incident on FreeBSD Infrastructure - dous
http://www.freebsd.org/news/2012-compromise.html

======
Zenst
This is how you tell people about a security breach. Inform them soon as you
know with what you know and assume the worst with your appraoch to restoring
things.

Much respect and defineing the word professional for many.

------
meaty
I have a couple of FreeBSD machines which pulled binary packages between those
dates. I'm not overly worried. The packages have been removed and installed
again from ports after a fresh portsnap dump and the systems have been
verified with "freebsd-update IDS" against known good signatures. Any modified
files were manually checked. I use MAC on each machine and pf up front on
firewalls so I know what is going in and out as well.

The fact that these mechanisms are available is the reason I use such a
system.

Also, if you consider any problems like this happening to a closed source
vendor, you may never know it's happened. And don't tell me they don't do it
as I've worked for a couple of companies that felt that burying security fuck
ups was acceptable practice. It's why I don't work for them any more.

~~~
zobzu
Actually, even with opensource you may never know it happened. Does not matter
where it comes from:

\- They have to find it out first.

\- Then they've to be willing to disclose the incident

\- Even if, you still trust the source of the packages, the developers, etc.
There's a zillion bugs that look like "just an error" which can also be "just
a backdoor".

For these reasons, running mac, checking modified files, etc is ALWAYS good
practice (that you seem to follow, don't get me wrong - but that's pretty
rare)

~~~
meaty
I only follow them after I got owned in '97 on the end of a dialup for running
an open telnet server (on FreeBSD) with a crappy root password :)

~~~
olefoo
Nobody takes security seriously until they've learned to distrust a computer
they know intimately. It's a fact of life.

------
lifeguard
Take note:

"We unfortunately cannot guarantee the integrity of any packages available for
installation between 19th September 2012 and 11th November 2012, or of any
ports compiled from trees obtained via any means other than through
svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest
any tampering took place and believe such interference is unlikely, we have to
recommend you consider reinstalling any machine from scratch, using trusted
sources."

------
lhm
I'm a bit suprised that the affected machines were powered off instead of just
disconnected. Would that not make an audit more complicated?

~~~
hendi_
Yes, it's more complicated than just SSH'ing into the server.

But on a compromised machine you can't trust anybody, not even the kernel.
Assuming the worst, the attacker could have gained root privileges and
modified the kernel or the base tools like ls and grep. You also can't trust
the log files if they're not stored off-site. The modified kernel or ls could
hide the attacker's traces from you.

Thus, the only possibility to really make sure nothing is hidden from you is
to (power off the machine and) attach its hard disks to a trusted computer
where they're mounted and investigated.

~~~
rdl
Live forensics are preferred to just reading disk; lots of problems with that
and solutions to those problems on various types of machine.

~~~
hendi_
Could you please elaborate on that? How do you do "trusted" live forensics on
systems with possibly infected kernels and stuff? Assuming these servers were
normal COTS and nothing fancy (thinking of CPU-bypassing memory access...)

~~~
jevinskie
DMA memory out using FireWire (if available)? That would be my approach!

~~~
rdl
Firewire is awesome for the attacker, unfortunately few servers have it,
especially not externally exposed ports. Also, smart OSes use some of the
newer Intel features (VT-d) to lock down DMA while the OS is running, which
usually protects from rogue firewire, and can theoretically help against rogue
PCIe, although usually badly implemented in chipset and OS.

Another option is a reboot onto a custom OS which is designed specifically to
preserve memory (you get a safe few seconds of holdover). LiveKd is pretty
cool (sysinternals)

There are PCIe cards which do processor/network and let you explore main
memory -- WindowsSCOPE CaptureGUARD for PCIe or ExpressCard. Probably enough
time to pop the case open and throw one in before memory degrades.

Countermeasures are numerous -- everything from doing memory encryption inside
the CPU die (putting code in the cache, like TRESOR) and doing hypervisor
tricks ("TresorVisor") (<http://www1.informatik.uni-erlangen.de/tresor>) to
using Hardware Security Modules (like the SafeNet or Thales nCipher) to just
keeping your servers physically secured from intruders who might memory-
analyze them (although a software bootloader and remote-reboot could still be
applied).

Forensics as a field seems to be a lot more interested in attacking mobile
phones (which is one of the things I'm talking about at RSA 2013), but
desktops and servers are still interesting targets.

------
darkf
FreeBSD reports are always extremely professional, I love it.

------
0x0
Interesting choice that some machines will not be reinstalled, only
"thoroughly audited".

~~~
cperciva
There are some systems (generally speaking, ones which were installed in the
past few weeks) for which we know exactly what files should be installed and
what their SHA256 hashes are. Thoroughly audited means "every single bit is
correct".

~~~
0x0
Thanks for clarifying.

I assume if they had root though, they could theoretically install a rootkit
in the MBR. Did you SHA256 verify the MBR too? :)

~~~
cperciva
I didn't (I'm not involved in cluster management) but I suspect that it was
done by someone.

------
chmike
Excuse the naive question, but how does one detect intrusion when using bi-key
authentication ?

------
ladzoppelin
How does one know the offsite repository's are clean if svnsync runs at set
intervals? What if part of the attack was to make it look like happened at
much late date/time after the malicious code was mirrored and backed up to the
offsite repositories?

------
bulibuta
Scary stuff.

Please use passwords for your keys and allow key access only to a small set of
known IP addresses.

Also do share other security techniques you're using besides the ones above.

~~~
danielpal
Passwords are useful, but impossible to enforce. If anyone decides not to use
the passwords for their key you are in the same spot. You need better tools,
something you can enforce server side. We use two-factor: (note that I founded
Authy) <http://blog.authy.com/two-factor-ssh-in-thirty-seconds>

------
niels_olson
End-user question: any word on how this affects my pc-bsd laptop recenttly
updated to 9.1 RC-2?

------
DrCatbox
They use SVN still?

~~~
wazoox
What's wrong with svn?

~~~
harshreality
Lacking cryptographic hashes used by most DVCS software, SVN repositories need
some external mechanism to be pretty sure the source repository hasn't been
compromised. For instance, one might have snapshots of the SVN tree taken
every week.

It's much easier with git or hg, even in the single-master-repo model, even if
all developers are using the CVS gateway so they don't have local copies of
commit history. All such a project would need in order to protect against
compromise (ignoring physical redundancy) is a backup git or hg repo, kept
offline except for periodic pulls. As long as there are no warnings about
upstream rebases, and you trust all visible commits, then everything's
fine[1].

With SVN, as long as there's no built-in cryptographically secure way to
connect one snapshot to the next when backing up SVN repos, determining the
existence of a backdoor requires comparing entire backup SVN repos against the
live main repo. Even if sufficient backups exist, the process is slow and
requires taking the main SVN repo offline if there's any chance of a root
compromise.

[1] Caveat 1: in any RCS, there could be backdoors masquerading as legitimate-
looking commits if the attacker had commit access. Caveat 2: The security is
subject to the cryptographic security of SHA-1.

In this FreeBSD case, if they're certain the svn repo wasn't compromised, all
they have to do is validate the checkout on the two compromised machines
against the svn repo.

~~~
tptacek
This is kind of silly. They can simply diff to find changes. All the
developers have several checkouts.

There are lots of things that are great about git, but it's not a security
cure-all.

