
MacOS monitoring the open source way - el_duderino
https://blogs.dropbox.com/tech/2018/04/4696/
======
tptacek
This is super interesting and ambitious and I'm always sort of envious when I
look at big engineering orgs, like Dropbox and Slack, where everything has
fine-grained instrumentation.

We do this kind of work for startups, and we have a small roster of companies
where we're on the hook for this kind of corpsec.

But then I think about what it'd be like to actually deploy and operationalize
this stuff, and it gives me pause.

I have two big questions --- real questions, like, I have no pretense of
having an answer and am speaking from ignorance --- about how this stuff works
in practice.

The first is privacy. They're deploying this across a fleet of company
laptops. People do all sorts of stuff on their corp laptops. I don't know
enough about Dropbox's company culture to know whether people ever use their
machines for personal stuff, but I do know that occasional personal use is an
SFBA startup norm. They're collecting essentially a continuous bash history
from every user in their fleet. How comfortable are engineers with that? I
don't have a strong opinion, but it gives me enough pause that I'd be a little
afraid to ask a client to do it.

The second is: what do you do with all that information? I see how this
addresses malware response, and I see how this would be useful as a forensic
archive for general IR. But as a day-to-day tool, connection-by-connection,
file-by-file granular data from a mostly technical team seems incredibly
noisy. Most engineers look like attackers: they make out-of-process
connections to random backend resources, copy things around, and run new code.

I acknowledge right away that my vantage point is small startups, not
organizations running at the scale Dropbox is.

~~~
antoncohen
> They're deploying this across a fleet of company laptops ... How comfortable
> are engineers with that?

I worked for the corporate infrastructure team at Dropbox, at the time my team
was responsible installing/managing things like osquery on laptops, with the
data going to the security team's tools, so I have thought about this.
Speaking for myself, the Dropbox culture takes trustworthiness very seriously.
"Be worthy of trust" is #1 on the list of company core values, and it is
deeply embedded into the culture. The trust covers so much of how the company
operates, including privacy. Spying on employees is so against company
culture, even though I knew exactly what data being sent, I trusted the
security team would not use that for anything other than protecting me. The
detection and response team is made up of the most security and privacy
conscious people I've ever met.

I think the mindset of respecting coworker privacy is important for anyone in
corporate infrastructure, including network and system administrators.

But I've worked for other big corporations too, without that trust. I think
employees need to know that when they use corporate computers and
communications systems, they aren't private. Don't do stuff you shouldn't be
doing at work.

> what do you do with all that information? I see how this addresses malware
> response, and I see how this would be useful as a forensic archive for
> general IR. But as a day-to-day tool, connection-by-connection, file-by-file
> granular data

When I worked there the security team reached out to me to ask about anomalous
behavior, so there is active detection going on. That was a couple years ago,
I'm sure they have improved since then too. Keep in mind, they have a full
time detection and response team, in addition to all the other security teams.

~~~
stirner
When Dropbox deliberately circumvented Apple's security features to make
itself difficult to remove [1], was the company's #1 core value of "Be worthy
of trust" in mind, or was that only added in retrospect?

[1] [http://applehelpwriter.com/2016/08/29/discovering-how-
dropbo...](http://applehelpwriter.com/2016/08/29/discovering-how-dropbox-
hacks-your-mac/)

~~~
antoncohen
This issue has been addressed already:

[https://news.ycombinator.com/item?id=12464730](https://news.ycombinator.com/item?id=12464730)

I don't believe there was any intention of making Dropbox hard to uninstall. I
think the intention was to make the process seamless for users, and there was
a bug that caused a setting to get reapplied after a user changed it.

~~~
saagarjha
I believe that Dropbox doesn't really have poor intentions here: they're
trying to make their product easy to use or work better or whatever. I don't
think their intentions are malicious. However, I strongly disagree with the
method that they use to achieve this: installing kernel extensions, bypassing
Accessibility prompts, having a hooks in every process they could possibly get
their hands on, etc. is going too far. There's a reason those checks are
there: they keep the user safe. Trying to get around these is, in my mind,
arrogant. They think that they're better than every other company that abides
by the rules. Google Drive doesn't do this. Box doesn't do this. Even _iCloud
Drive_ doesn't do what Dropbox does.

I was talking to someone just two days ago, who was tearing their hair out
because their application's "Open" panel took something like ten seconds to
open. Why? Because the Dropbox extension decided it didn't want to play along
nicely. It's foolish to think that your software will not have bugs, and
outright foolhardy to do this for so little benefit.

~~~
antoncohen
Dropbox pushes the state of the art for file sync. They added sync status
icons on Mac OS X before Apple had an API for it, later Apple added an API.
Those icons are part of what makes the product so usable.

The kernel extension is another example. AFAIK it was added for Smart Sync[1]
(née Infinite[2]). Infinite is amazing, truly amazing, and the only way to
implement it is via a kernel extension. Microsoft tried to implement it via
the GUI with OneDrive's "smart files" (aka placeholders), they removed it
because the files didn't work in too many places, like via syscalls and from
command line.

Dropbox Infinite is kind of like the source control systems Google and
Microsoft created to handle their huge repos. Microsoft's GVFS uses a file
system filter driver (kernel extension) called GvFlt (or ProjFS)[3]. From what
I understand, Google's Piper uses FUSE[4], which would be a third-party kernel
extension on macOS.

My point is, these are technical achievements that provide seamless and
intuitive user experiences, they aren't betrayal of trust.

[1] [https://www.dropbox.com/smartsync](https://www.dropbox.com/smartsync)

[2] [https://blogs.dropbox.com/tech/2016/05/going-deeper-with-
pro...](https://blogs.dropbox.com/tech/2016/05/going-deeper-with-project-
infinite/)

[3] [https://www.visualstudio.com/learn/gvfs-
architecture/](https://www.visualstudio.com/learn/gvfs-architecture/)

[4] [https://cacm.acm.org/magazines/2016/7/204032-why-google-
stor...](https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-
billions-of-lines-of-code-in-a-single-repository/fulltext)

~~~
fapjacks
> My point is, these are technical achievements that provide seamless and
> intuitive user experiences, they aren't betrayal of trust.

You having put a lot of effort in this thread into weaving trust into your
description of Dropbox operations, I'm surprised that you wouldn't _trust_ an
end user when they tell you that this in fact _is_ a betrayal of their trust
in Dropbox.

------
untangle
The MacOS security products (donationware) are also excellent and overlap the
functionality of these.

[https://objective-see.com/products.html](https://objective-
see.com/products.html)

~~~
armitron
They don't really. Objective See's tools are not bad for what they are
(utilities), but they are lacking in:

1\. Code quality (Patrick Wardle is not an engineer and it shows in his code).
This is not terribly important for an end-user utility but becomes very
important when you're deploying kernel modules across your entire organization
that need to be absolutely rock solid and not introduce additional threat
vectors.

2\. Distributed nature (They're utilities meant to be executed by end-users on
their own machines, not distributed agents syncing up with cloud servers)

and so on..

------
discussedbefore
osquery came up a few days ago; in use at Etsy, including a direct link to a
common setup.

Ask HN: Is no anti-virus software still best practice for mac?

[https://news.ycombinator.com/item?id=16904103#16904721](https://news.ycombinator.com/item?id=16904103#16904721)

> _[https://github.com/facebook/osquery/blob/master/packs/osx-
> at...](https://github.com/facebook/osquery/blob/master/packs/osx-
> attacks.conf) _

Also a discussion of how this type of monitoring worked out in practice in the
Google/Uber/Lewandowski case last year:

[https://news.ycombinator.com/item?id=13860890#13861475](https://news.ycombinator.com/item?id=13860890#13861475)

> _the level of detail that Google has over the logs and actions of their
> laptop_

> _[https://github.com/google/grr](https://github.com/google/grr) _

------
cheerioty
After they praised OSS a lot in that post, one (me ;)) would have assumed they
announce the open source release of their "plumbing" code to make it all work
nicely, build process trees, etc.

------
dguido
Trail of Bits has done a study on how large technology companies are
increasingly switching to osquery for their endpoint monitoring needs. You can
read the results here:

[https://blog.trailofbits.com/2017/11/09/how-are-teams-
curren...](https://blog.trailofbits.com/2017/11/09/how-are-teams-currently-
using-osquery/)

[https://blog.trailofbits.com/2017/12/21/osquery-pain-
points/](https://blog.trailofbits.com/2017/12/21/osquery-pain-points/)

[https://blog.trailofbits.com/2018/04/10/what-do-you-wish-
osq...](https://blog.trailofbits.com/2018/04/10/what-do-you-wish-osquery-
could-do/)

We also offer a commercial service to make custom modifications, bugfixes, and
feature enhancements to osquery. It's little known at this point, but we do
the same for Google Santa too!

[https://www.trailofbits.com/services/osquery-
support/](https://www.trailofbits.com/services/osquery-support/)

~~~
newman314
FWIW, I'd love to see AIX support for osquery. Wondering if there is any
interest from others.

~~~
dguido
We have a client that may ask us to add that! Please get in touch if you're
willing to sponsor that kind of development.

------
grouseway
I started to do something like this for my Windows box but then I realized
that tracking process execution is not all that useful if you don't also track
DLLs (because it's simple to just dump DLLs in a trusted applications folder
and wait for them to be loaded). Tracking DLLs would just produce too much
noise to monitor which put me off the whole thing altogether.

------
hugofromboss
Shameless plug but anyone interested in deploying osquery and google's santa
into their environment. Should check out
[https://www.zercurity.com](https://www.zercurity.com) \- it supports Linux
and Windows too.

------
mlosapio
OpenBSM is awesome except you’re forced to invent your own way of log
gathering - which becomes more painful when you’re mobile or offline and then
you’ve got to keep state on what’s been transmitted to the mothership.

Would be nice for some insight into Dropbox’s solution here...

~~~
antoncohen
To address logging offline you need a log shipper that will do reliable
logging and pick of where it left off. I think rsyslog, Elastic Beats, and
Splunk forwarder will all do that. Then logs are sent when a machine connects
to a network.

For mobile (online but outside corporate network) there are two options I've
heard of being done:

1\. Have each endpoint have a unique TLS certificate, and have the log shipper
do mutual TLS to the logging server which has a public IP.

2\. Have a backhaul VPN that is allows connected, automatically, to the
monitoring network, and send the logs over that. That VPN is different than
the user VPN that gives access to the corporate network.

~~~
Khaine
Yes, but to get BSM into Elastic Beats, you either need to make a shim to
convert from BSM binary format into json for FileBeat to consume, or you need
to write your own Beats for BSM files.

------
Khaine
This is nice, but what would be better is the glue they use to integrate them
into other monitoring solutions. How do these merge these tools into their
existing infrastructure? Do they parse the logs from BSM, santa and osquery
into ELK? If so, how.

The real difficulty is not finding useful open source tools, it is integrating
them into existing monitoring solutions used within an organisation to get a
single view of activity on a system and on a collection of systems (i.e. how
do you make the tool scale).

------
mekazu
Was anybody else thinking these tools would be great for general development
and debugging purposes? Anything that uses the network or file system anyway.
The fact that they can detect malware reads as just an aside to me.

------
p2t2p
There is also intellectual property issue. In my case whatever code I write on
corporate laptop belongs to the company I work for so it’s kinda wrong to use
it for pet projects.

------
bradknowles
Okay, so how do we use these kinds of tools to monitor our own personal
machines?

------
rand39120427389
What is the equivalent of these 3 tools on Linux (Ubuntu)?

