
Osquery: SQL-powered operating system instrumentation, monitoring, and analytics - vsurabhi
https://github.com/osquery/osquery
======
gen220
Our security team uses this, we’re about 1500 employees. However, I believe
they opted to use a fork over this linked version, citing (IIRC) that
Facebook’s roadmap for this project was a little bit too unpredictable, and
that they weren’t responsive to PRs and community requests. I think we went
with [https://github.com/osql/osql](https://github.com/osql/osql).

It’s installed on every laptop (chrome books and MacBooks), and I _believe_ on
every EC2 instance, and they have nothing but good things to say about it. We
did have to come up with an aggregator solution for storing the results of the
queries, but I’m under the impression that it wasn’t too big of a lift.

~~~
groobongithub
Facebook has since transfered the project to the Linux Foundation, and the
group behind osql is largely the same group of maintainers on the current
osquery.

[https://www.linuxfoundation.org/press-release/2019/06/the-
li...](https://www.linuxfoundation.org/press-release/2019/06/the-linux-
foundation-announces-intent-to-form-new-foundation-to-support-osquery-
community/)

~~~
gen220
This is great news! Thank goodness that something like the Linux Foundation
exists; this is a perfect piece of software for it to absorb.

~~~
caniszczyk
The osquery foundation is just getting started and holding office hours,
you're welcome to join!
[https://github.com/osquery/foundation](https://github.com/osquery/foundation)

------
terracatta
Kolide's CEO here! ([https://kolide.com](https://kolide.com))

We have been a big part of the Osquery community for a while and we think it's
an awesome project that can be used to get an insane level of visibility
across your fleet. We also think in the context of rolling this out to end-
users, companies need to really consider the privacy implications of the data
Osquery is capable of collecting.

To that end, we soft-launched a product in May that helps fast growing tech
companies use tools like Osquery to implement something called User Focused
Security. User Focused Security involves treating employees like adults and
understanding the context in which they work before rolling out a security
strategy.

We want to be the best choice for organizations that want to get serious about
the security of their laptops but don't want to lock-down devices, violate
their user's privacy, or hurt their internal culture with opaque surveillance.

The three values that we use to build our software:

1\. User Education over Enforcement

2\. Trust through Transparency

3\. Quality conclusions over Quantity of data

We use Osquery because it helps us fulfill that second value by giving end-
users visibility into what is running on their device.

If your team uses Slack and want to see our approach you should check us out
at [https://kolide.com](https://kolide.com)

~~~
terracatta
BTW, if you are interested in learning more about User Focused Security and
how it might scale to really large companies, I definitely recommend reading a
recent interview we did with Jesse Kriss at Netflix
[https://blog.kolide.com/ufs-spotlight-jesse-kriss-of-
netflix...](https://blog.kolide.com/ufs-spotlight-jesse-kriss-of-
netflix-b5c22923f53e)

------
tony
Amazing

CMake, clang-format, watchman

The code looks beautiful, examaple:
[https://github.com/osquery/osquery/blob/master/osquery/files...](https://github.com/osquery/osquery/blob/master/osquery/filesystem/posix/fileops.cpp)

>
> [https://github.com/osquery/osquery/blob/master/LICENSE](https://github.com/osquery/osquery/blob/master/LICENSE)
> > If you're using osquery you are free to choose one of the provided
> licenses.

If GPLv2 is optional, then why have it all? (To give the user flexibility
since GPLv2 isn't forward compatible to from Apache2)

A simpler license w/ a separate patent exception would give the user choice, a
forker could "upcast" to any forward-compatible license.

------
jasonjayr
Something like this already exists for Windows[1]

There is apparently a standard[2] for exposing operating system objects into a
schema that can have a standard query.

I don't think I've ever seen any common monitoring projects built using
WMI/WBEM at their core ...

[1]: [https://docs.microsoft.com/en-
us/windows/win32/wmisdk/about-...](https://docs.microsoft.com/en-
us/windows/win32/wmisdk/about-wmi)

[2]: [https://stackoverflow.com/questions/921617/is-there-
somethin...](https://stackoverflow.com/questions/921617/is-there-something-
like-wmi-for-linux)

~~~
jpcosta
afaik one of the main motivations that led to start osquery project was
precisely to have a cross platform tool to allow collecting much needed
information from all your hosts in an enterprise setting

~~~
cryptonector
Which WMI is when you call it CIM so there's no 'W' in it. Sun had an
implementation too -- it was never really Windows-specific.

------
cube2222
I love the idea of providing various functionality under a sql interface. Sure
sometimes it doesn't fit, but overall it's one of the better lowest common
denominators I've met so far.

I've used osquery a few times on my personal laptop (this post reminds me to
try to get the company I work in to adopt it!) and for me it was one of the
bigger inspirations for creating OctoSQL[1] as a means for such tools to
interoperate.

[1]:[https://github.com/cube2222/octosql/](https://github.com/cube2222/octosql/)

------
NohatCoder
Am I the only person who read "SQL-powered operating system"?

It is just plausible enough to be imaginable. Everything is a table! Write
applications using only SQL statements!

~~~
aasasd
A filesystem isn't so different from a database in the first place. Not really
relational, but still.

The olde PalmOS had databases as primary storage. Though databases seem to
have had capacity for blobs, since apps themselves were stored that way, aside
from text files, images and whatnot.

In fact, afaik some mainframe OSes were built around databases.

------
y4mi
I stil remember the initial announcement of this years ago... I wasn't able to
use it back then but saved it for later.

I'm currently in a situation in which I'd love to use osquery which is why I
tried it out a few month ago.

Sadly, there wasn't any inbuilt multi-node/cluster functionality to speak of.

I gave up on it as it's utility is pretty low if you're constrained to
localhost queries... And the third party "cluster" tools looked pretty
barebones and seemed a hassle to setup. And not even really useful, as they
just enable you too execute queries on several nodes.

I would want to do queries across servers. (I.e. select load,uptime, hostname
where servertype "worker" and kernelversion "3.4")

there was very little value for me which I could already get with an adhoc
Ansible task on my servers

~~~
zercurity
Hey, there are a few open-source projects doing this:

\- [https://github.com/kolide/fleet](https://github.com/kolide/fleet) \-
[https://github.com/mwielgoszewski/doorman](https://github.com/mwielgoszewski/doorman)
\-
[https://github.com/zentralopensource/zentral](https://github.com/zentralopensource/zentral)

Then there are a few freemium:

\- [https://www.zercurity.com/](https://www.zercurity.com/) (Shameless plug -
free for as long as you like) \- [https://kolide.com/](https://kolide.com/)
(Though no SQL interface anymore) \-
[https://www.uptycs.com/](https://www.uptycs.com/) (Free trial)

Hope that helps. I'm sure there are a few others.

------
athenot
OSQuery is pretty powerful and the SQL-like query makes it easier to correlate
various system metrics in 1 step.

However compared to a central metric system that can aggregate metrics across
all the hosts, its use quickly becomes less important.

Also there are some CPU considerations, as OSQuery is not as lightweight as
other metrics gather tools. Several times I've ran into OSQuery interfering
with the actual application, competing for resources. So if you do run it,
make to renice it to mitigate this, especially if you're running time-
sensitive apps like video/audio.

------
cryptonector
WEBM/WMI/CIM[0] exist already.

I'd like something that integrates canned DTrace/eBPF scripts, along with
authorization (some canned scripts might leak sensitive data). Really, a bit
of an idempotent, extensible, remote OS observability protocol.

[0]
[https://en.wikipedia.org/wiki/Windows_Management_Instrumenta...](https://en.wikipedia.org/wiki/Windows_Management_Instrumentation)

------
techntoke
The Linux kernel already has a pretty good API available via file nodes. And
there are other lightweight tools to gather and parse information. Not sure I
understand the benefits of exposing it through SQL, but I know some people are
obsessed with SQL.

~~~
smitty1e
Normalizing the data without vast sort/sed/awk gymnastics seems the key win.

~~~
brazzledazzle
My understanding is there’s security and standardization benefits as well. If
I’m remembering correctly there was a local keychain credential stealing
attack around the time I was first looking at it and they had a plug-in
available for detection the same day. While it wasn’t something magical you
couldn’t write, test and run on your fleet yourself a central place to
deduplicate that sort of work/test cycle and collaborate was compelling.

------
todaysmistakes
This sounds like my hell on Earth

~~~
collyw
You don't like SQL? The language syntax isn't great but the relational model
behind it is a thing of beauty once you get it IMHO.

It sounds like a really interesting idea to me.

I was disappointed that Microsoft attempt at a relational database was it Win
FS failed. Not that I use Windows, but it also seemed to kill the open source
attempts at doing something similar.

