Hacker News new | past | comments | ask | show | jobs | submit login

As the kind-of-sort-of maintainer of Linux's syscall infrastructure on x86, I have a public service announcement: the syscall auditing infrastructure is awful.

It is inherently buggy in numerous ways. It hardcodes the number of arguments a syscall has incorrectly. It screws up compat handling. It doesn't robustly match entries to returns. It has an utterly broken approach to handling x32 syscalls. It has terrifying code that does bizarre things involving path names (!). It doesn't handle containerization sensibly at all. I wouldn't be at all surprised if it contains major root holes. And last, but certainly not least, it's eminently clear that no one stress tests it.

If you really want to use it for production, invest the effort to fix it, please. (And cc me.) Otherwise do yourself a favor and stay away from it. Use the syscall tracing infrastructure instead.




Hi - Ryan from the blog post here.

eBPF is great, and we plan to support it as our collection mechanism, but not today and maybe not soon. When we wrote go-audit, there were few, (if any?), distros that had eBPF kernel support. Now Ubuntu 16.04 has support, as do many others, and that is great!

But..

Releasing a tool that will work for most people today is very different to releasing one that will work for most people in a couple of years. We are sharing this imperfect tool that uses an using an imperfect syscall monitoring mechanism, both of which were written by imperfect humans.

We have found plenty of problems with auditd during development (some real facepalm stuff, to be sure), but it also does many useful things. We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot.

If you don't trust auditd, that's a totally valid opinion to hold. I welcome criticism of our approach, but a PSA that doesn't involve having testing our tool is rather unfair.


> If you don't trust auditd

I know basically nothing about auditd. It's the kernel code I don't trust. Go-auditd may well be fantastic, but treating the kernel part as a reliable black box seems unwise to me.

Edit: you might not need eBPF to get something better. Plain old "perf script" and the underlying ringbuffer API should work decently well on older kernels. There's a performance hit, but Steven Rostedt has a fix, and it should get backported to RHEL at least.


> When we wrote go-audit, there were few, (if any?), distros that had eBPF kernel

> support. Now Ubuntu 16.04 has support, as do many others, and that is great!

This! I'm rather tired of people preaching about eBPF when in reality many of us have to run long term supported kernels/ distros like RedHat or Ubuntu LTS releases.


Where did amluto "preach" eBPF?

What is said is "If Slack really want to use syscall auditing for production, invest the effort to fix it, please. (And cc me.)", which seems totally reasonable to me.

Slack's answer is "We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot." As I understand, this means "No, Slack will not invest any effort to fix Linux syscall auditing in upstream kernel, because we have already worked around limitations." Which is kind of sad and expected.


> We have worked around limitations

Why not try to fix those limitations and push to upstream?


True that, I've been writing [libaudit-go](https://github.com/mozilla/libaudit-go/) which aims to provide a replacement of the c version of auditd libraries and is in constant development. During this period I looked very closely in the auditd source code and I can say it looks like a bunch of things are patched together without much prior thought to make it work.


Is Syscall tracing == perf trace? Ive been doing the same sort of thing with perf. I hadn't heard of the auditing tools.


Yes, more or less. There are bunch of different "tracing" mechanisms in the kernel, and perf trace is the common way to use it. Syscalls trigger tracepoints, and anything that can see tracepoints can see them. Using eBPF to trace syscalls is probably quite useful.


Offtopic (and forgive me if I mess up some details):

I was reading about eBPF and how it's instruction set is easily JITable across several architectures. I was wondering if it would be good as a VM for retro games. Something like Pico-8 [1], but where one could write games in this restricted C or eBPF-ASM.

As I understand it the VM calls specific functions with limits of calling convention (10 64-bit registers shared between arguments, returns etc.). VM can also expose helper functions. It seems quite capable for such a purpose.

Where could I find more materials to explore this side of eBPF? I found a user space eBPF VM [2] - probably a good start.

EDIT: In case anyone is wondering: Why? For fun!

[1] http://www.lexaloffle.com/pico-8.php

[2] https://github.com/rlane/ubpf


If nothing else, you'll need to rework the verifier. The time complexity of verification is super-linear right now. Awhile back I proposed a linear time algorithm that was a bit stricter, but it didn't get implemented.

Also, from memory, there aren't real stack frames right now. (This is a limitation of the implementation and the verifier, not a fundamental issue.)


Thanks for this information! I have a few questions though:

> It doesn't robustly match entries to returns

Do you mean it fails to match the call/return (annoying), or that it may mismatch them? (pretty bad...)

> It doesn't handle containerization sensibly at all.

Do you mean it's just oblivious to namespaces (but works as far as I know) or something actually not working?


>> It doesn't robustly match entries to returns

> Do you mean it fails to match the call/return (annoying), or that it may mismatch them? (pretty bad...)

__audit_syscall_entry() and __audit_syscall_exit() have some interesting checks in them, and I was never convinced that every entry would get paired with the corresponding exit and logged correctly.

>> It doesn't handle containerization sensibly at all.

> Do you mean it's just oblivious to namespaces (but works as far as I know) or something actually not working?

As far as I know, there is one global audit daemon and audit log, and you have to be globally privileged to use it.


We (unintentionally) stress test it. Our Linux admins vociferously insist using it to audit everything that is happening on a 120 core box that spawns tens of heavy processes every second.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: