Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Pixie, Instant Kubernetes-Native Application Observability Using EBPF (pixielabs.ai)
108 points by zasgar 63 days ago | hide | past | favorite | 18 comments

Having used an earlier version of this, I highly recommend it if you're carrying a pager in micro-service land, and don't have Brendan Gregg on your team.

I love the idea of instrumenting my clusters with less work. Signed up for the BETA and starting to get data out of one of my GKE clusters right away. Also I like how the cli just interacted with the cluster itself instead of tons of piping the output to other commands. Good luck Pixie team!

Would love to see intrumentation without any code changes. Is it possible to write EBPF based tracers for JVM and Node/Python interpreters?

Co-founder/CEO of Pixie here.

EBPF allows you to access static tracepoints that are defined in many runtime, and can be used to capture information about the state of the runtime.

Since most VM’s/runtimes allow monkey patching you can usually get to the same level of information without using EBPF. We plan to add support for this in Pixie in the future and provide a seamless experience regardless of what underlying tracing technology is used.

Lots of good stuff from Brendan Gregg here: http://www.brendangregg.com/Slides/Velocity2017_BPF_superpow...

Great to see that zasgar. Monkey patching is pretty gnarly to replace functions at load time or in JVM/.NET CLR. Python is easier. Every version change, the monkey patch has to be updated. But you can get a lot of good value from it in the form of performance charts and call graphs of an application. It would be great value if you could reduce the gnarliness of the monkey patching work and possibly replace it with a simple configuration file. Have to look into eBPF in more detail.

Founding engineer at Pixie here.

To add to what zasgar mentioned, I just wanted to point out that our instrumentation-free approach does apply to JVM and Node/Python applications for many of the traces we gather. For example, we use EBPF to trace protocols like HTTP as the data passes through the kernel. By gathering the data in the kernel, these EBPF tracers are completely language agnostic.

What do you do for HTTPS which would be encrypted as it passes through the kernel and is only decrypted in the application process?

Great question. You're right that tracing in the kernel doesn't work for encrypted traffic (that means anything over TLS, including HTTPS). For encrypted connections, we still want to give the no-manual-instrumentation-required experience to our users, so what we do is trace the SSL/TLS library to capture the traffic. Right now, for example, we trace traffic going through OpenSSL. This has the benefit of covering a wide array of programs in different languages, including any dynamic languages that use OpenSSL. And we plan on adding more TLS libraries soon (e.g. GoTLS) to fill in the gaps.

We'll be publishing a blog post on this soon, so please stay tuned. In the meanwhile, this other post (https://docs.pixielabs.ai/tutorials/simple-go-tracing/) gives an idea of how one can use EBPF user-space probes to trace applications and system libraries.

this is really amazing. the marketing page is also top notch from a visual perspective

Whats with .ai for an ops tooling?

Co-founder/CEO of Pixie here.

At Pixie, we are building a new type of data system that can deal with the data volume that we collect and process. We have AI/ML models to help classify traffic and make analysis easier. Over the next few weeks, we will release some blog posts that will discuss how this works.


Lol. I love how irrationally upset people get about this.

How is this any worse than downloading a random tarball and running a makefile blindly or installing a deb with god knows what pre or post scripts exist.

If youre that concerned download it and read the script.

>If youre that concerned download it and read the script.

It's possible to detect the use of curl|bash and serve different content accordingly [1]. This adds a burden to to the person checking it trying to make sure that any trickery has been accounted for and that you're actually getting a non-malicious installer.

Aside from that, I don't want scripts shitting files all over my file system - I am firmly of the opinion that software installation should be handled in the package manager so it can be cleanly removed later on, upgraded, and so on using the standard tools.

I also see curl|bash as a red flag because it indicates that either they don't have the skill required to build a deb/rpm/etc. package, or they simply don't care to do so - which I feel indicates they have a lazy or uncaring attitude towards software quality and craftsmanship. This is less of a red flag if they say something "debian/ubuntu users download .deb here [link], everyone else please use curl|bash" to limit the variety of package management systems they have to support, but still concerning in my opinion.

[1] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b... and related discussion https://news.ycombinator.com/item?id=17636032

I doubt the skill problem. Building a deb or rpm is trivial with fpm. Anecdotal but from past experience at a tiny startup delivering dev-tools we found our pipeline to get people to use our tool was significantly better when we just had them curl|bash a script.

Getting people to use your junk is more important than doing it "The Right Way".

> Aside from that, I don't want scripts shitting files all over my file system

Word. One of the things I love about Linux is that the prevalence of package management as a defining feature of a distribution is that it's almost always the case that application files are pretty well organized according to a standard (my home directories are a mess, but stuff outside is mostly pristine). Even things that aren't quite up to par you can at least clean up by just uninstalling the package--not perfect, but it generally does work.

The various companies that couldn't be arsed to fit their shit inside an RPM or DEB (not universal, but most other systems will have a functional enough means of converting a DEB or RPM until something gets popular enough that the community repackages it), or nowadays Flatpack, or even Docker (Docker has its own set of issues, but it's a decent enough isolation mechanism, certainly for filesystem stuff) are slowly re-creating Windows' mess of infinity different install/uninstall systems, all broken in their own unique small ways, and are happy to just yeet everything they need over the root directory (ah, the joys of driver installers designed by electrical engineers), are a plague on the modern Linux ecosystem.

"curl | bash" nonsense doesn't even make your installer distro-agnostic anyway--it just means that you'll slowly uncover the incompatibilities over time. Oh, your devs all use OS X, and weren't aware that BSD-lineage and GNU core utils don't all act the same, or are even entirely different, and didn't account for that? You've never encountered differences between FS layouts and your script falls over and dies because it relies on everything looking exactly like the build environment? Cool, you gotta reinvent the wheel now, and will probably do so poorly.

My fav example was a former coworker who loved to spout the adage "____ is a security company, and doing things securely is our top priority" in response to many security-related questions, but was also responsible for a set of instructions that had a "curl -k https://example.com/thing.sh | bash" line. Said engineer justified this with the rationale that the end users they worked with often had borked environments with ancient broken OpenSSL installations (or TLS intercept proxies not properly configured in Linux machines, whatever, some half-assed reason--I assure you that the download server certificates validity was not an issue), and lacked the expertise to fix them, so in the interest of expediency, they took a shortcut and didn't even bother to mark it as a problem to fix later. Our industry never ceases to impress me.

I want gpg signatures, not a bare tarball or whatever nonsense.

There's a list of these somewhere [1] it's not a great habit to pick up, but it is convenient!!

[1] https://curlpipesh.tumblr.com/

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact