I’ve been testing the whole stack on a local server, finding kinks, documenting workflows, because I hope I move us to this soon. Now with Pyroscope I would love to try it out even more. I kid you not, we were just testing Datadog Continuous Profiling for our legacy Ruby application not two days ago and it was quite lackluster.
Not saying this will be better (have yet to try), but I’d prefer to support and report feedback back on this for a brighter future for our community.
Ryan here from Pyroscope -- Awesome to hear you're interested in migrating over. Would love for you to try out our Ruby integration and let us know what you think: https://pyroscope.io/docs/ruby/.
Hey! It was alright, but at least for Ruby it didn’t report on the memory allocation of the calls, which is what we were looking for in the first place. We do not use it anywhere else, we only tried it for this pesky legacy system.
At the company I work at, OpsVerse, we offer a single-click, packaged, pre-configured OSS-driven observability stack [1] that makes heavy use of Grafana (UI + Loki) as well as Pyroscope so it's great to see them take the route of merging projects rather than building a whole new project. Continuous application profiling is such a game-changer and as eBPF becomes more popular, I'm sure tools like pyroscope will become bread and butter for devs.
As an addon, we recently published a blog on how we ended us dogfooding pyroscope to debug a pesky memory leak [2].
We've heard of 2-4% performance hit in terms of performance using pyroscope on Node, but in practice, we've seen ~1% hit in terms of perf (Though we're not running high-performance systems where we're trying to eke out every bit of perf from each pod, so YMMV)
My favorite little anecdote that I like to tell is that the first thing people often see when they add Pyroscope to their apps is that it takes way less CPU than other signals like tracing or logging. It's pretty common to see logging taking 5-10% of overall CPU utilization.
The other 90% is usually spent doing serialization / deserialization (half-joking).
I wonder if Grafana is going to rebrand at some point. They have been aggressively diversifying, but keeping the Grafana name makes people immediately associate the company with the Grafana product and consider the rest as side-projects.
As an engineer, I care about the other products of monitoring: tickets, alerts, reports. My management? The people with the money? All they think about from that stack is the graphs they can view on Grafana.
This is being worked out, but the gist is likely that we end up merging things rather than choosing between them.
Pyroscope have driven more of the clients and have a deeper understanding of the UI — so we'll probably bias towards Pyroscope for those — and Phlare has driven more of the scalable storage and database — so we'll probably bias towards Phlare for the database.
Both teams bring different perspectives of the same problem space, and the two products and their teams complement each other well, so we'll figure out how to take the best from each.
The biggest win is the people, the Pyroscope people are incredible and they join a few of our best people at Grafana Labs who were already working on Phlare — in doing so they create this high talent density with a strong purpose to make profiling be an essential, easy-to-use and scalable 4th pillar of observability. For me that is the real potential, to amplify the possibilities of both teams and products by bringing them together.
I've heard "new" Grafana is running something like chromium for server-side rendering under the hood. Is it true? (it's freaking me out a bit) Sorry if it's a silly question.
What is "new" Grafana in this context? The only thing that sounds similar to this is the Grafana Image Renderer plugin, which uses Chromium. But that's hardly new (it's been around for about 4 years), and it's not part of core Grafana...
Continuous profiling is a next big thing IMO - easier to get started with than distributed tracing and delivers immediate value.