
Ask HN: Distributed tracing service with critical path analysis? - pseudonom-
The Dapper paper mentions &quot;An engineer working on long tail latency debugging built a small library which infers hierarchical critical paths from DAPI Trace objects. These critical path structures were then used to diagnose problems and prioritize prospective performance improvements for universal
search.&quot;<p>Do any of the distributed tracing services (e.g. Datadog APM, AWS X-Ray) offer this? I don&#x27;t see any services which tout this functionality.
======
itamarst
See if you can find a tool that creates flamegraph input files from the
traces, or if not write one yourself. My causal logging library, Eliot
([https://eliot.readthedocs.io](https://eliot.readthedocs.io)), has an
experimental branch for that, and it wasn't particularly hard to create the
input files.

[https://github.com/brendangregg/FlameGraph](https://github.com/brendangregg/FlameGraph)
will then create nice graphs that can help spot bottlenecks.

------
albertwang
Disclosure: I work at Datadog --

Tracing requests across services and hosts to identify bottlenecks is one
reason why Datadog decided to build a tracing component in APM! A flamegraph
is one such method to visually infer critical path analysis that Datadog APM
supports. Some obvious things to consider are ease of setup and whether the
libraries and languages you use are supported. Some not-so-obvious things to
consider are:

    
    
      (1) whether you wish to infer this from a single request, or across multiple requests, 
      (2) how to define "interesting" traces -- as you could be collecting millions or billions of traces in production 
      (3) how to create a bigger, "aggregate" picture across individual traces.  
    

We're working on additional ways to easily identify critical paths as well.

------
programd
I think what you're looking for is Zipkin [0]. We use it in our microservers
at Nulladmin.com via the opentracing Go library [1] and the zipkin-go-
opentracing library [2]

[0]
[https://github.com/openzipkin/zipkin](https://github.com/openzipkin/zipkin)

[1] [https://github.com/opentracing/opentracing-
go](https://github.com/opentracing/opentracing-go)

[2] [https://github.com/openzipkin/zipkin-go-
opentracing](https://github.com/openzipkin/zipkin-go-opentracing)

~~~
pseudonom-
My company actually has and uses Zipkin. The thing that I'm looking for that I
think is lacking is a way to identify critical paths without manually clicking
through a bunch of traces and eyeballing them.

