
Twitter's Tool For Debugging At Scale - cdl
http://twitter.github.io/zipkin/
======
bmdhacks
Hey, developer on zipkin here. Happy to answer any questions you have.

~~~
Ologn
I'm happy to see the video of the Zipkin presentation at Strange Loop was put
up a few months ago (
[http://www.infoq.com/presentations/Zipkin](http://www.infoq.com/presentations/Zipkin)
).

People working on distributed tracing systems all tend to eventually come up
with a similar architecture. Even going back to the IPS research system made
at University of Wisconsin in the late 1980s (that's the earliest distributed
tracing system I know of).

They all tend to do tracing via minimalistic, low overhead logging of RPC
calls between machines. They tend to do tracing via low-level libraries which
application developers can ignore. The trace systems seems to be good at
uncovering latency bottlenecks. I am ignorant of what success systems like
Zipkin or Google's Dapper may or may not have had in areas outside of latency
checks.

~~~
bmdhacks
One of the projects I've been working on is doing more aggregate analysis of
zipkin data:
[https://github.com/twitter/zipkin/pull/276](https://github.com/twitter/zipkin/pull/276)

We're using this at Twitter to better understand usage patterns for services
upstream and downstream. For example, Gizmoduck, the user store at twitter is
backed by memcache, and some disk-based storage behind the cache. While we can
view individual traces that hit by memcache, the aggregate info shows us both
the proportion of traffic for services calling Gizmoduck, as well as the
proportion of time Gizmoduck spends in memcache versus the backend store.

Furthermore, it can be useful for notifying of unusual behavior. If a
service's aggregate durations has changed since yesterday, perhaps that's
something we want to look at. Or if the ratio of traffic from some upstream
service doubles, that's interesting to know.

------
alxndr
some actual info:
[https://github.com/twitter/zipkin/blob/master/README.md](https://github.com/twitter/zipkin/blob/master/README.md)

------
arrowgunz
Hey guys, the images on the project page are missing. Here's a Pull Request
that fixes that:
[https://github.com/twitter/zipkin/pull/278](https://github.com/twitter/zipkin/pull/278)

~~~
bmdhacks
Merged thanks

~~~
arrowgunz
Hey forgot images in the other files. Sent another one. Sorry 'bout that.
[https://github.com/twitter/zipkin/pull/279](https://github.com/twitter/zipkin/pull/279)

------
codex
How is this not just a clone of Dapper?

~~~
nacs
Very first paragraph of website:

"We closely modelled Zipkin after the Google Dapper paper"

[http://research.google.com/pubs/pub36356.html](http://research.google.com/pubs/pub36356.html)

------
joshfraser
an open source version of New Relic?

~~~
theatrus2
New relic is not a distributed tracing system.

