
You can't debug distributed systems with dashboards - forrestbrazeal
https://read.acloud.guru/why-you-cant-effectively-debug-your-modern-systems-with-dashboards-57fe3ecd26bf
======
peterwwillis
Yeah, I'm sorry, but this is ridiculous clickbait written by hipster sysengs
using fringe architectures to generalize about a gigantic range of complex
systems and saying "all you need to do is sample some logs, guys, why are you
still using dashboards".

You know what solves ops issues? System visibility connected to automation
designed to diagnose root causes and provide simple solutions to common
problems. Serverless adds extra complexity and points of failure, but that
doesn't mean "you can't debug distributed systems with dashboards". Give me a
break.

~~~
fhood
I clicked on that link ready to experience a revelation that was going to
change how I debugged infrastructure problems.

I came away with nothing. I got nothing out of this article. I have re-read it
twice now hoping that I was missing something, or had failed to understand
some point.

~~~
peterwwillis
The only point was that distributed tracing apparently solves all problems.

------
suff
You can if your loggung structure is uniform, and includes a correlation ID.
This happens every day... pretty simple actually.

~~~
icedchai
Sounds simple. Unfortunately, having uniform logging structure in a non-
trivial project is about as likely to happen as winning the lottery.

~~~
drb91
...only if you view software development like gambling. All problems can be
methodically addressed; fixing log formatting is actually rather easy to track
down.

~~~
icedchai
Yes, in your own code, it is easily fixed... With third party code, like
dependencies, it becomes more of a problem...

~~~
drb91
That's true. But you only need to re-format the events you care about from
that dependency.

~~~
Fellshard
That's still a non-trivial problem, depending on how easy it is to adapt logs
from a library; maybe with a proper logging facade, it's easy to attach those
adapters to those logs, but I suspect most libraries don't have strong
contracts around the formats of their logs, as it is. Maybe that's something
to improve.

------
hinkley
Console logging is the easiest but least effective method of debugging. Some
people refuse to call it debugging because they believe a debugger or
diagnostic tools are involved with the act of “debugging” (vs
troubleshooting?)

Distributed systems mean you have to fall back to the most basic form of
debugging.

You had better be getting something really good for that bargain. You’ve
nearly permanently compromised your ability to reason about obscure bugs in
the system in order to have it, so be sure it was worth it.

~~~
daotoad
You need to take that basic form of "debugging" and turn it into
instrumentation. Collect structured data and build tools to process it. Once
you have a nice collection of data you can build profilers and execution
traces, things classical debugging tools provide.

I've done it and the tools can work quite well. The same consistent
instrumentation of your code also makes it easy to build really great
dashboard monitoring.

All this time building tools costs money and adds complexity to your stack,
so, yes, distribution had better be worth it for your business needs before
you take it on. Pretty much like any engineering trade-off.

~~~
orweis
Do you mean something like this? (Rookout.com)
[https://youtu.be/bAxNWg6QYL8?t=62](https://youtu.be/bAxNWg6QYL8?t=62)

------
drb91
What’s the pricing for the pitched product?

~~~
bboreham
$70 per month and up, based on GB/day transmitted.

[https://www.honeycomb.io/pricing/](https://www.honeycomb.io/pricing/)

------
k__
I'm still wating for the day when these products use serverless-pricing too.

~~~
scarface74
With CloudWatch you can literally do everything the article proposes just by
sending your logs there - you get easy to create dashboards and alerts.

~~~
k__
Nice, thanks for the info :)

Maybe I should to a course on CloudWatch. Seems like a powerful tool, but a
bit convulted at times...

------
kozak
Such a massive unscrollable top bar that I had to open my browser's Dev Tools
and remove it from the DOM it before reading the article.

